Deep Music Generation via
Music Frameworks

Music Framework is a hierarchical music structure representation. With Music Frameworks and new musical features, we combine music domain knowledge with deep learning, and factor music generation into sub-problems, which allows simpler models, requires less data, and achieves high musicality.

Screen Shot 2021-11-01 at 13.11.20.png
Screen Shot 2021-11-01 at 13.18.06.png

I. Controllable Deep Melody Generation via Music Frameworks

Demo

  • Example 0 contains some cherry-picked demos, Examples 1 - 6 are non-cherry-picked demos.

  • We directly use the chord progression from the original input song. In Example 0, we use the original accompaniment from the input song as well.

  • "#G" is the group number we used in the paper to identify different generating controls.

  • In Examples 1 - 6,  #G group numbers 3 and 5 are control groups not using Music Frameworks.

  • All music Copyright 2021 Shuqi Dai. Please do not copy or distribute any AI-generated music in this page.

Example 0

Original Input Song1

Output New Melody

(Copy Basic Melody & Rhythm, Generate Melody)

(Generate Basic Melody, Copy Rhythm, Gen Mel)

(Copy Basic Melody, Generate Rhythm, Gen Mel)

Original Input Song2

Output New Melody

(Copy Basic Melody & Rhythm, Generate Melody)

(Generate Basic Melody, Copy Rhythm, Gen Mel)

(Generate Basic Melody & Rhythm, Gen Mel)

(Generate Basic Melody & Rhythm, Gen Mel)

Original Input Song3

Output New Melody

(Copy Basic Melody & Rhythm, Generate Melody)

(Copy Basic Melody & Rhythm, Generate Melody)

(Copy Basic Melody & Rhythm, Generate Melody)

(Generate Basic Melody, Copy Rhythm, Gen Mel)

(Generate Basic Melody, Copy Rhythm, Gen Mel)

(Copy Basic Melody, Generate Rhythm, Gen Mel)

Original Input Song4

Output New Melody

(Copy Basic Melody, Generate Rhythm, Gen Mel)

(Generate Basic Melody & Rhythm, Gen Mel)

Example 1

Original Input Song

#G

Rhythm

Generated New Melody

Realized Melody

Basic Melody

generate

copy

copy

generate

generate

copy

generate

without

copy

1

2

3*

generate

copy

generate with Basic Rhythm Form 

4

generate

copy

generate without Basic Rhythm Form 

5*

control

group

control

group

generate

generate

6

generate with Basic Rhythm Form 

original

Example 2

Original Input Song

#G

Rhythm

Generated New Melody

Realized Melody

Basic Melody

generate

copy

copy

generate

generate

copy

generate

without

copy

1

2

3*

generate

copy

generate with Basic Rhythm Form 

4

generate

copy

generate without Basic Rhythm Form 

5*

generate

generate

6

generate with Basic Rhythm Form 

control

group

control

group

original

Example 3

Original Input Song

#G

Rhythm

Generated New Melody

Realized Melody

Basic Melody

generate

copy

copy

generate

generate

copy

generate

without

copy

1

2

3*

generate

copy

generate with Basic Rhythm Form 

4

generate

copy

generate without Basic Rhythm Form 

5*

generate

generate

6

generate with Basic Rhythm Form 

control

group

control

group

original

Example 4

Original Input Song

#G

Rhythm

Generated New Melody

Realized Melody

Basic Melody

generate

copy

copy

generate

generate

copy

generate

without

copy

1

2

3*

generate

copy

generate with Basic Rhythm Form 

4

generate

copy

generate without Basic Rhythm Form 

5*

generate

generate

6

generate with Basic Rhythm Form 

control

group

control

group

original

Example 5

Original Input Song

#G

Rhythm

Generated New Melody

Realized Melody

Basic Melody

generate

copy

copy

generate

generate

copy

generate

without

copy

1

2

3*

generate

copy

generate with Basic Rhythm Form 

4

generate

copy

generate without Basic Rhythm Form 

5*

generate

generate

6

generate with Basic Rhythm Form 

control

group

control

group

original

Example 6

Original Input Song

#G

Rhythm

Generated New Melody

Realized Melody

Basic Melody

generate

copy

copy

generate

generate

copy

generate

without

copy

1

2

3*

generate

copy

generate with Basic Rhythm Form 

4

generate

copy

generate without Basic Rhythm Form 

5*

generate

generate

6

generate with Basic Rhythm Form 

control

group

control

group

original