Functional Generation of Harmony and Melody
José Pedro Magalhães
Hendrik Vincent Koops
Functional Art, Music, Modeling and Design 2014
6 September 2014
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
1 / 15
An extremely useful model of musical harmony
A couple of years ago I worked on a modelling musical harmony as a
GADT in Haskell (together with W. Bas de Haas). From this model
came:
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
2 / 15
An extremely useful model of musical harmony
A couple of years ago I worked on a modelling musical harmony as a
GADT in Haskell (together with W. Bas de Haas). From this model
came:
I
A pretty-printer that shows the tree structure of a sequence of
chords (ICFP 2011);
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
2 / 15
An extremely useful model of musical harmony
A couple of years ago I worked on a modelling musical harmony as a
GADT in Haskell (together with W. Bas de Haas). From this model
came:
I
A pretty-printer that shows the tree structure of a sequence of
chords (ICFP 2011);
I
A system for finding cover songs based on harmonic similarity
(ISMIR 2011);
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
2 / 15
An extremely useful model of musical harmony
A couple of years ago I worked on a modelling musical harmony as a
GADT in Haskell (together with W. Bas de Haas). From this model
came:
I
A pretty-printer that shows the tree structure of a sequence of
chords (ICFP 2011);
I
A system for finding cover songs based on harmonic similarity
(ISMIR 2011);
I
An improved chord recogniser from audio that is aware of the rules
of musical harmony (ISMIR 2012);
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
2 / 15
An extremely useful model of musical harmony
A couple of years ago I worked on a modelling musical harmony as a
GADT in Haskell (together with W. Bas de Haas). From this model
came:
I
A pretty-printer that shows the tree structure of a sequence of
chords (ICFP 2011);
I
A system for finding cover songs based on harmonic similarity
(ISMIR 2011);
I
An improved chord recogniser from audio that is aware of the rules
of musical harmony (ISMIR 2012);
I
A company (http://chordify.net);
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
2 / 15
An extremely useful model of musical harmony
A couple of years ago I worked on a modelling musical harmony as a
GADT in Haskell (together with W. Bas de Haas). From this model
came:
I
A pretty-printer that shows the tree structure of a sequence of
chords (ICFP 2011);
I
A system for finding cover songs based on harmonic similarity
(ISMIR 2011);
I
An improved chord recogniser from audio that is aware of the rules
of musical harmony (ISMIR 2012);
I
A company (http://chordify.net);
I
A system for automatic melody harmonisation (FARM 2013);
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
2 / 15
An extremely useful model of musical harmony
A couple of years ago I worked on a modelling musical harmony as a
GADT in Haskell (together with W. Bas de Haas). From this model
came:
I
A pretty-printer that shows the tree structure of a sequence of
chords (ICFP 2011);
I
A system for finding cover songs based on harmonic similarity
(ISMIR 2011);
I
An improved chord recogniser from audio that is aware of the rules
of musical harmony (ISMIR 2012);
I
A company (http://chordify.net);
I
A system for automatic melody harmonisation (FARM 2013);
I
FComp: a system for automatic generation of harmony and
accompanying melody (this talk).
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
2 / 15
An example: visualising harmonic structure
Piece
Phrase
Ton
I : Maj
Dom
Sub
Ton
Dom
I : Maj
C : Maj III : Min IV : Maj II : Dom7 V : Dom7 C : Maj
E : Min
F : Maj
D: Dom7 G : Dom7
You can see this tree as having been produced by taking the chords in
green as input. . .
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
3 / 15
An example: generating harmonic structure
Piece
Phrase
Ton
I : Maj
Dom
Sub
Ton
Dom
I : Maj
C : Maj III : Min IV : Maj II : Dom7 V : Dom7 C : Maj
E : Min
F : Maj
D: Dom7 G : Dom7
You can see this tree as having been produced by taking the chords in
green as input. . . or the chords might have been dictated by the structure!
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
3 / 15
System structure
Define harmony model
Set rule weight
Take all possible
melody notes per chord
Filter melody notes
Pick one focal melody
note per chord
Embellish melody
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
4 / 15
A functional model of harmony
Piece M → [Phrase M ]
José Pedro Magalhães
(M ∈ {Maj, Min})
Functional Generation of Harmony and Melody, FARM 2014
5 / 15
A functional model of harmony
Piece M → [Phrase M ]
(M ∈ {Maj, Min})
Phrase M → TonM DomM TonM
|
DomM TonM
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
5 / 15
A functional model of harmony
Piece M → [Phrase M ]
(M ∈ {Maj, Min})
Phrase M → TonM DomM TonM
|
DomM TonM
TonMaj → IMaj
m
TonMin → IMin
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
5 / 15
A functional model of harmony
Piece M → [Phrase M ]
(M ∈ {Maj, Min})
Phrase M → TonM DomM TonM
|
DomM TonM
TonMaj → IMaj
m
TonMin → IMin
7
DomM → VM
| VM
| VII 0M
| Sub M DomM
7
| II 7M VM
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
5 / 15
A functional model of harmony
Piece M → [Phrase M ]
(M ∈ {Maj, Min})
Phrase M → TonM DomM TonM
|
DomM TonM
TonMaj → IMaj
m
TonMin → IMin
Sub Maj → II m
Maj
| IV Maj
| III m
Maj IV Maj
Sub Min → IV m
Min
7
DomM → VM
| VM
| VII 0M
| Sub M DomM
7
| II 7M VM
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
5 / 15
A functional model of harmony
Piece M → [Phrase M ]
(M ∈ {Maj, Min})
Phrase M → TonM DomM TonM
|
DomM TonM
TonMaj → IMaj
m
TonMin → IMin
7
DomM → VM
| VM
| VII 0M
| Sub M DomM
7
| II 7M VM
José Pedro Magalhães
Sub Maj → II m
Maj
| IV Maj
| III m
Maj IV Maj
Sub Min → IV m
Min
IMaj
m
IMin
7
VM
VII 0M
→ C : Maj
→ C : Min
→ G : Dom7
→ B: Dim
Functional Generation of Harmony and Melody, FARM 2014
5 / 15
A functional model of harmony
Piece M → [Phrase M ]
(M ∈ {Maj, Min})
Phrase M → TonM DomM TonM
|
DomM TonM
TonMaj → IMaj
m
TonMin → IMin
7
DomM → VM
| VM
| VII 0M
| Sub M DomM
7
| II 7M VM
Sub Maj → II m
Maj
| IV Maj
| III m
Maj IV Maj
Sub Min → IV m
Min
IMaj
m
IMin
7
VM
VII 0M
→ C : Maj
→ C : Min
→ G : Dom7
→ B: Dim
Simple, but enough for now, and easy to extend.
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
5 / 15
Now in Haskell—I
A GADT encoding musical harmony:
data Mode = MajMode | MinMode
data Piece = ∀µ :: Mode.Piece [ Phrase µ ]
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
6 / 15
Now in Haskell—I
A GADT encoding musical harmony:
data Mode = MajMode | MinMode
data Piece = ∀µ :: Mode.Piece [ Phrase µ ]
data Phrase (µ :: Mode) where
PhraseIVI :: Ton µ → Dom µ → Ton µ → Phrase µ
PhraseVI ::
Dom µ → Ton µ → Phrase µ
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
6 / 15
Now in Haskell—I
A GADT encoding musical harmony:
data Mode = MajMode | MinMode
data Piece = ∀µ :: Mode.Piece [ Phrase µ ]
data Phrase (µ :: Mode) where
PhraseIVI :: Ton µ → Dom µ → Ton µ → Phrase µ
PhraseVI ::
Dom µ → Ton µ → Phrase µ
data Ton (µ :: Mode) where
TonMaj :: SD I Maj → Ton MajMode
TonMin :: SD I Min → Ton MinMode
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
6 / 15
Now in Haskell—I
A GADT encoding musical harmony:
data Mode = MajMode | MinMode
data Piece = ∀µ :: Mode.Piece [ Phrase µ ]
data Phrase (µ :: Mode) where
PhraseIVI :: Ton µ → Dom µ → Ton µ → Phrase µ
PhraseVI ::
Dom µ → Ton µ → Phrase µ
data Ton (µ :: Mode) where
TonMaj :: SD I Maj → Ton MajMode
TonMin :: SD I Min → Ton MinMode
data Dom (µ :: Mode) where
Dom1 :: SD V Dom7 → Dom µ
Dom2 :: SD V Maj → Dom µ
Dom3 :: SD VII Dim → Dom µ
Dom4 :: SDom µ → Dom µ → Dom µ
Dom5 :: SD II Dom7 → SD V Dom7 → Dom µ
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
6 / 15
Now in Haskell—II
Scale degrees are the leaves of our hierarchical structure:
data DiatonicDegree = I | II | III | IV | V | VI | VII
data Quality
= Maj | Min | Dom7 | Dim
data SD (δ :: DiatonicDegree) (γ :: Quality ) where
SurfaceChord :: ChordDegree → SD δ γ
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
7 / 15
Generating harmony
Now that we have a datatype representing harmony sequences, how do
we generate a sequence of chords?
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
8 / 15
Generating harmony
Now that we have a datatype representing harmony sequences, how do
we generate a sequence of chords?
QuickCheck! We give Arbitrary instances for each of the datatypes in our
model.
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
8 / 15
Generating harmony
Now that we have a datatype representing harmony sequences, how do
we generate a sequence of chords?
QuickCheck! We give Arbitrary instances for each of the datatypes in our
model.
. . . but we don’t want to do this by hand, for every datatype, and to have
to adapt the instances every time we change the model. . . so we use
generic programming:
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
8 / 15
Generating harmony
Now that we have a datatype representing harmony sequences, how do
we generate a sequence of chords?
QuickCheck! We give Arbitrary instances for each of the datatypes in our
model.
. . . but we don’t want to do this by hand, for every datatype, and to have
to adapt the instances every time we change the model. . . so we use
generic programming:
gen :: (Representable α, Generate (Rep α))
⇒ Gen α
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
8 / 15
Generating harmony
Now that we have a datatype representing harmony sequences, how do
we generate a sequence of chords?
QuickCheck! We give Arbitrary instances for each of the datatypes in our
model.
. . . but we don’t want to do this by hand, for every datatype, and to have
to adapt the instances every time we change the model. . . so we use
generic programming:
gen :: (Representable α, Generate (Rep α))
⇒ [ (String ,Int) ] → Gen α
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
8 / 15
Examples of harmony generation—I
testGen
testGen
example
example
:: Gen (Phrase MajMode )
= gen [("Dom4", 3), ("Dom5", 4)]
:: IO ()
= let k = Key (Note \ C ) MajMode
in sample 0 testGen >
>= mapM (printOnKey k)
printOnKey :: Key → Phrase MajMode → IO String
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
9 / 15
Examples of harmony generation—I
testGen
testGen
example
example
:: Gen (Phrase MajMode )
= gen [("Dom4", 3), ("Dom5", 4)]
:: IO ()
= let k = Key (Note \ C ) MajMode
in sample 0 testGen >
>= mapM (printOnKey k)
printOnKey :: Key → Phrase MajMode → IO String
> example
[C : Maj, D: Dom7 , G : Dom7 , C : Maj ]
[C : Maj, G : Dom7 , C : Maj ]
[C : Maj, E : Min, F : Maj, G : Maj, C : Maj ]
[C : Maj, E : Min, F : Maj, D: Dom7 , G : Dom7 , C : Maj ]
[C : Maj, D: Min, E : Min, F : Maj, D: Dom7 , G : Dom7 , C : Maj ]
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
9 / 15
Examples of harmony generation—II
n œœœ
œ
œ
#œœœ
? œœœ
œœœ
œ
? œœœ
José Pedro Magalhães
nœœœ
œœœ
œœœ
œœœ
? œœœ
œœœ
œœœ
œœœ
? œœœ
? œœœ
œœœ
œœ
œ
n œœ
œœ
#œœœœ
œœœ
œœœ
#œœœœ
œœœ
n œœœ
œ
Functional Generation of Harmony and Melody, FARM 2014
œœœ
10 / 15
Generating a melody for a given harmony
We then generate a melody in 4 steps:
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
11 / 15
Generating a melody for a given harmony
We then generate a melody in 4 steps:
1. Generate a list of candidate melody notes per chord;
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
11 / 15
Generating a melody for a given harmony
We then generate a melody in 4 steps:
1. Generate a list of candidate melody notes per chord;
2. Refine the candidates by filtering out obviously bad candidates;
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
11 / 15
Generating a melody for a given harmony
We then generate a melody in 4 steps:
1. Generate a list of candidate melody notes per chord;
2. Refine the candidates by filtering out obviously bad candidates;
3. Pick one focal candidate melody note per chord;
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
11 / 15
Generating a melody for a given harmony
We then generate a melody in 4 steps:
1. Generate a list of candidate melody notes per chord;
2. Refine the candidates by filtering out obviously bad candidates;
3. Pick one focal candidate melody note per chord;
4. Embellish the candidate notes to produce a final melody.
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
11 / 15
Generating a melody for a given harmony
We then generate a melody in 4 steps:
1. Generate a list of candidate melody notes per chord;
2. Refine the candidates by filtering out obviously bad candidates;
3. Pick one focal candidate melody note per chord;
4. Embellish the candidate notes to produce a final melody.
These four steps combine naturally using plain monadic bind:
melody :: Key → State MyState Song
melody k = genCandidates >
>= refine >
>= pickOne >>= embellish
>
>= return ◦ Song k
More details in the paper!
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
11 / 15
Example I
œ œ œ œ œ œ œ œ œ œ œ œ œ œ œ œ œ œ
& œ œ
n œœœ
œœœ
œœœ
œ
? œœœ
#œœœœ
œœœ
Phrase
Ton
I : Maj
Ton
Dom
Sub
Dom
I : Maj
C : Maj III : Min IV : Maj II : Dom7 V : Dom7 C : Maj
E : Min
José Pedro Magalhães
F : Maj
D: Dom7 G : Dom7
Functional Generation of Harmony and Melody, FARM 2014
12 / 15
Example II
#
œ œœœœœœœœ
& œ œ œ œœœœœœ
œœ œ œ œ
œ
œœœ
œ
œœ
œœœ
? # œœ
œœœ
œ œ œ œ
œœœ
œœœ
œ
Phrase
Ton
I : Maj
Ton
Dom
Sub
G : Maj III : Min IV : Maj
B: Min
Sub
C : Maj III : Min IV : Maj
B: Min
José Pedro Magalhães
I : Maj
Dom
Dom
G : Maj
V : Maj
C : Maj D: Dom7
Functional Generation of Harmony and Melody, FARM 2014
13 / 15
Example III
&
#œ
œ
œ
? # œœœ
œ œ œ œ œ
œ
œœœ
œœœ
œ
œ
##œœœœ
œ
œ
#n œœœœ
œœœ
Phrase
Ton
I : Min
Dom
Sub
Ton
Dom
E : Min IV : Min
Sub
I : Min
Dom
E : Min
A: Min IV : Min II : Dom7 V : Dom7
A: Min F] : Dom7 B: Dom7
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
14 / 15
Conclusion
FComp: a simple and easy to understand and improve functional system
for automatic generation of harmony and accompanying melody.
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
15 / 15
Conclusion
FComp: a simple and easy to understand and improve functional system
for automatic generation of harmony and accompanying melody.
Lots of room for improvement:
I
Voice leading and counterpoint
I
I
Handle repetition
Improve embellishment
I
Rhythm, form, instrumentation, dynamics
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
15 / 15
Conclusion
FComp: a simple and easy to understand and improve functional system
for automatic generation of harmony and accompanying melody.
Lots of room for improvement:
I
Voice leading and counterpoint
I
I
Handle repetition
Improve embellishment
I
Rhythm, form, instrumentation, dynamics
Thank you for your attention!
José Pedro Magalhães
Functional Generation of Harmony and Melody, FARM 2014
15 / 15