Articulation Rules For Automatic Music Performance
Roberto Bresin
Department of Speech, Music and Hearing
Royal Institute of Technology, KTH, Stockholm
email: [email protected]
Abstract
Rules for legato, staccato, and repeated notes articulation
in automatic piano performance are presented. These rules
were obtained applying the analysis-by-measurement
method to performances of Mozart piano sonatas on
MIDIfied acoustic grand pianos. The effects produced by
the rules vary with tempo indications, with expressive
intentions and with musical context. The articulation rules
are part of the Director Musices system for automatic music
performance. It has been demonstrated that proper setting
of the articulation rules enables more realistic automatic
performances that can also differ in emotional expression.
These rules are also being applied in control models for
sound synthesis algorithms.
1
Introduction
Articulation is one of the most important cues in music
performance, since it allows control of the way in which two
consecutively played notes are acoustically connected, and
contributes to the character of the performance. Much effort
is spent on the study of articulation in musical training. In
the past, few researchers have paid attention to the analysis
of articulation in piano performance. This could be due to
two main reasons. First, it is difficult to detect the instant
when a key is released and when the associated sound has
passed the threshold of audibility. Second, a precise
measurement of the mechanical movements of piano keys
and hammers is possible only in commercial MIDIfied
pianos like Disklavier and Bösendorfer, or in pianos
provided with various sensors. Examples of the latter are
photocells, as used by Shaffer (1981), and accelerometers
on the hammers and the keys, as used by Askenfelt and
Jansson (1990). Mathews (1975) observed that tone overlap
was required in order to produce a legato effect in tones
generated by electroacoustic means. His observation was
later corroborated by Purbrick (2000) who pointed out the
difficulty in producing expressive performances with
computer-controlled synthesizers, and proposed an
automatic generation of legato articulation in a guitar sound
generated with a physical model-based synthesis.
In investigations of articulation in both digital and
acoustic piano playing, Repp examined both perception and
production of legato and staccato articulation. He found that
an acoustic overlap was required to produce a legato, while
a micropause was needed to produce a staccato (Repp 1995,
1997, 1998). Palmer (1989) reported that in legato
articulation the inter-onset interval (IOI) between two
overlapping notes is a major factor for the amount of
overlap. Gabrielsson and Juslin pointed out how
articulation, with its variations, is one of the most important
and effective cues in communication and perception of
motional-emotional character in music performance
(Bengtsson and Gabrielsson 1983; Gabrielsson 1994, 1995;
Gabrielsson and Juslin 1996).
In the work presented in this paper a further step has
been taken toward understanding the practice of articulation.
It is a preliminary model for the rendering of articulation
strategies in automatic piano performance.
2
Material and Method
Two performance databases were used in this study. One
consisted of performances by five diploma students, at the
Venice Music Conservatory, who played the Andante
movement of W A Mozart’s Piano Sonata in G major, KV
545. They were asked to play the piece in nine different
performance styles on a Disklavier connected to a PC. The
styles were given in terms of adjectives (glittering, dark,
heavy, light, hard, soft, passionate, flat, and natural, the
latter meaning the style preferred by the pianist). The other
database consisted of recordings of thirteen Mozart’s piano
sonatas played by a professional pianist on a Bösendorfer
SE290 computer–monitored concert grand piano. Gerhard
Widmer used the same database in research presented in
these proceedings (Widmer 2001).
The data available in the two databases allowed analysis
of articulation based on the movement of the piano keys and
not on the acoustic realization. Apart from the IOI, two
parameters have been used here for describing the
articulation in piano performance. The first is the key
overlap time (KOT), i.e., the time during which the keys
corresponding to two successive notes are depressed
simultaneously. The second parameter is the key detached
time (KDT), defined as the time during with neither of two
keys corresponding to two successive notes is depressed,
such that there is a micropause between the tones (Figure 1).
3.1 Score legato articulation rule
DM Lisp function name: Score-legato-art.
Description: this rule produces an overlap of tones, or
legato. The pseudocode of the rule is presented below.
Affected sound parameter: key overlap time, KOT [ms].
Usage and limitations: the rule can be used to control the
quantity of legato articulation. It is applied to notes that
are marked legato in the score, as suggested by the name
of the rule. Groups of legato notes are marked in the score
file with the Lisp commands (LEGATO-START T) and
(LEGATO-END T). (See the DM documentation for
information about the score format and the Lisp
commands available).
Pseudocode for the Score Legato Articulation rule:
Figure 1. (a) Definition of inter-onset interval (IOIn),
duration (DRn) and key overlap time (KOTn) for TONEn
followed by an overlapping TONEn+1. (b) Definition of
inter-onset interval (IOIn), duration (DRn) and key
detached time (KDTn) for TONEn followed by a nonoverlapping TONEn+1.
3
Articulation Rules
In previous research by Bresin and Battel (2000) and
Bresin and Widmer (2000), measurements of the
performances stored in the above-mentioned databases were
statistically analyzed. From these analyses the following
main results emerged; (1) KOT in legato notes is dependent
from IOI of the first of two overlapping notes, (2) KDT in
staccato notes is independent from IOI, and (3) KDT in
repeated notes is generally dependent from IOI. These
results led to the definition of a new set of rules for
automatic articulation in expressive piano music. The new
rules are four. They are included in Director Musices (DM),
a rule system for music performance written in Common
Lisp language (Friberg, et al. 2000). All the articulation
rules contribute with different effects according to
expressive indications (first database), tempo indications
(second database; adagio, andante, allegro, presto, and
menuetto), and articulation marks written in the score. The
<Score legato articulation> rule is applied to notes marked
legato. The <Score staccato articulation> rule is applied to
notes marked staccato. The <Articulation of repetition> rule
inserts a micropause between two consecutive tones with
the same pitch. The <Duration contrast articulation> rule
can be used to control the type of articulation, ranging from
staccato to legato, of notes that are marked neither legato
nor staccato. In the next paragraphs a description of these
rules is provided. It follows the format used in the DM
documentation (for more information see the list of links at
paragraph 5).
if 1 < K <= 5
then KOT ← ((0.5·10-6·K - 0.1110-3)·IOI+ 0.01105·K +
+ 0.16063)·IOI
else if 0 < K <= 1
then KOT ← ((-4.3·10-6·K - 6.6·10-6)·IOI +
+ 58.533·10-3·K + 113.15·10-3)·IOI
where K is a weighting parameter determining the
magnitude of KOT.
The K values can be associated with the different
playing styles corresponding to the adjectives used for the
experiment described in a recent work by Bresin and Battel
(2000):
K=5
K=1
K = 0.1
==>
==>
==>
passionate legato
natural legato
flat legato
It means that by varying the emphasis parameter K from
0.1 to 5 it will be possible to sweep through a variety of
legato styles.
3.2 Score staccato articulation rule
DM Lisp function name: Score-staccato-art.
Description: this rule introduces a micropause after a
staccato tone. The pseudocode of the rule is presented
below.
Affected sound parameter: key detached time, KDT [ms].
Usage and limitations: the rule can be used to control the
quantity of staccato articulation. It is applied to notes
marked staccato in the score, as suggested by the name of
the rule. Staccato is marked in the score with the Lisp
command (STACCATO T). An extra parameter, Tempoindication, can be used to achieve different quantities of
staccato for different tempo indications. The DM
command line for the Score Staccato Articulation rule is
therefore:
Score-staccato-art <K> :Tempo-indication <tempo>
where tempo is the user input value for the Tempoindication variable.
Pseudocode for the Score Staccato Articulation rule:
if 1 < K <= 5
then KDT ← (0.0216·K + 0.643) ·IOI
else if 0 < K <= 1
KDT ← (0.458·K + 0.207) ·IOI
KDT ← pitch-contour · context · Tempo-indication · KDT
where K is a weighting parameter determining the
magnitude of KDT, pitch-contour, context and Tempoindication are three variables realizing the effects due to
pitch contour, staccato context and tempo indication as
described by Bresin and Widmer (2000). In particular they
found that (1) KDT is larger for repeated notes and smaller
for ascending pitch contours, (2) KDT is larger for isolated
staccato notes and smaller for notes in a sequence of
staccato notes, (3) KDT is smaller for slower tempi. The K
values are associated with the different playing styles given
below, corresponding to the adjectives used in a previous
experiment on expressive intentions described by Bresin and
Battel (2000):
K=5
K=3
K=1
K = 0.6
K = 0.5
K = 0.1
==>
==>
==>
==>
==>
==>
default staccatissimo (KDT = 0.75·IOI)
light staccato
natural staccato (KDT = 0.5·IOI)
default staccato
heavy staccato
default mezzostaccato (KDT = 0.25·IOI)
Default value for both variables pitch-contour and
context is 1. Their values can be automatically modified by
DM, according to the findings by Bresin and Widmer
(2000), shortly described above. The Tempo-indication
values are associated with the different tempi given below
(Bresin and Widmer 2000):
Tempo-indication = 1.3
Tempo-indication = 1.15
Tempo-indication = 1
==>
==>
==>
Presto and Menuetto
Allegro
Adagio and Andante
3.3 Articulation of repetition rule
DM Lisp function name: Repetition-art.
Description: the rule inserts a micropause between two
consecutive tones with same pitch. The pseudocode of the
rule is presented below.
Affected sound parameter: key detached time, KDT [ms].
Usage and limitations: the rule inserts a micropause
between two consecutive tones with the same pitch. An
expressive parameter Expr can be used to achieve two
different kinds of articulation, one with constant KDT, the
other with KDT dependent on IOI. The DM command
line for the Articulation of Repetition rule is therefore:
Repetition-art <K> :Expr <kdt-type>
where kdt-type is the user input value for the Expr
variable.
Pseudocode for the Repetition of Articulation rule:
if Expr = constant-kdt
then KDT← 20·K;
else if Expr = varying-kdt
if K > 1
then KDT← ((-46·10-6·IOI – 23.67·10-3) ·K –
– 878·10-6·IOI + 0.98164) ·IOI
else if K <= 1
then KDT← ((- 532·10-6·IOI + 0.3592) ·K –
248·10-6·IOI + 0.3578) ·IOI
where K is a weighting parameter determining the
magnitude of the rule effect.
The Expr and K values are associated with the different
playing styles given below corresponding to the adjectives
used for the experiment reported by Bresin and Battel
(2000), shortly described above:
if Expr = constant-kdt then:
K=1
==> natural
K = 0.7 ==> heavy
if Expr = varying-kdt then:
K=5
==> dark
K=4
==> soft
K=2
==> passionate
K=1
==> glittering
K = 0.5 ==> flat and light
K = 0.1 ==> hard
3.4 Duration contrast articulation rule
The first version of this rule was presented in Bresin and
Friberg (1998). The current, slightly modified version is
described here.
DM Lisp function name: Duration-contrast-art.
Description: the rule inserts a micropause between two consecutive tones if the first note is a short one, i.e., if it has
duration between 30 and 600 milliseconds (see Table 1).
Affected sound parameter: offset-to-onset duration, DRO
[ms].
Usage and limitations: this rule can be used for the purpose
of articulation, as suggested by its name. It can also be
inverted, in the sense that it produces overlap of tones, or
legato. Thus, the rule can be used to control the type of
articulation, ranging from staccato to legato. It applies to
notes which are marked neither legato nor staccato in the
Table 1. Relation between tone duration (DR, in ms)
and offset-to-onset duration (DRO, in ms) according
to the Duration Contrast Articulation rule.
DR
DRO
< 30
200
400
> 600
0
-16.5
-10.5
0
score, as such notes are processed by the Score Legato
Articulation and the Score Staccato Articulation rules.
The rule is not applied to the first tone of tone repetitions.
4
Conclusions
The effect of these rules has not yet been tested with a
listening test. Nevertheless it is equal to or larger than the
just noticeable quantity necessary for perceiving legato or
staccato tones according to findings by Woody (1997). The
analysis-by-synthesis method confirmed the importance of
the articulation rules to the improvement of the quality in
automatically generated performances. The articulation
rules were successfully used in the production of
emotionally expressive performances of different scores,
which where classified as happy or sad in accordance with
hypotheses by Juslin, Friberg and Bresin (in press). The
articulation rules are being applied in the design of control
models for sound synthesis (Bresin, R., Friberg, and Dahl
submitted), the main idea is to provide a more natural and
realistic control to synthesis algorithms (Dannenberg and
Derenyi 1998).
The rules will be further developed in the near future. In
order to achieve a more complete model of articulation, the
acoustic quantity corresponding to key overlap and key
detached times in legato and staccato respectively is under
investigation.
5
Links
The art of music performance:
http://www.speech.kth.se/music/performance
Articulation rules and sound examples:
http://www.speech.kth.se/music/performance/articulation
The Director Musices program:
http://www.speech.kth.se/music/performance/download/
6
Acknowledgments
This work was supported by the European Union (SOb The Sounding Object project – no. IST-2000-25287;
http://www.soundobject.org), and by the Bank of Sweden
Tercentenary Foundation (FEEL-ME - Feedback Learning
in Musical Expression - Dnr 2000 - 5193:01;
http://www.psyk.uu.se/forskning/projekt.html#pj). This research was also funded in part by the START programme
Y99-INF, financed by the Austrian Federal Ministry for
Education, Science, and Culture (http://www.ai.univie.ac.at/
/oefai/ml/music/musicproject.html).
References
Askenfelt, A. and E. Jansson. 1990. “From touch to string
vibrations. I: Timing in the grand piano action.” Journal of
Acoustical Society of America, 88(1), 52-63
Bengtsson, I. and A. Gabrielsson. 1983. “Analysis and synthesis of
musical rhythm.” In J. Sundberg (Ed.), Studies of music
performance, Stockholm: Publications issued by the Royal
Swedish Academy of Music, 39:27-59
Bresin, R., A. Friberg, and S. Dahl. Submitted. “Toward a new
model for sound control”
Bresin, R., and G.U. Battel. 2000. “Articulation strategies in
expressive piano performance. Analysis of legato, staccato,
and repeated notes in performances of the Andante movement
of Mozart's sonata in G major (K 545).” Journal of New Music
Research, 29 (2000), No 3, 211-224
Bresin, R. and A. Friberg. 2000. “Emotional Coloring of
Computer-Controlled Music Performances.” Computer Music
Journal, 24:4, 44-63
Bresin, R., and G. Widmer. 2000. “Production of staccato
articulation in Mozart sonatas played on a grand piano.
Preliminary results.” Speech Music and Hearing Quarterly
Progress and Status Report, Stockholm: KTH, 2000(4):1-6
Dannenberg, R., and I. Derenyi. 1998. “Combining Instrument and
Performance Models for high-quality Music Synthesis.”
Journal of New Music Research, 27(3):211-238
Friberg, A., V. Colombo, L. Frydén, and J. Sundberg. 2000.
“Generating Musical Performances with Director Musices.”
Computer Music Journal, 24(3):23-29
Gabrielsson, A. 1994. “Intention and emotional expression in
music performance.” In A. Friberg, J. Iwarsson, E. Jansson &
J. Sundberg (Eds.) Proceedings of the Stockholm Music
Acoustics Conference 1993, Stockholm: Royal Swedish
Academy of Music, 108-111
Gabrielsson, A. 1995. “Expressive intention and performance.” In
R. Steinberg (Ed.) Music and the Mind Machine: the
Psychophysiology and the Psychopathology of the Sense of
Music. Berlin, Heidelberg, New York: Springer Verlag, 35-47
Gabrielsson, A., and P.N. Juslin. 1996. “Emotional expression in
music performance: between the performer's intention and the
listener's experience.” Psychology of Music, 24:68-91
Juslin, P. N., A. Friberg, and R. Bresin. In press. “Toward a
computational model of expression in performance: The
GERM model.” Musicae Scientiae.
Mathews, M.V. 1975. “How to make a slur.” Journal of Acoustical
Society of America, Vol. 58 (S1), S132
Palmer, C. 1989. “Mapping musical thought to musical
performance.” Journal of Experimental Psychology: Human
Perception and Performance, 15:331-346
Purbrick, J. 2000. “Automatic Synthesizer Articulation.” Computer
Music Journal, 24(1):20-31
Repp, B. 1995. “Acoustics, perception, and production of legato
articulation on a digital piano.” Journal of Acoustical Society
of America, 97(6):3862-3874
Repp, B. 1997. “Acoustics, perception, and production of legato
articulation on a computer-controlled grand piano.” Journal of
Acoustical Society of America, 102(3):1878-1890
Repp, B. 1998. “Perception and Production of Staccato articulation
on the Piano.” Unpublished manuscript, Haskins Laboratories,
http://www.haskins.yale.edu/haskins/STAFF/repp.html
Shaffer, L. 1981. “Performances of Chopin, Bach and Bartok:
studies in motor programming.” Cognitive Psychology 13:326276
Widmer, G. 2001. “Inductive learning of general and robust local
expression principles.” In Proceedings of the International
Computer Music Conference 2001, ICMA: San Francisco
Woody, R.H. 1997. “Perceptibility of changes in piano tone
articulation.” Psychomusicology, 16:102-109