IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING , VOL. 14, NO. 5, SEPTEMBER 2006
1845
Separation of Synchronous Pitched Notes
by Spectral Filtering of Harmonics
Mark R. Every and John E. Szymanski
Abstract—This paper discusses the separation of two or more simultaneously excited pitched notes from a mono sound file into separate tracks. In fact, this is an intermediate stage in the longer-term
goal of separating out at least two interweaving melodies of different sound sources from a mono file. The approach is essentially
to filter the set of harmonics of each note from the mixed spectrum
in each time frame of audio. A major consideration has been the
separation of overlapping harmonics, and three filter designs are
proposed for splitting a spectral peak into its constituent partials
given the rough frequency and amplitude estimates of each partial
contained within. The overall quality of separation has been good
for mixes of up to seven orchestral notes and has been confirmed
by measured average signal-to-residual ratios of around 10–20 dB.
Index Terms—Music note separation, partial extraction, separation of overlapping harmonics.
I. INTRODUCTION
T
HIS paper presents a data-driven approach, based upon an
analysis in the spectral domain, to separating multiple simultaneously excited pitched notes from a mono recording. An
attempt has been made to separate a mix of between two and
seven notes into the same number of tracks plus a residual. The
notes have approximately equal energies, are initially excited simultaneously and have almost the same duration. This research
is ultimately directed at separating a longer recording of an instrumental ensemble into its constituent instrumental parts or
melodic lines.
Potential applications of the aforesaid “mono-to-multitrack”
system are numerous. For example, classic recordings only
available in mono could be separated into individual instrumental parts, remastered track by track, and remixed again,
potentially even with new instruments. Alternatively, one
might want to remove a disturbing cough in a live recording,
and this might be achieved by separating the recording into
harmonic and residual components. Other applications exist in
the areas of effects processing, audio spatialization, restoration,
structured compression and coding, and music cataloguing and
retrieval.
The basic idea in this approach to the problem is that if the
pitch of each note in the mix is known in a particular time frame,
Manuscript received September 16, 2004; revised June 30,2005. This work
was carried out when the authors were with the Department of Electronics, University of York, York, U.K. The associate editor coordinating the review of this
manuscript and approving it for publication was Dr. Gerald Schuller.
M. R. Every is with CVSSP, SEPS, University of Surrey, Guildford, GU2
7XH, U.K. (e-mail: [email protected]; [email protected]).
J. E. Szymanski is with the Media Engineering Research Group, Department
of Electronics, University of York, Heslington, York, YO10 5DD, U.K. (e-mail:
[email protected]).
Digital Object Identifier 10.1109/TSA.2005.858528
then it is possible to identify the harmonics of each note in the
spectrum, and then to construct comb-like filters to filter the set
of harmonics of each note out of the composite spectrum. Pitchbased separation techniques have already been applied to speech
separation and enhancement, for example, in [1]–[3], and [4]
reviews a wide range of approaches to sound segregation related
to auditory scene analysis. In [1], vocalic speech was separated
from a mix of two competing talkers. There, crosstalk arising
from spectral peaks that were shared by the two speakers was
identified as a cause of degradation in the separated waveforms.
This issue becomes even more important when one is attempting
to separate out more than two pitched sources, since one would
expect many more partials to be overlapping. Furthermore, as
the occurrence of overlapping partials is far more common in
music than in speech due to the tendency to play harmonically
related notes together, the treatment of overlapping harmonics
is highly relevant to music source separation.
The full separation task consists of 1) detecting all salient
spectral peaks while the spectrum typically contains some lowlevel broadband energy due to noise and spectral leakage, 2)
estimating note pitch trajectories over all time frames using a
multipitch estimator, 3) matching spectral peaks to note harmonics, and 4) constructing filters to remove the individual note
spectra from the mixed spectrum. A multipitch estimator is used
to estimate the pitch trajectory of each note in the mix over all
time frames. Although accuracy in the former stages is essential
to achieving a realistic separation, we prefer here to emphasise
the filtering stage, and, in particular, the problem of overlapping
harmonics, which is felt is the area that has been least explored.
To clarify what class of sounds these algorithms have been
applied to, by the term “pitched,” it is implied that a note has
a perceivable pitch, and it contains most of its energy in harmonics that are at roughly integer multiples of the fundamental
frequency. The algorithms have been tested on bassoon, cello,
B clarinet, E clarinet, flute, French horn, oboe, piano, saxophone, trombone, and violin samples.
II. METHOD
To begin with, in each time frame the original mix of the
individual notes , is split into overlapping time frames and
a fast Fourier transform (FFT) is computed on the Hamming
, where is the window function. Time
windowed data,
samples at
frames of 186 ms in length (FFT length
a sampling rate
kHz) have been used with an 87.5%
indicates the value of the complex FFT specoverlap.
, and
is the
trum at frequency bin
corresponding frequency in Hertz.
1558-7916/$20.00 © 2006 IEEE
1846
IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING , VOL. 14, NO. 5, SEPTEMBER 2006
Fig. 1.
Thresholding and peak picking of the amplitude spectrum jF
A. Spectral Peak Identification
A reliable method for detecting spectral peaks was necessary
for multipitch estimation and to locate harmonics in the spectrum. Peak detection was performed successfully at frequencies
with a freup to the Nyquist limit, by thresholding
quency dependent threshold
, where is the shape
of the threshold and th is a frequency-independent threshold
height. Peak picking was then performed on the resulting thresholded spectrum.
The reason for using a variable threshold is that the typical
rolloff of harmonic amplitudes at higher frequencies often results in higher harmonics being too small to be detected by ap. These higher harmonics
plying a constant threshold to
are, however, very helpful for pitch estimation and are perceptually significant.
was arrived at in the following manner. The smoothed
with
amplitude envelope was calculated by convolving
a normalized Hamming window of length
samples.
An odd-numbered window length was chosen for symmetry
involves a weighted sum
reasons, i.e., the calculation of
, at an equal number of bins on either side
of terms
of bin . An alternative method for calculating the spectral
envelope is the regularized calculation of the cepstrum coefficients [5]. The Hamming windowing method was preferred due
to its computational efficiency, effectiveness and the fact that
the latter method involves the calculation of a matrix inversion
which was sometimes found to be numerically unstable. Then,
up to the Nyquist limit, where
we define
was used for the
a suitable range for is [0.5, 1), and
results given here. Smaller values of produce a flatter envelope, and this helps to avoid spurious peaks being detected in
regions of low spectral amplitude. To satisfy scaling invariance,
. Fig. 1 shows the amplitude spectrum
j
using a frequency-dependent threshold e
1
E^ (k).
and threshold for a mix of two violin notes with pitches A5
(880 Hz) and E6 (1319 Hz).
Next, a search was made to find all local maxima in
above the threshold. A frequency bin was considered to be a
peak maximum if
(1)
, where
is in the range (0,1],
is the length of vector , and we have chosen
.
This peak picking algorithm incorporates the simplest case,
, of checking whether the amplitude in each discrete Fourier
transform (DFT) bin is larger than only its nearest neighbors, but
can also be adapted to more noisy spectra by assigning a longer
vector to . The algorithm is not computationally expensive and
is easy to implement, although a systematic comparison has not
yet been made with other methods for peak-picking, such as
sinusoidal modeling of the DFT spectrum [6].
Once the spectral peaks had been identified, a refinement
was made
of the center frequency of each DFT maximum
to subfrequency bin resolution using a DFT frequency interpolator. At the same time, the peak amplitudes were refined
using an amplitude interpolator. A number of interpolation
methods were examined [7]: Quinn’s first and second interpolator, Grandke’s interpolator, the quadratic interpolator, the
baricentric interpolator and the DFT method implemented in
the software package InSpect [8]. The accuracy of the various
DFT interpolators are dependent on the type of windowing
applied to the data. It was found that the DFT and Grandke’s
method were both suitable frequency and amplitude interpolation methods for Hamming windowed data, although both
methods are in fact more accurate for Hanning windowed data.
The DFT method involves the calculation of two FFTs in each
time frame, and is not, hence, as computationally efficient as
EVERY AND SZYMANSKI: SEPARATION OF SYNCHRONOUS PITCHED NOTES
1847
Fig. 2. Estimated pitch trajectories of two synchronous flute notes played with vibrato using the multipitch estimator. The reference lines show the transcribed
note pitches (G5 = 784 Hz and A5 = 880 Hz).
Grandke’s method. However, the DFT performed marginally
better than Grandke’s method in preliminary tests measuring
the frequency and amplitude interpolation errors for a sinusoid
to 20 dB
in white noise, in which the SNR was varied from
and, hence, was chosen as the preferred interpolator. Another
method for estimating peak frequencies [9] follows from minimizing, in a least-squares sense, the difference between the
observed spectrum and the first-order limited expansion of the
Fourier transform of the window function around each peak.
B. Multipitch Estimation
Following spectral peak detection and given a priori, the
number of notes in the mix, a multipitch estimator was designed
and used to estimate the pitch trajectories of all notes in the mix.
As there is typically some variation in pitch over the note duration, for example, due to vibrato, and since the filtering stage is
sensitive to slight pitch variations, it was necessary to estimate
of all
notes in every time
the pitches
frame. Pitch estimates in individual frames were then combined
in such as way as to form smooth note pitch trajectories while
ignoring isolated and clearly incorrect pitch estimates. Fig. 2
shows the results of using the multipitch estimator to estimate
the pitch trajectories of two flute notes played with vibrato. The
multipitch estimator, while being moderately reliable for 2–3
synchronous notes, was markedly worse at higher polyphonies
as discussed in Section IV, and its implementation will not be
expanded upon since better note error rates have been reported
for another multipitch estimator [10].
C. Estimating the Harmonic Frequencies
Once the pitch trajectories of all notes were calculated, then,
in a particular time frame, each detected spectral peak could potentially be matched with any single note that contained a harmonic within a limited range about the peak center frequency.
A match was not made when more than one harmonic from different notes happened to exist within this range. In this case, we
refer to these as overlapping harmonics, and since their amplitude spectra are likely to be significantly overlapping, the resulting spectral content shared by the harmonics will be called
an overlapping peak. To clarify, although, in Fig. 3, two peaks
were detected in the peak detection stage, the term “overlapping
peak” refers to the entire peak shared by both harmonics. On the
other hand, a “nonoverlapping peak” is a spectral peak that is
within range of at most one harmonic. Overlapping peaks will
be discussed in Section II-E. Following the matching process
and an adequate treatment of overlapping peaks, a filter was designed for each note, whose effect when multiplied by the DFT
spectrum, was to remove from the spectrum the peaks that were
matched uniquely to harmonics of that note, and a portion of the
energy in any overlapping peaks that the unmatched harmonics
of this note may have contributed to. The result of filtering the
composite spectrum with this set of filters was a segmentation
of the mixed spectrum into several constituent spectra corresponding to each note, and a residual basically containing the
low-level noise envelope of the mixed spectrum and any inharmonic partials. For the present being, we consider nonoverlapping spectral peaks.
A spectral peak was matched to a note if its frequency
was within a range
of any
, where
is
the frequency of the th harmonic of note . Typically, was
chosen to be in the range [0.01, 0.1]. If more than one peak was
, the largest peak was matched
found within this range of
with note and the others ignored. An identity was not used
in the above expression for the harmonic frequencies due to the
following reasons. First, the deviation of harmonic frequencies
from exact harmonicity can be quite significant, especially in
piano notes which will be discussed shortly. Second, any inacwould be compounded when mulcuracy in a pitch estimate
tiplying by
to find the th harmonic frequency. Last, separation results were improved by using nonrigid estimates of
the harmonic frequencies, and this is believed to be partly due
to the better frequency localization of constituent harmonics in
overlapping spectral peaks. The procedure for extrapolating eswas, first, to determine the frequency of the
timates of the
fundamental frequency component of note . If there existed
was set
a nonoverlapping peak within range of , then
1848
IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING , VOL. 14, NO. 5, SEPTEMBER 2006
its partial frequencies are stretched rather more substantially
than in most Western instruments according to [11]
(3)
This comes about by physical consideration of the piano string
stiffness in the equation of motion for transverse waves in a vibrating bar. is the inharmonicity coefficient, and, for a typin the middle register, the 17th parical value of
tial would be shifted to about the frequency of the 18th par. An iteratial had the note been purely harmonic (
tive method was tested for producing better estimates of the
in (2), based upon (3). A predicpredicted frequencies
tion can be made by forming a joint estimate of
using
dea least-squares error minimization of (3) over the first
, and then substituting and
back
tected partials
. This improved tracking mainly of higher
into (3) to find
piano partials, which become progressively further apart in frequency as increases. The technique was not used, in general,
however, because of the increase in computation time arising
. It predictably yielded results of
from the estimation of
around zero when applied to other pitched orchestral instruments.
Fig. 3. Filtering of a spectral peak arising from two overlapping harmonics: a)
construction of the filters H (k ) using (5) and (6) is determined by the predicted
harmonic frequencies f , and f , and predicted harmonic amplitudes A and
A ; b) comparison of the filtered and original amplitude spectra of the individual
harmonics.
to ; otherwise, it was set to . The remaining harmonic frequencies of note were calculated iteratively according to (2)
a nonoverlap peak
(2)
otherwise
is the predicted frequency of the
where
th harmonic of note . As (2) relies on knowing whether the
peak is nonoverlapping, and this, in turn, depends on whether
multiple harmonics of separate notes have been predicted in
the local vicinity of this peak, this iterative process had to be
performed concurrently for all pitches. The synchronization of
these iterative processes was determined by always applying the
next iteration of (2) to the note corresponding to the minimum
. To clarify,
means the
of the set
th harmonic of note . To begin with,
, and
is incremented with each iteration for note .
We found that, as exact harmonicity was not enforced above,
i.e., the predicted harmonic frequencies were allowed to shift
slightly to coincide with nonoverlapping peak frequencies, the
above method was flexible enough to detect partials slightly detuned from exact harmonicity. However, in the case of the piano,
D. Filtering of Nonoverlapping Harmonics
The width of a nonoverlapping spectral peak , centered at the
frequency bin
, was found by searching
at frequency bins
for the first minima in
and
on opposite sides of the main lobe. If the peak
was
was matched with note , the amplitude of the filter
set to unity across the entire width of the peak,
.
Appreciably better results were achieved using these variable
width filter notches at the peak frequencies, rather than using
fixed width filter notches. Thus, in the resynthesis stage, when
is multiplied by the DFT spectrum, this has the effect of
filtering the entire main lobe of the peak from the original mixed
spectrum.
is real, it follows that the DFT spectrum is
Since
complex conjugate symmetric about the Nyquist frequency,
. Thus, frequency components
i.e.,
above the Nyquist limit are easily removed from the mixed
. However, it is only
spectrum by using
actually exists.
in (10) that an imaginary component in
An advantage of the approach used here as opposed to models
in which separated harmonics are synthesized using well-behaved sinusoids [10], [12]–[15] is that, since the amplitude of
the filter notches is unity across the width of all the nonoverlapping peaks matched to harmonics, then the residual contains
at most some traces of the detected harmonics due to sidelobes,
which do not tend to be noticeable. This also holds for overlapping peaks due to a normalization (5), which will be discussed in
Section II-E. In the case of sinusoidal models, if the harmonics
are not well modeled by sinusoids with slowly time-varying amplitude and frequency, and the residual is calculated by subtracting the set of sinusoids from the original waveform, then
there could be some leakage of the harmonics into the residual.
EVERY AND SZYMANSKI: SEPARATION OF SYNCHRONOUS PITCHED NOTES
E. Filtering of Overlapping Harmonics
Previous approaches to separating overlapping partials in
both speech and music fields include those that rely on sinusoidal models [1], [10], [12]–[15], a perceptually motivated
smoothing of the amplitude spectrum of each source [12], linear
models for overtone amplitudes [15], spatial mixing models
[16], and a multistrategy approach [17]. Of the techniques
based upon a sinusoidal model, [1] iteratively subtracts larger
amplitude partials to reveal partially hidden weaker partials,
[12], [15] iteratively estimates the phases and amplitudes of
closely spaced sinusoids alternately with frequency estimates,
and [10], [13], [14] use pre-estimated sinusoidal frequency
estimates to calculate amplitude and phase estimates. In [1],
when two overlapping partials were closely spaced and of comparable amplitude, a linear amplitude interpolation between
neighboring harmonics was used to share the peak between the
two vocal sources. Amplitude modulation or beating resulting
from closely spaced sinusoids was used in [10] to resolve the
amplitude trajectories of closely spaced sinusoids. A multistrategy approach was employed in [17] for separating duet
signals. Beating was exploited if two partials were separated
by less than 25 Hz, and the duration of the overlap was longer
than two beat periods. When the overlap was shorter than two
periods, an amplitude interpolation method like the one in [1]
was used. When the partials were separated by between 25 and
50 Hz, a linear equations method was used to determine the
amplitudes of the two partials given the measured composite
spectrum, and partials separated by more than 50 Hz were
not considered to be overlapping. In [14], when the frequency
difference between intersecting sinusoids was less than 25 Hz,
a linear amplitude and cubic phase multiframe interpolation
method was used to interpolate the sinusoids between boundary
frames at which the amplitudes and phases of the individual
sinusoids could be resolved. Finally, a method was developed
in [16] for resolving overlapping partials across multiple time
frames, which combined spatial demixing techniques with
inference based on the fact that neighboring harmonics of a
single note usually have common amplitude and frequency
modulation characteristics. The method effectively estimates
in the frequency regions
frequency masks similar to
where overlaps occur. The technique applies to additive mixing
models in which
microphones in a room record
different
mixtures of the sources, where, in general,
.
Here, when dealing with overlapping harmonics, filters
were designed to split the spectral content shared by
overlapping harmonics into
parts using
overlapping
filters. Three filter designs are proposed for this purpose; the
first two are alternative methods for partitioning the energy in a
shared spectral peak, and the third uses a model of the sum of
DFT spectra to recover the DFTs of the individual harmonics.
The filters designs were all dependent on the extrapolated
harmonic frequencies
, and it was also found beneficial to
include some dependency on the predicted amplitudes
of
the
harmonics. The
were predicted using a simple linear
interpolation between the amplitudes of the nearest harmonics
of each pitch that were matched to nonoverlapping peaks.
1849
The first energy-based filter design for separating overlapping
harmonics is
(4)
where
is the frequency in Hertz of bin , and
are the first minima
in
above and below the set of predicted harmonic frenotes that contain a
quencies. is the set of the particular
harmonic within the overlapping peak, and a suitable value for
is
. For appearance,
, i.e., the index of the harmonic of pitch that constitutes a part of the overlapping peak,
has been shortened to . (4) is followed by a normalization to
obtain
(5)
The second energy-based filter design introduces a dependency on the Fourier transform of the continuous window function
(6)
and this is, again, normalized using (5) to obtain
. In
practice, we approximate the continuous spectrum
by
the DFT of the zero-padded window function (a zero-padding
factor of 64 has been used) and
is rounded to the
describes the shape
nearest equivalent frequency bin.
of the window function in the spectral domain, which is a maximum at
and decreases as increases. Fig. 3(a) illusobtained using (5) and (6)
trates the shape of the filters
applied to a spectral peak comprised of two harmonics. Fig. 3(b)
compares the filtered amplitude spectra using these filters, with
the amplitude spectra of the original unmixed harmonics. This
is, evidently, a good separation of the overlapping peak into its
constituent harmonics.
The above two filter designs, (4) and (6), were proposed
simply as way of splitting the energy in an overlapping peak
into
parts in a way that reflects the predictions of the amplitudes and frequencies of the constituent harmonics. In ideal
conditions in which these predictions are exact, and the peak
arises from stationary sinusoids of nearly equal frequency and
pre-estimated phase offset, then it is possible to separate the
overlapping peak almost exactly into its constituent parts using
complex filters. Suppose we use the following signal model to
describe a cluster of frequencies
giving rise to an
overlapping peak in the DFT
(7)
where
at the start of the current frame and the signal is
assumed to be continuous in the range
. Then,
it can be shown that the Fourier transform of the continuous
1850
IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING , VOL. 14, NO. 5, SEPTEMBER 2006
Fig. 4. Filtering of an overlapping spectral peak arising from two windowed sinusoids: a) mixed peak is the sum of the peaks arising from the individual sinusoids;
b) resulting peak corresponding to the first sinusoid after filtering [filtered a, b, and c used (4), (6), and (10), respectively]; c) as in b), but for the second sinusoid.
signal multiplied by the continuous window function , is a
convolution of the individual Fourier transforms and results in
Assuming the model is accurate and apart from an arbitrary
constant,
is approximately equal to
evaluated at
the discrete frequency bins. For the moment, we are observing
in a limited frequency range between the minima on opposite sides of the positive frequency peak, to which the second
term in brackets in (8) has very little effect. Thus
across
only approximately accurate, and as
the width of the peak, any leakage of the peak into the residual
is avoided.
Equation (10) is, in practice, applied to our set of
overlapping harmonics by using the substitution
. However, we still need a way
to predict
. Suppose we measure
at
different frequency bins:
, with equivalent frequencies in Hertz:
which are chosen to be the nearest frequency
bins to
under the condition that
.
Then, we obtain a set of independent linear equations which
can be solved by a matrix inversion to find
(9)
(11)
(8)
where
Suppose that all the
possible to design a filter
and
are known. Then, it is
..
.
..
.
(10)
that, when multiplied by
, results in approximately the DFT
of the windowed sinusoid . One could correctly argue that
given
and , it would be easier to compute the expected
shape of the DFT of the windowed sinusoid and simply subtract it from the overlapping peak in
. However, any slight
error in these parameters will result in an imperfect subtraction, and leakage of this sinusoid into the residual. A similar
effect will occur if the original signal model was at all inaccurate, for instance, due to the sinusoid being only approximately
stationary during the frame. The filter design of (10) performs
a reasonably good separation when the parameter estimates are
..
.
..
.
Therefore
(12)
is approximated by the DFT of the zeroOnce again,
padded window function.
Fig. 4 compares the result of filtering a spectral peak arising
from two synthesized overlapping sinusoids, using the filters in
EVERY AND SZYMANSKI: SEPARATION OF SYNCHRONOUS PITCHED NOTES
Fig. 5.
Hz).
1851
Filter shapes calculated using (5) and (6) and the amplitude spectrum of three violins with pitches (A5 = 880 Hz), (D[6 = 1109 Hz), and (E 6 = 1319
(4), (6), and (10). Whereas, in Fig. 3, the DFT amplitude was
shown, in Fig. 4, the real component of the DFT spectrum is
shown to illustrate that (10) was much better than (4) and (6) at
resolving the phases of the original sinusoids correctly. A similar observation was made for the imaginary spectra. In Fig. 4,
small random errors were added to the known sinusoidal amplitudes and frequencies in (4), (6), and (10) to simulate normal
conditions of operation in which these quantities would be estimated imperfectly from the data.
Finally, Fig. 5 shows the mixed amplitude spectrum of three
violin notes in a single time frame, and Fig. 6 illustrates its separation using (5) and (6) into three source spectra plus a residual
spectrum.
having a random amplitude in the range [0, 1], and a random
, were added together to simuphase offset in the range
late a random overlapping peak. The robustness of the three filter
or
was then evaldesigns as a function of the error in
uated, by substituting into (4), (6), and (10) either the correct
amplitude of both sinusoids and a rough estimate of their frequencies, or vice versa. In the former case, the rough estimates
were produced by adding to each known sinusoidal freof
. In the
quency, a random frequency in the range
were produced by mullatter case, the rough estimates of
tiplying each known sinusoidal amplitude by a random number
. We used as a measure of the error
in the range
between the original and separated DFTs of the two sinusoids,
the quantity
F. Resynthesis of Separated Notes
In any particular frame, the filtered spectrum for note was
by
. The time waveform of
obtained by multiplying
each separated note was synthesized by performing an inverse
FFT of the corresponding filtered spectrum, then dividing by the
original Hamming window used in the analysis, and then using
an overlap-add method with triangular windows to interpolate
the resulting time segments between frames. The residual time
waveform was produced by subtracting the sum of the separated
note waveforms from the original time signal.
III. FILTER PERFORMANCE
To evaluate the relative performance of the filters, they were
applied in turn to the task of separating two overlapping sinusoids, with results shown in Fig. 7. The measurements were
made as follows: Two sinusoids with a random relative frequency difference in the range [0, 4] frequency bins, and each
(13)
is the
where is the DFT of the windowed mixed sinusoids,
is the filter
DFT of the windowed unmixed sinusoid , and
for sinusoid . Fig. 7 shows the average value of
over
iterations for each value of , as is varied from 0 to 1. It is clear
that the separation performances of all filters decrease when the
frequency and amplitude estimates decrease in accuracy, i.e.,
increases. The results reveal that when the frequency and amplitude estimates are accurate, the complex filter (10) is the most
precise, but as these estimates become more inaccurate, eventually, (10) becomes misleading and less robust than (4), and (6)
and (10) appear to have an advantage only when errors in
are less than 0.2 frequency bins. Equation (4) demonstrates the
most stable behavior and is the most accurate method when the
errors in the frequency estimates are large.
1852
IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING , VOL. 14, NO. 5, SEPTEMBER 2006
Fig. 6. Segmentation of the spectrum in Fig. 5 using the filters shown, into three harmonic sources and a residual (note different amplitude scales).
based upon perceptual judgement, since the key aim of the
algorithms is to achieve perceptually acceptable separation
of the sources. Hence, some sound examples have been presented on the internet [18] so that the separated sounds can
be compared directly with the unmixed and mixed originals.
We also use a quantifiable measure, the signal-to-residual ratio
(SRR), to evaluate the similarity between the time waveform
, and its corresponding original
of each separated note
dB
Fig. 7. Filter error R(r ) when separating two overlapping sinusoids, as a
function of the inaccuracy in a) the sinusoidal amplitude estimates A and b)
the sinusoidal frequency estimates f [filtered a, b, and c correspond to (4),
(6), and (10), respectively].
IV. RESULTS
The real sound examples used were all western orchestral
instrument note samples of length 2–8 s, 16 bits and sampled
at 44.1 kHz. All samples aside from the piano were recorded
in an anechoic chamber, although it has been observed that
adding small amounts of reverb diminishes the separation
quality only slightly. These samples were scaled to have equal
mean squared amplitude, and summed to produce mixed note
samples on which the separation algorithms were applied,
allowing the direct comparison of the separated sounds and
the original recordings. The most meaningful way to evaluate the performance of the separation algorithms would be
(14)
The correct matching of the set of original notes to the set of
separated notes was achieved by swapping the order of the separated notes until the maximum of
was achieved
for each .
sources is
The mean signal-to-residual ratio (MSRR) over
defined as
(15)
and the average increase in the sum of SRRs is
(16)
is the mixed original signal. These quanwhere
tities measure how well the original sound has been separated
into individual notes, with larger values indicating better separation performance. No attempt has been made yet to split the
residual waveform any further, and so it is expected that for
mixes of notes containing large nonharmonic components, the
should decrease.
MSRR and
EVERY AND SZYMANSKI: SEPARATION OF SYNCHRONOUS PITCHED NOTES
1853
TABLE I
SRRs AND =M , FOR THE SEPARATION OF 2–7 SYNCHRONOUS VIOLIN NOTES
assess the performance of the filters in ideal conditions. The
audio samples corresponding to Tables I and Table II are available at [18]. Fig. 8 shows an example of the original, separated
and residual spectrograms obtained when separating a mix of
three notes (sample number in Table II).
C. Average Separation Results
A. Separating Harmonically Unrelated Notes
for sample
Table I shows the calculated SRRs and
mixes of between two and seven violin notes, using (4) and
(5) to separate overlapping harmonics. Although the separation
performances shown are very good, one must take into consideration that these examples mostly consist of notes that are not
related to each other by harmonic intervals like major thirds,
fourths, fifths or octaves. The results should, thus, be interpreted
as showing that the removal of nonoverlapping harmonics with
unit amplitude filters across the width of harmonic peaks is
capable of producing high SRRs.
B. Separating Harmonically Related Notes
To measure the effectiveness of the three filter designs proposed for separating overlapping harmonics, three real sample
mixes were produced consisting of multiple notes from different
instruments, in which the pitches were deliberately chosen to
have harmonic relationships, thus resulting in a higher proportion of overlapping harmonics. Three synthesized sound examples were also produced and the results of the six test samples
are shown in Table II.
is given for the three
For each sample, the MSRR and
filter designs in Section II-E and also using 1) no treatment of
overlapping harmonics, 2) Parsons’ method for splitting overlapping harmonics by interpolating between the amplitudes of
neighboring harmonics [1], and 3) the nonlinear least-squares
(NLS) method for estimating parameters of a model of closely
spaced sinusoids [13]. For Parsons’ method, a filter
was used, followed by the normalization in (5). In [13], the
NLS method is explained in the context of two overlapping harmonics, but the method can easily be generalized to more than
two overlapping harmonics. Nonoverlapping harmonics were
treated identically for all methods as described in Section II-D.
The first synthetic sample is a mix of three synthesized notes
and
Hz. The second synthesized
with
sample is a sum of two notes: the first note has a constant pitch
of 440 Hz and the second is a linear glissando between 400 and
480 Hz. In the last synthetic sample, the first note has a constant
pitch of 300 Hz, and the second note has a frequency-modulated
(FM) pitch centered on 450 Hz with FM amplitude 10 Hz and
FM frequency 5 Hz to simulate vibrato. In all the synthetic examples, the first 20 harmonics are present with decreasing har, and the exact time-varying harmonic amplitudes
monic frequency and amplitude trajectories were provided to
Finally, results are presented in Table III for the MSRR and
for polyphonies of 2–5 instruments, each averaged over
100 random sample mixes, and using the same set of methods
for separating overlapping harmonics as in Table II. Random
mixes were constructed by firstly selecting a random set of
unique instruments from a set of 11 orchestral instrument types
(bassoon, cello, B clarinet, E clarinet, flute, French horn, oboe,
piano, saxophone, trombone, and violin), and then selecting a
note for each instrument randomly from within its complete
pitch range. The individual notes were drawn from a set of 479
samples extending in pitch from A0 (27.5 Hz) to C8 (4186 Hz).
In initial studies, the multipitch estimator was used. It was
able to detect all pitches correctly in a random mix 53.2%,
37.7%, 17.4%, and 4.0% of the time for polyphonies of 2, 3, 4,
and 5, respectively, where a correct pitch estimate is defined as
one that is within 3%, i.e., half a semi-tone, of the known transcribed pitch. The mixes for which multipitch estimation was
unsuccessful tended to occur when there were strong harmonic
relationships between the constituent notes, and for these mixes
lower SRRs would be expected due to the greater likelihood
of overlapping harmonics. It was found that by averaging
SRRs over only those sample mixes for which the multipitch
estimator was correct, the average results were biased to “easy
mixtures,” and resulted in SRRs of approximately 5 dB higher
than those given in Table III. Thus, it was decided that instead,
the rough pitches would be provided in advance, so the results
in Table III show the average performance over all random
mixes. The multipitch estimator was used only to refine and
track each pitch over time within a limited pitch range around
the provided rough pitch.
V. DISCUSSION
The violin separation in Table I showed that by multiplying a
mixed spectrum by a filter of unit amplitude across the width of
each peak containing a single harmonic, these harmonics could
be extracted very effectively. This was done with relatively little
computational expense in comparison to spectral subtraction
methods involving sinusoidal parameter estimation.
Table II provides some insight into the performances of the
three filter designs in comparison to previous approaches to separating overlapping harmonics. In relation to Table I, one notices
an overall decrease in performance, which is not surprising due
to the relatively larger proportion of overlapping harmonics in
Table II.
In Table II, the two energy-based filter designs produced
overall the highest SRRs with real samples (samples {4}–{6}).
They also performed very well for the synthesized samples
(samples 1–3). Equation (10) was the best performer for sample
{1}. As the harmonic frequency and amplitude trajectories
were provided for the synthesized samples, this indicates that
1854
IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING , VOL. 14, NO. 5, SEPTEMBER 2006
TABLE II
SRRs AND =M , FOR THE SEPARATION OF HARMONICALLY RELATED MIXED NOTES
under ideal conditions of stationary sinusoids, (10) performs
the best, which is confirmed by observing Fig. 7 at low values
of . The relatively low performance of Parsons’ method can be
explained by the lack of frequency dependence in the filter design. Nevertheless, it provides some advantage over removing
overlapping harmonics from the separated notes altogether (the
first column of data in Table II). The low performance of the
NLS method is surprising, and upon further examination of the
separated spectra some insight was obtained. The low SRRs are
mostly due to a few instances in which the amplitudes of the
closely spaced sinusoids modeling an overlapping peak were
grossly overestimated. The explanation could be that there is
very little preventing a least-squares method from interpreting
a small overlapping peak as the addition of two sinusoids of
very large amplitude but nearly opposite phase, which destructively interfere. Although this interpretation may correspond
to a minimum least-squares error, given that the spectrum is
unlikely to be composed of stationary sinusoids, the small
overlapping peak would more likely be the addition of two
relatively low amplitude harmonics. The addition of a term into
the NLS estimation that penalizes joint amplitude estimates
with large variance might partially overcome this problem. As
the frequency and amplitude estimates of the closely spaced
sinusoids need to be optimized to find the least-squares fit, the
computational cost of the NLS method is relatively large. There
is also a risk that this optimization does not converge to the
global best fit.
The worst overall SRRs in Table II were obtained for sample
{5}, which corresponds with a general observation that separation performance is worse for lower pitched samples. This is an
unavoidable consequence of using a fixed frequency resolution
transform, the DFT. It means that harmonic frequency estimates
are worse relative to for lower pitched notes. Also, the ratio
of the width of harmonics to the spacing between harmonics becomes larger for lower pitches; hence, there is relatively more
overlapping spectral content for lower pitched note mixes.
Table III again validates the use of frequency-dependent filters by showing a consistent improvement in SRR over Par-
EVERY AND SZYMANSKI: SEPARATION OF SYNCHRONOUS PITCHED NOTES
1855
Fig. 8. Original spectrograms of a cello, soprano saxophone, and flute, and the spectrograms of the corresponding notes after separation (gray scales of all figures
are equivalent).
TABLE III
AVERAGE MSRR AND =M , FOR POLYPHONIES OF 2–5 INSTRUMENTS
of (10) is demonstrated in Fig. 7; notably, (10) is not very robust
to large errors in harmonic frequency and amplitude estimates.
Overall, not only is (4) the most predictable with respect to errors in harmonic amplitude and frequency estimates as shown
in Fig. 7, but Table III shows that it also achieves the highest
SRRs.
The SRRs in Table III for (4) are about 7 dB higher than the
average separation results reported in [15]. In [15], a larger selection of 26 different instruments was used although pitches
were restricted to between 65 and 2100 Hz, and the results reported were the average over clean mixes and mixes with addi-dB pink noise. The cases in which the multipitch estive
timator failed were not accounted for in the average separation
results in [15].
VI. CONCLUSION
sons’ method for the three proposed filter designs. Similarly
to in Table II, the two energy-based filter designs, (4) and (6),
achieved the highest SRRs. The NLS method and (10) were both
derived by assuming that the sinusoids in an overlapping peak
are stationary within each time frame. Given that these methods
performed worse than (4) and (6), and the fact that relatively
long window lengths (186 ms) were used, it would be reasonable to conclude that this assumption is inaccurate for most real
samples. Another contributing factor to the lower performance
Results have been presented for separating mixes of between
two and seven synchronous notes from a mono track. Average
SRRs of around 10–20 dB have been achieved, and mixes of
two notes were separated with almost imperceptible differences
between the separated and original notes. In some sample mixes,
SRRs of up to 30 dB were obtained. With increasing numbers
of notes in the mix, the separation quality predictably decreases,
but mixes of up to seven notes have been separated with enough
fidelity to easily allow the listener to match the separated and
original notes correctly. This work has been extended [19] to
separating synchronous note sequences or instrumental parts.
1856
IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING , VOL. 14, NO. 5, SEPTEMBER 2006
A product of the separation is the residual, which contains
the inharmonic partials and noise characteristics of the original
mix. This has been used elsewhere to make audible subtle attack
characteristics of piano notes, and may have application in the
synthesis of realistic instrument sounds. No attempt has been
made to split the residual into separate sources, and, thus, there
are audible artifacts when attempting to separate notes with significant noise or nonharmonic content.
A solution has been proposed to the problem of separating
overlapping harmonics from different notes, which is an important issue when attempting to separate musical sources. Three
filter designs have been developed for separating multiple harmonics from an overlapping spectral peak. The first two designs
are alternative methods for splitting the energy in the overlapping peak by using predictions of the frequencies and amplitudes of each harmonic constituting the peak. The third filter
design is complex and attempts to recover the DFTs of the individual unmixed harmonics. All of these filter designs resulted
in overall improvements to separation performance, and the first
energy-based filter design, (4) was overall the best performer.
The work has confirmed that via the use of a priori
model-based information, in the form of both prespecified
models of harmonic structures of pitched instruments and a
filtering methodology to handle overlapping spectral peaks,
it is possible to separate the harmonic structure of multiple
instruments from a mono recording to a high fidelity.
ACKNOWLEDGMENT
[8] M. Desainte-Catherine and S. Marchand, “High precision fourier analysis of sounds using signal derivatives,” J. Audio Eng. Soc., vol. 48, no.
7/8, pp. 654–667, Jul./Aug. 2000.
[9] P. Depalle and T. Hélie, “Extraction of spectral peak parameters using
a short-time fourier transform modeling and no sidelobe windows,”
presented at the IEEE Workshop Applications of Signal Processing to
Audio and Acoustics, New Paltz, NY, Oct. 1997.
[10] A. Klapuri, T. Virtanen, and J.-M. Holm, “Robust multipitch estimation
for the analysis and manipulation of polyphonic musical signals,” presented at the COST-G6 Conf. Digital Audio Effects, Verona, Italy, Dec.
2000.
[11] N. H. Fletcher and T. D. Rossing, The Physics of Musical Instruments,
2nd ed. New York: Springer-Verlag, 1998.
[12] T. Virtanen and A. Klapuri, “Separation of harmonic sounds using multipitch analysis and iterative parameter estimation,” presented at the IEEE
Workshop Applications of Signal Processing to Audio and Acoustics,
New Paltz, NY, Oct. 2001.
[13] T. Tolonen, “Methods for separation of harmonic sound sources using
sinusoidal modeling,” presented at the AES 106th Convention, Munich,
Germany, May 1999.
[14] T. F. Quatieri and R. G. Danisewicz, “An approach to co-channel talker
interference suppression using a sinusoidal model for speech,” IEEE
Trans. Acoust., Speech, Signal Process., vol. 38, no. 1, pp. 56–69, Jan.
1990.
[15] T. Virtanen and A. Klapuri, “Separation of harmonic sounds using linear
models for the overtone series,” presented at the IEEE Int. Conf. Acoustics, Speech, Signal Processing, Orlando, FL, May 2002.
[16] H. Viste and G. Evangelista, “A method for separation of overlapping
partials based on similarity of temporal envelopes in multi-channel mixtures,” in IEEE Trans. Audio, Speech, Lang. Process., to be published.
[17] R. C. Maher, “Evaluation of a method for separating digitized duet signals,” J. Audio Eng. Soc., vol. 38, no. 12, pp. 956–979, Dec. 1990.
[18] Note Separation Demonstrations, M. R. Every and J. E. Szymanski.
(2004, Jun.). [Online]. Available: http://www-users.york.ac.uk/~jes1/
Separation1.html
[19] M. R. Every and J. E. Szymanski, “A spectral-filtering approach to music
signal separation,” presented at the 7th Int. Conf. Digital Audio Effects,
Naples, Italy, Oct. 2004.
The authors would like to thank the three anonymous reviewers for their well informed and helpful suggestions.
Mark R. Every received an Honours degree in
physics from the University of the Witwatersrand,
South Africa, in 1999. He is currently pursuing the
M.S. degree in music technology at the University
of York, York, U.K., on a British Commonwealth
Scholarship.
He is currently an academic fellow at the Centre
for Vision, Speech, and Signal Processing, University
of Surrey, Guildford, U.K. His current research interests include music and audio signal processing, content description and extraction, and machine learning
REFERENCES
[1] T. W. Parsons, “Separation of speech from interfering speech by means
of harmonic selection,” J. Acoust. Soc. Amer., vol. 60, no. 4, pp. 911–918,
Oct. 1976.
[2] G. Hu and D. L. Wang, “Monaural speech segregation based on pitch
tracking and amplitude modulation,” IEEE Trans. Neural Netw., vol. 15,
no. 5, pp. 1135–1150, Sep. 2004.
[3] L. Ottaviani and D. Rocchesso, “Separation of speech signal from complex auditory scenes,” in Proc. COST G-6 Conf. Digital Audio Effects,
Limerick, Ireland, Dec. 2001, pp. 87–90.
[4] M. Cooke and D.P.W. Ellis, “The auditory organization of speech and
other sources in listeners and computational models,” Speech Commun.,
vol. 35, no. 3–4, pp. 141–177, 2001.
[5] O. Cappé, J. Laroche, and E. Moulines, “Regularized estimation of
cepstrum envelope from discrete frequency points,” presented at the
Workshop Applications of Signal Processing to Audio and Acoustics,
WASPAA, Mohonk, NY, Oct. 1995.
[6] X. Rodet. Musical sound signal analysis/synthesis: Sinusoidal residual
and elementary waveform models. presented at IEEE Time-Frequency and Time-Scale Workshop. [Online]. Available: http://mediatheque.ircam.fr/articles/textes/Rodet97e/
[7] (Revised 1999, May). How to Interpolate Frequency Peaks.
dspGuru, Iowegian Intern. Corp., M. Donadio. [Online]. Available: http://www.dspguru.com/howto/tech/peakfft2.htm
+
techniques.
John E. Szymanski received an Honours degree
in mathematics and the Ph.D. degree in theoretical
physics from the University of York, York, U.K., in
1980 and 1984, respectively.
He joined the Department of Electronics, University of York, in 1986, where he is currently a Senior
Lecturer within the Media Engineering Group. His
research interests include physical modelling, computational signal processing, inverse problems, and
optimization methods.