Method and system for separating musical sound source

This application claims the benefit of Korean Patent Application No. 10-2009-0080684, filed on Aug. 28, 2009, and No. 10-2009-0122217, filed on Dec. 10, 2009, in the Korean Intellectual Property Office, the disclosures of which are incorporated herein by reference.

1. Field of the Invention

Embodiments of the present invention relate to a method of separating a musical sound source, and more particularly, to an apparatus and method of separating a musical sound source, which may re-construct mixed signals into target sound sources and other sound sources directly using sound source information performed using a predetermined musical instrument when the sound source information is present, thereby more effectively separating sound sources included in the mixed signal.

2. Description of the Related Art

Along with developments in audio technologies, a method of separating a predetermined sound source from a mixed signal where various sound sources are recorded has been developed.

However, in a conventional method of separating sound sources, the sound sources may be separated utilizing statistical characteristics of the sound sources based on a model of an environment where signals are mixed and thus, only mixed signals having a same number of sound sources to be separated as a number of sound sources in the model may be applicable.

Accordingly, there is a need for a method of separating a predetermined sound source from commercial musical signals that usually have a number of sound sources greater than that of the mixed signals when obtaining only one or two mixed signals.

An aspect of the present invention provides an apparatus of separating a musical sound source, which may re-construct mixed signals into target sound sources and other sound sources directly using sound source information performed using a predetermined musical instrument when the sound source information is present, thereby more effectively separating sound sources included in the mixed signal.

According to an aspect of the present invention, there is provided an apparatus of separating musical sound sources, the apparatus including: a Nonnegative Matrix Partial Co-Factorization (NMPCF) analysis unit to perform an NMPCF analysis on a mixed signal and a predetermined sound source signal using a sound source separation model, and to obtain a plurality of entity matrices based on the analysis result; and a target instrument signal separating unit to separate, from the mixed signal, a target instrument signal corresponding to the predetermined sound source signal by calculating an inner product between the plurality of entity matrices.

In this instance, the plurality of entity matrices obtained by the NMPCF analysis unit may include a frequency domain characteristic matrix U of the predetermined sound source signal, a location and intensity matrix Z in which U is expressed in a time domain of the predetermined sound source signal, a location and intensity matrix V in which U is expressed in a time domain of the mixed signal, a frequency domain characteristic matrix W of remaining sound sources included in the mixed signal, and a location and intensity matrix Y in which W is expressed in the time domain of the mixed signal.

Also, the NMPCF analysis unit may determine the predetermined sound source signal as a product of U and Z, and determine the mixed signal as a product of ½ of U and V summed with a product of ½ a weight of W and Y to thereby obtain the plurality of entity matrices U, Z, V, W, and Y.

Also, the apparatus may further include a time-frequency domain conversion unit to receive the mixed signal and the predetermined sound source signal of a time domain, to convert the received mixed signal and predetermined sound source signal of the time domain into the mixed signal and the predetermined sound source signal of a time-frequency domain to transmit the converted signals to the NMPCF analysis unit, and to extract phase information from the received mixed signal and predetermined sound source signal of the time domain, and a time domain signal conversion unit to convert the target instrument signal into a time domain signal using the phase information, and to separate, from the mixed signal, the sounds performed using the predetermined musical instrument.

According to another aspect of the present invention, there is provided a method of separating musical sound sources, the method including: converting a mixed signal and a predetermined sound source signal of a time domain into a mixed signal and a predetermined sound source signal of a time-frequency domain; extracting phase information from the mixed signal and the predetermined sound source signal of the time domain; performing an NMPCF analysis on the mixed signal and the predetermined sound source signal of the time-frequency domain using a sound source separation model; obtaining a plurality of entity matrices based on the NMPCF analysis result; separating, from the mixed signal, a target instrument signal corresponding to the predetermined sound source signal by calculating an inner product between the plurality of entity matrices; and separating, from the mixed signal, sounds performed using a predetermined musical instrument by converting the target instrument signal into a time-domain signal using the phase information.

Additional aspects, features, and/or advantages of the invention will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the invention.

According to embodiments of the present invention, there is provided an apparatus of separating a musical sound source, which may re-construct mixed signals into target sound sources and other sound sources directly using sound source information performed using a predetermined musical instrument when the sound source information is present, thereby more effectively separating sound sources included in the mixed signal.

Also, according to embodiments of the present invention, there is provided an apparatus of separating a musical sound source which may separate a desired sound source from a single mixed signal and thus, may be applicable in separating commercial musical sounds obtaining only two mixed signals or less.

These and/or other aspects, features, and advantages of the invention will become apparent and more readily appreciated from the following description of exemplary embodiments, taken in conjunction with the accompanying drawings of which:

FIG. 1 illustrates an example of an apparatus of separating a musical sound source according to an embodiment of the present invention;

FIG. 2 is a flowchart illustrating a method of separating a musical sound source according to an embodiment of the present invention;

FIG. 3 illustrates an example of an apparatus of separating a musical sound source according to another embodiment of the present invention; and

FIG. 4 is a flowchart illustrating a method of separating a musical sound source according to another embodiment of the present invention.

Reference will now be made in detail to exemplary embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. Exemplary embodiments are described below to explain the present invention by referring to the figures.

FIG. 1 illustrates an example of an apparatus of separating a musical sound source according to an embodiment of the present invention.

The apparatus includes a database 110 , a time-frequency domain conversion unit 120 , a Nonnegative Matrix Partial Co-Factorization (NMPCF) analysis unit 130 , a target instrument signal separating unit 140 , and a time domain signal conversion unit 150 .

The database 110 may store information about a solo performance using a predetermined musical instrument, and transmit the information about the solo performance as a type of a predetermined sound source signal x 1 .

In this instance, the predetermined sound source may have a significantly great amount of data to include various characteristics of the predetermined sound source. In this case, a great amount of database signals may need to be processed for each sound source separation operation.

Accordingly, as for the predetermined sound source, a scheme of more effectively compressing database signals converted into a time domain or a time-frequency domain may be used. In this instance, the compression scheme may have a condition such that characteristics required for the separation of the predetermined sound source are maintained even after performing the compression scheme, which is different from a general audio compression scheme.

The time-frequency domain conversion unit 120 may receive the predetermined sound source signal x 1 of the time domain transmitted from the database 110 and a mixed signal x 2 of the time domain inputted from a user, and convert the received sound source signal x 1 and mixed signal x 2 into a sound source signal X 1 and mixed signal X 2 of a time-frequency domain. In this instance, the mixed signal may be a musical signal where performances of various musical instruments or voices are mixed.

Also, the time-frequency domain conversion unit 120 may extract phase information Φ 2 from the received predetermined sound source signal x 1 and mixed signal x 2 .

In this instance, the time-frequency domain conversion unit 120 may transmit the sound source signal X 1 and the mixed signal X 2 to the NMPCF analysis unit 130 , and transmit the phase information Φ 2 to the time domain signal conversion unit 150 .

The NMPCF analysis unit 130 may perform an NMPCF analysis on the mixed signal and the predetermined sound source signal using a sound source separation model, and obtain a plurality of entity matrices based on the analysis result.

In this instance, the NMPCF analysis unit 130 may determine, as a signal satisfying Equation 1 below, X (1) and X (2) , that is, a magnitude of the sound source signal X 1 and the mixed signal X 2 , and arbitrary frequency domain characteristic matrices U and W, location and intensity matrices Z, V, and Y in which U and W are expressed in a time domain may be obtained based on the following Equation 1. In this instance, X (1) and X (2) may be a matrix X (1) n×m 2 and a matrix X (2) n×m 2 , respectively.

In this instance, U, Z, V, W, and Y may be expressed as entity matrices U n×p 2 , Z m 2 ×p 2 , V m 2 ×p 2 , W n×p 2 , and Y m 2 ×p 2 , respectively, and may be non-negative real numbers. Also, U may be included in both of X (1) and X (2) and thus, may be shared.

Specifically, under an assumption that X (1) is obtained through a relationship between U and Z, the NMPCF analysis unit 130 may determine input signals as a product of frequency domain characteristics such as pitch, tone, and the like and time domain characteristics indicating an intensity the input signals are performed at in a predetermined time location.

Also, since a product U×V T of entity matrices included in X (2) shares the frequency domain characteristic matrix U identical to that used in X (1) , the NMPCF analysis unit 130 may determine a manner in which a frequency domain characteristic of a target sound source to be separated is included in X (2) .

Also, the NMPCF analysis unit 130 may define entity matrices W and Y regardless of information stored in the database 110 , and thereby may simultaneously perform a modeling of a state where remaining sound sources other than the target sound source comprise the mixed signal.

That is, X (2) may be comprised of a sum of a relationship of entity matrices expressing the target sound source signals to be separated and a relationship of entity matrices expressing remaining sound source signals.

The NMPCF analysis unit 130 may derive and use an optimized target function, as illustrated in the following Equation 2, based on Equation 1.

In this instance, a weight λ of Equation 2 may be a weight between a second section for restoring sounds performed using a predetermined musical instrument and a first section for the mixed signal.

Also, the NMPCF analysis unit 130 may update U, Z, V, W, and Y by applying U, Z, V, W, and Y to the following Equation 3 in accordance with an NMPCF algorithm.

That is, the NMPCF analysis unit 130 may initialize U, Z, V, W, and Y to be non-negative real numbers in accordance with the NMPCF algorithm, and repeatedly update U, Z, V, W, and Y until approaching a predetermined value based on Equation 3.

In this instance, a multiplicative characteristic of Equation 3 may not change signs of elements included in the entity matrices.

The target instrument signal separating unit 140 may separate, from the mixed signal, a target instrument signal corresponding to the predetermined sound source signal by calculating an inner product between the entity matrices obtained by the NMPCF analysis unit 130 . In this instance, the target instrument signal may be a signal including the sounds performed using the predetermined musical instrument from among the mixed signal X 2 .

Specifically, the target instrument signal separating unit 140 may separate the target instrument signal included in the mixed signal X 2 by calculating an inner product between U and V, and convert the separated target instrument signal into an approximation signal UV T expressed in a magnitude unit of a time-frequency domain.

The time domain signal conversion unit 150 may convert the target instrument signal into a signal of the time domain using the phase information Φ 2 extracted by the time-frequency domain conversion unit 120 .

Specifically, the time domain signal conversion unit 150 may convert UV T into the time-domain signal using the phase information Φ 2 to thereby obtain an approximation signal s of the target instrument signal.

FIG. 2 is a flowchart illustrating a method of separating a musical sound source according to an embodiment of the present invention.

In operation S 210 , the time-frequency domain conversion unit 120 may receive a mixed signal and predetermined sound source signal of a time domain, and convert the received mixed signal and predetermined sound source signal of the time domain into a mixed signal and predetermined sound source signal of a time-frequency domain to thereby extract phase information from the received mixed signal of the time domain.

In operation S 220 , the NMPCF analysis unit 130 may perform, using a sound source separation model, an NMPCF analysis on the mixed signal and predetermined sound source signal converted in operation S 210 to thereby obtain entity matrices.

Specifically, the NMPCF analysis unit 130 may obtain, based on Equation 1, a frequency domain characteristic matrix U of the predetermined sound source signal, a location and intensity matrix Z in which U is expressed in a time domain of the predetermined sound source signal, a location and intensity matrix V in which U is expressed in a time domain of the mixed signal, a frequency domain characteristic matrix W of remaining sound sources included in the mixed signal, and a location and intensity matrix Y in which W is expressed in the time domain of the mixed signal, and update U, Z, V, W, and Y based on Equation 3.

In operation S 230 , the target instrument signal separating unit 140 may separate, from the mixed signal, a target instrument signal corresponding to the predetermined sound source signal by calculating an inner product between the entity matrices obtained in operation S 220 .

In operation S 240 , the time domain signal conversion unit 150 may convert, using the phase information extracted in operation S 210 , the target instrument signal separated in operation S 230 into a signal of a time domain to thereby obtain an approximation signal of the target instrument signal.

FIG. 3 illustrates an example of an apparatus of separating a musical sound source according to another embodiment of the present invention.

The apparatus according to the other embodiment may be used to overcome complexity in calculation and difficulties in an aspect of utilization of a memory, which are generated when the NMPCF analysis unit 130 receives a large amount of single sound source information as the sound source signal X 1 of the time-frequency domain, and may be an example of reducing an amount of data while maintaining characteristics of database storing information about a solo performance using a predetermined musical instrument.

The apparatus according to the other embodiment includes, as illustrated in FIG. 3 , a database 110 , a database signal compression unit 310 , a time-frequency domain conversion unit 120 , a time-frequency domain signal compression unit 320 , an NMPCF analysis unit 330 , a target instrument signal separating unit 140 , and a time domain signal conversion unit 150 . The apparatus may compress a predetermined sound source signal, and perform an NMPCF analysis on the compressed predetermined sound source signal.

In this instance, the database 110 , the time-frequency domain conversion unit 120 , the target instrument signal separating unit 140 , and the time domain signal conversion unit 150 may have the same configurations as those of FIG. 1 and thus, further descriptions thereof will be omitted.

The database signal compression unit 310 may compress a predetermined sound source signal of a time domain transmitted from the database 110 .

For example, the database signal compression unit 310 may extract only sounds performed by percussion instruments from predetermined sound source signals of a time domain including only signals of the percussion instruments while disregarding remaining sounds other than the percussion sounds, thereby extracting only relevant parts of the database.

The time-frequency domain signal compression unit 320 may compress the predetermined sound source signal that is converted into the time-frequency domain in the time-frequency domain conversion unit 120 .

For example, the time-frequency domain signal compression unit 320 may perform a Nonnegative Matrix Factorization (NMF) analysis on the predetermined sound source signal of the time-frequency domain, and thereby a database signal of a time-frequency domain may be expressed as a product of a base vector matrix X 1 ′ and a weight matrix. Also, the time-frequency domain signal compression unit 320 may transmit, to the NMPCF analysis unit, only the base vector matrix X 1 ′ as the compressed database signal.

Also, the database signal compression unit 310 and the time-frequency domain signal compression unit 320 may be complementarily operated.

The NMPCF analysis unit 320 may perform an NMPCF analysis on the mixed signal and the base vector matrix using the sound source separation model to thereby obtain a plurality of entity matrices based on the analysis result.

Specifically, the NMPCF analysis unit 320 may obtain U, Z, V, W, and Y using the base vector matrix X 1 ′ extracted by the time-frequency domain signal compression unit 320 instead of the sound source signal X 1 .

FIG. 4 is a flowchart illustrating a method of separating a musical sound source according to another embodiment of the present invention.

In operation S 410 , the database signal compression unit 310 may compress a predetermined sound source signal of a time domain transmitted from the database 110 to thereby transmit the compressed signal to the time-frequency domain conversion unit 120 .

In operation S 420 , the time-frequency domain conversion unit 120 may receive a mixed signal of a time domain and the predetermined sound source signal compressed in operation S 410 , convert the received predetermined sound source signal and mixed signal into a mixed signal and predetermined sound source signal of a time-frequency domain, and extract phase information from the received mixed signal and predetermined sound source signal of the time domain.

In operation S 430 , the time-frequency domain signal compression unit 320 may perform an NMF analysis on the predetermined sound source signal of the time-frequency domain converted in operation S 420 to thereby extract a base vector matrix.

In operation S 440 , the NMPCF analysis unit 320 may perform an NMPCF analysis on the mixed signal converted in operation S 420 and the base vector matrix extracted in operation S 430 to thereby obtain entity matrices.

Specifically, the NMPCF analysis unit 320 may obtain, based on Equation 1, a frequency domain characteristic matrix U of the predetermined sound source signal, a location and intensity matrix Z in which U is expressed in a time domain of the predetermined sound source signal, a location and intensity matrix V in which U is expressed in a time domain of the mixed signal, a frequency domain characteristic matrix W of remaining sound sources included in the mixed signal, and a location and intensity matrix Y in which W is expressed in the time domain of the mixed signal, and update U, Z, V, W, and Y based on Equation 3.

In operation S 450 , the target instrument signal separating unit 140 may separate a target instrument signal corresponding to the predetermined sound source signal from the mixed signal by calculating an inner product between the entity matrices obtained in operation S 440 .

In operation S 460 , the time domain signal conversion unit may convert, using the phase information extracted in operation S 420 , the target instrument signal separated in operation S 450 into a signal of a time domain to thereby obtain an approximation signal of the target instrument signal.

As described above, according to embodiments of the present invention, there is provided an apparatus of separating a musical sound source, which may re-construct mixed signals into target sound sources and other sound sources directly using sound source information performed using a predetermined musical instrument when the sound source information is present, thereby more effectively separating sound sources included in the mixed signal.

Also, according to embodiments of the present invention, there is provided an apparatus of separating a musical sound source which may separate a desired sound source from a single mixed signal and thus, may be applicable in separating commercial musical sounds obtaining only one or two mixed signals.

Also, there is no need for entire processes of inputting a separator for separately extracting characteristics of the target sound source signal and characteristics of the segmented mixed signal, and there is no need for learning the separator.

Although a few exemplary embodiments of the present invention have been shown and described, the present invention is not limited to the described exemplary embodiments. Instead, it would be appreciated by those skilled in the art that changes may be made to these exemplary embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.