Acoustic signal separation apparatus, learning apparatus, method, and program thereof

This application is a U.S. 371 Application of International Patent Application No. PCT/JP2019/019833, filed on 20 May 2019, which application claims priority to and the benefit of JP Application No. 2018-109327, filed on 7 Jun. 2018, the disclosures of which are hereby incorporated herein by reference in their entireties.

The present invention relates to a technique for separating an acoustic signal, and particularly relates to a technique for separating an acoustic signal based on a difference in the distance from a sound source to a microphone.

Acoustic signal separation is a method for separating an acoustic signal based on a difference in some signal characteristic between a target sound and noise. A typical acoustic signal separation method includes a method in which separation is performed based on a difference in tone quality (DNN (Deep Neural Network) sound source enhancement or the like) (see, e.g., NPL 1 or the like), and a method in which separation is performed based on a difference in the direction of a sound (an intelligent microphone or the like).

Non Patent Literature

[NPL 1] Yuma Koizumi, “A Research on the Design of Statistical Objective Functions for Estimating Acoustic Information using Deep Learning”, The University of Electro-Communications, Graduate school of Informatics and Engineering, September 2017

Technical Problem

In order to separate the acoustic signal based on the difference in the distance from the sound source to the microphone, it is necessary to obtain “spatial information” of a sound field elaborately. In order to obtain the spatial information, a large number of microphones are usually required. In this case, as in the conventional DNN sound source enhancement, when an acoustic feature value of an observed signal obtained by each microphone is used as learning data of DNN without being altered, the amount of learning data and the amount of learning time become enormous, and it becomes difficult to perform the separation of the acoustic signal. Although a plan that the acoustic feature value is devised can be adopted, most of the conventional acoustic feature values are related to tone quality such as MFCC (mel-frequency-cepstrum-coefficient) and log-mel-spectrum, or are related to a direction of an output sound of a beamformer and the like, and the acoustic feature value to be used for separating the acoustic signal based on the difference in the distance from the sound source to the microphone is still unknown.

The present invention is achieved in view of such a point, and an object thereof is to separate an acoustic signal based on a difference in the distance from a sound source to a microphone.

Means for Solving the Problem

A value corresponding to an estimated value of a short-distance acoustic signal is associated with a value corresponding to an estimated value of a long-distance acoustic signal, to obtain a filter. The value corresponding to an estimated value of a short-distance acoustic signal and the value corresponding to an estimated value of a long-distance acoustic signal are obtained from a second acoustic signal, which is derived from signals collected by “the plurality of microphones”, using “a predetermined function”. The short-distance acoustic signal means a signal emitted from a position close to “the plurality of microphones” and the long-distance acoustic signal means a signal emitted from a position far from “the plurality of microphones. By using this filter, a desired acoustic signal representing at least one of a sound emitted from a position close to “a specific microphone” and a sound emitted from a position far from “the specific microphone” is acquired from a first acoustic signal derived from a signal collected by “the specific microphone”. Note that “the predetermined function” is a function which uses such an approximation that a sound emitted from the position close to “the plurality of microphones” is collected by “the plurality of microphones” as a spherical wave, and a sound emitted from the position far from “the plurality of microphones” is collected by “the plurality of microphones” as a plane wave.

Effects of the Invention

By using the filter obtained by associating the value corresponding to the estimated value of the short-distance acoustic signal with the value corresponding to the estimated value of the long-distance acoustic signal, it becomes possible to separate the acoustic signal based on the difference in the distance from the sound source to the microphone.

FIG. 1 is a block diagram illustrating the functional configuration of an acoustic signal separation system of an embodiment.

FIG. 2 is a block diagram illustrating the functional configuration of a learning device of the embodiment.

FIG. 3 is a block diagram illustrating the functional configuration of an acoustic signal separation device of the embodiment.

FIG. 4 is a flowchart for explaining learning processing of the embodiment.

FIG. 5 is a flowchart for explaining separation processing of the embodiment.

Hereinbelow, embodiments of the present invention will be described with reference to the drawings.

[Principle]

First, a principle will be described.

In the embodiment described below, from signals collected by M+1 microphones, at least one of a sound source positioned near the microphones (near sound source) and a sound source positioned far from the microphones (distant sound source) is separated. Note that the distance from each microphone to each near sound source is shorter than the distance from each microphone to each distant sound source. For example, the distance from each microphone to each near sound source is not more than 30 cm, and the distance from each microphone to each distant sound source is not less than 1 m. Note that M is an integer of not less than 1, and is preferably an integer of not less than 2. An observed signal in a time-frequency domain in a time interval t at a frequency f, which is obtained by sampling an observed signal in a time domain collected by the m∈{0, . . . , M}-th microphone and further converting the observed signal to the observed signal in the time-frequency domain, is given by

X t,f (m)   [Formula 1]

and is defined as follows:

X t,f (m) =S t,f (m) +N t,f (m) ( 1)  [Formula 2]

where

S t,f (m)   [Formula 3]

is a component corresponding to a short-distance acoustic signal in the time-frequency domain in the time interval t at the frequency f which is obtained by sampling a short-distance acoustic signal obtained by collecting a near sound emitted from the near sound source with the m-th microphone and further converting the short-distance acoustic signal to the short-distance acoustic signal in the time-frequency domain.

N t,f (m)   [Formula 4]

is a component corresponding to a long-distance acoustic signal in the time-frequency domain in the time interval t at the frequency f which is obtained by sampling a long-distance acoustic signal obtained by collecting a distant sound emitted from the distant sound source with the m-th microphone and further converting the long-distance acoustic signal to the long-distance acoustic signal in the time-frequency domain. t∈{1, . . . , T} and f∈{1, . . . , F} are indexes of the time interval (frame) and the frequency (discrete frequency) in the time-frequency domain. Each of T and F is a positive integer, the time interval corresponding to the index t is written as “a time interval t”, and the frequency corresponding to the index f is written as “a frequency f”. Due to restriction of description and notation, in the following description, in some cases,

X t,f (m) ,S t,f (m) ,N t,f (m)   [Formula 5]

are written as X t,f (m) , S t,f (m) , and N t,f (m) . Although the detailed description thereof will be omitted, S t,f (m) is dependent on each transmission characteristic from an original signal of each near sound source to the m-th microphone from the near sound source, and N t,f (m) is dependent on each transmission characteristic form an original signal of each distant sound source to the m-th microphone from the distant sound source. The conversion to the time-frequency domain can be performed by, e.g., the fast Fourier transform (FFT) or the like.

<Near Sound Extraction by Internal Sound Field Prediction Based on Spherical Harmonic Expansion>

First, a description will be given of a near sound collection method which uses a spherical microphone array including a microphone disposed at the center of a sphere and M microphones disposed at regular intervals on the spherical surface of the sphere. Suppose that, among the above-mentioned M+1 microphones, the 0-th microphone is disposed at the center of the sphere, and the other first to M-th microphones are disposed at regular intervals on the spherical surface of the sphere. In this method, attention is focused on such an approximation that the sound wave of a distant sound comes to the microphone as a plane wave, and the sound wave of a near sound comes to the microphone as a spherical wave. In the case where only a sound which comes from the outside of a spherical surface having a radius r (r is a positive value) is present, it is possible to predict a sound pressure on the spherical surface having a radius r0 (r0<r) from a spherical harmonic spectrum (spherical harmonic expansion coefficient) of a sound pressure distribution observed on the spherical surface. Herein, the sound pressure at the center of the sphere is predicted by using observed signals at the first to M-th microphones disposed on the spherical surface, and a difference between the predicted sound pressure at the center of the sphere and the sound pressure observed by the microphone disposed at the center of the sphere is obtained. The distant sound has excellent approximation accuracy as the plane wave, and hence the difference approaches 0. On the other hand, in the case of the near sound, plane wave approximation is difficult, and hence the near sound corresponds to the difference as an approximation error. As a result, near sound source enhancement (i.e., to separate an estimated value of a short-distance acoustic signal emitted from a position close to the microphone from the observed signal) is implemented. This processing can be written as follows (see, e.g., Reference 1 or the like):

[ Formula ⁢ ⁢ 6 ] S ^ t , f , D = X t , f , D ( 0 ) - ∑ m ′ = 1 M ⁢ 1 J 0 ⁡ ( kr ) ⁢ 1 M ⁢ X t , f , D ( m ′ ) ( 2 )

wherein J 0 (kr) is a spherical Bessel function, and k is a wave number corresponding to a frequency f. The left side of Formula 2 represents the estimated value of the short-distance acoustic signal and, due to restriction of description and notation, in some cases, this is written as S{circumflex over ( )} t,f,D in the following description. Similarly, in some cases,

X t,f,D (m)   [Formula 7]

is written as X t,f,D (m) . D, which is a subscript, represents a down-sampled signal. That is, S{circumflex over ( )} t,f,D is obtained by down-sampling S{circumflex over ( )} t,f , and X t,f,D (m) is obtained by down-sampling X t,f (m) .

[Reference 1] Haneda Yoichi, Furuya Ken'ichi, Koyama Shoichi, Niwa Kenta, “Kyumen Chowa Kansu Tenkai ni Motozuku 2-Syurui no Cho-setsuwa Maikurohon Arei” (Two Types of Super Close-Talking Microphone Arrays Based on Spherical Harmonic Expansion), IEICE Transactions A, Vol. J97-A, No. 4, pp. 264-273, 2014.

The estimated value S{circumflex over ( )} t,f,D of the short-distance acoustic signal obtained by Formula (2) is a down-sampled signal. This is because the maximum frequency of the acoustic signal which can be separated by the above-described method is dependent on the radius r of the spherical microphone array. For example, in the case where the spherical microphone array having a radius r=5 (cm) is used, a forbidden frequency called “spherical Bessel zero” is present in the vicinity of 3.4 kHz. Accordingly, the observed signal has to be down-sampled to its Nyquist frequency or less before separation, or an algorithm has to be designed such that only the frequency of not more than the forbidden frequency is processed. On the other hand, in an application which handles the acoustic signal in voice recognition or the like, a signal in a frequency band equal to or higher than 4 kHz is used. Therefore, it is not possible to use the above method as preprocessing of such an application without altering the method.

<Estimation of Time-Frequency Mask which Uses Deep Learning>

Next, a description will be given of time-frequency mask processing serving as another sound source separation method. In the time-frequency mask processing, the estimated value S{circumflex over ( )} t,f of a target signal is obtained from the acoustic signal X t,f by the following formula:

Ŝ t,f =G t,f X t,f ( 3)  [Formula 8]

wherein G t,f is the time-frequency mask. In addition, due to restriction of description and notation, the left side of Formula (3) is written as S{circumflex over ( )} t,f . In the case where the target signal is the short-distance acoustic signal included in the acoustic signal X t,f and a noise signal is the long-distance acoustic signal, G t,f is obtained, e.g., as follows:

[ Formula ⁢ ⁢ ⁢ 9 ] G t , f =  S t , f ( 0 )   S t , f ( 0 )  +  N t , f ( 0 )  ( 4 )

That is, when the short-distance acoustic signal S t,f (0) and the long-distance acoustic signal N t,f (0) are known, the time-frequency mask G t,f is easily obtained. However, in general, the short-distance acoustic signal S t,f (0) and the long-distance acoustic signal N t,f (0) are unknown, and the time-frequency mask G t,f has to be estimated in some way. In DL (deep learning) sound source enhancement which uses DNN (Deep Neural Network) (also referred to as “DNN sound source enhancement”), a vector G t =(G t,1 , . . . , G t,F ) obtained by vertically arranging time-frequency masks G t,1 , . . . , G t,F at individual frequencies f∈{1, . . . , F} in the time interval t is estimated as follows (see, e.g., Reference 2 or the like):

G t =M (ϕ t |θ) (5)  [Formula 10]

wherein M is a regression function which uses a neural network, ϕ t is an acoustic feature value in the time interval t which is extracted from the observed signal, θ is a parameter of the neural network, and ⋅ T represents transposition of ⋅. In addition, 0≤G t,f ≤1 is satisfied.

[Reference 2] H. Erdogan, J. R. Hershey, S. Watanabe, and J. L. Roux, “Phase-sensitive and recognition-boosted speech separation using deep recurrent neural networks,” in Proc. ICASSP, 2015.

In order to estimate G t in the DL sound source enhancement elaborately, it is necessary to use the acoustic feature value ϕ t having a large mutual information amount with G t (see, e.g., Reference 3 or the like). In other words, the acoustic feature value ϕ t needs to include a clue (information) for distinguishing between the short-distance acoustic signal and the long-distance acoustic signal.

[Document 3] Y. Koizumi, K. Niwa, Y. Hioka, K. Kobayashi and H. Ohmuro, “Informative acoustic feature selection to maximize mutual information for collecting target sources”, IEEE/ACM Trans. Audio, Speech and Language Processing, PP. 768-779, 2017.

As described above, the short-distance acoustic signal corresponds to the original signal emitted from the near sound source, the long-distance acoustic signal corresponds to the original signal emitted from the distant sound source, and the distance from the microphone to the near sound source is different from the distance from the microphone to the distant sound source. Consequently, as the acoustic feature value ϕ t , the acoustic feature value representing the distance from the sound source to the microphone or the spatial feature of the sound field should be used. However, MFCC (mel-frequency-cepstrum-coefficient) or log-mel-spectrum, which is widely used in the DL sound source enhancement, is the feature value related to tone quality, and the feature value lacks the distance from the sound source to the microphone and the spatial information of the sound field. In addition, the spatial feature value significantly changes depending on the reverberations or shape of a room, and hence it has been difficult to use the spatial feature value as the acoustic feature value for the DL sound source enhancement. Accordingly, it has been difficult to implement near/distant sound source separation in which at least one of the short-distance acoustic signal and the long-distance acoustic signal is separated from the observed signal based on the DL sound source enhancement.

Method of Present Embodiment

In contrast to this, in the embodiment described below, the time-frequency mask which implements the near/distant sound source separation is estimated with deep learning by using the acoustic feature value obtained by spherical harmonic analysis. With this method, (1) it becomes possible to implement the near/distant sound source separation even in a high frequency band in which the near/distant sound source separation cannot be implemented in the spherical harmonic analysis. This is because, although only the acoustic feature value in a low frequency band can be used in learning of the time-frequency mask, it is possible to use the time-frequency mask obtained by the learning in a high frequency band. In addition, (2) By using the acoustic feature value obtained by the spherical harmonic analysis, it is possible to estimate the time-frequency mask allowing the near/distant sound source separation which has been difficult to implement in the DL sound source enhancement. The detailed description thereof will be given below.

It is known that, in deep learning, it is possible to input the observed signal to the neural network as the feature value without altering the observed signal (see, e.g., Reference 4 or the like).

[Reference 4] Q. V. Le, K. Chen, G. S. Corrado, J. Dean, and A. Y. Ng, “Building High-level Features Using Large Scale Unsupervised Learning,” in Proc. of ICML, 2012.

Therefore, it is intuitively conceivable to use a method in which the signal collected by the above-described spherical microphone array is directly input to the neural network as the acoustic feature value. However, realistically, it is difficult to use this method because of the following reasons. In most cases, the number of microphones M+1 of the spherical microphone array is larger than the number of microphones of a typical microphone array (for example, in Reference 1, 33 microphones are used). In sound source enhancement which uses deep learning, the acoustic feature value is often obtained by combining amplitude spectra of about five preceding frames and five subsequent frames (see, e.g., Reference 2 or the like). Accordingly, in the case where the observed signals obtained by 33 microphones are sampled, the observed signals in the time-frequency domain are obtained by using the fast Fourier transform (FFT) of 512 points, and the observed signals in the time-frequency domain are used as the input to the neural network without being altered, the number of dimensions of the input is 257 [points]×(1+5+5) [frames]×33 [channels]=93291 [dimensions] (6), which is enormous. In general, when the number of dimensions of the input to the neural network increases, enormous learning data and an enormous amount of calculation time are required in order to avoid overfitting. Therefore, in order to implement the near/distant sound source separation, the acoustic feature value which has the large mutual information amount with the above G t and the number of dimensions of the input which is as small as possible should be used. Accordingly, it is conceivable to use the estimated value S{circumflex over ( )} t,f,D of the short-distance acoustic signal obtained by the spherical harmonic analysis of Formula (2) as the acoustic feature value. This is because a component corresponding to the distant sound is reduced and a component corresponding to the near sound is enhanced in S{circumflex over ( )} t,f,D obtained by Formula (2), and S{circumflex over ( )} t,f,D is expected to include the clue for distinguishing between the short-distance acoustic signal and the long-distance acoustic signal. However, S{circumflex over ( )} t,f,D includes a component (residual noise of the distant sound) corresponding to the distant sound which is not erased by Formula (2), and the neural network may erroneously determine that the residual noise of the distant sound is the component corresponding to the near sound.

To cope with this, an estimated value N{circumflex over ( )} t,f,D of the long-distance acoustic signal corresponding to the distant sound is also calculated by the following method:

[ Formula ⁢ ⁢ 11 ] N ^ t , f , D =  X t , f , D ( 0 )  -  S ^ t , f , D   X t , f , D ( 0 )  · X t , f , D ( 0 ) ( 7 )

wherein |⋅| represents the absolute value of ⋅. Further, an acoustic feature value ϕ t obtained by associating a value corresponding to the estimated value S{circumflex over ( )} t,f,D of the short-distance acoustic signal obtained by Formula (2) with a value corresponding to the estimated value N{circumflex over ( )} t,f,D of the long-distance acoustic signal obtained by Formula (7) is calculated.

φ t =( ŝ t−C,D ,{circumflex over (n)} 1−C,D , . . . ,ŝ t+C,D ,{circumflex over (n)} t+C,D ) T (8)  [Formula 12]

where

ŝ t,D =ln( Mel [Abs[( Ŝ t,1,D ,Ŝ t,2,D , . . . ,Ŝ t,F,D )]]) (9)  [Formula 13]

{circumflex over (n)} t,D =ln( Mel [Abs[( {circumflex over (N)} t,1,D ,{circumflex over (N)} t,2,D , . . . ,{circumflex over (N)} t,F,D )]]) (10)  [Formula 14]

wherein C is a positive integer representing a context window length and, e.g., C=5 is satisfied. Abs[(⋅)] represents an operation for replacing each element of a vector (⋅) with the absolute value of each element. That is, the operation result of Abs[(⋅)] is a vector which has the absolute value of each element of the vector (⋅) as its element. Mel[(⋅)] represents an operation for obtaining a B-dimensional vector by multiplying the vector (⋅) by a Mel conversion matrix. That is, the operation result of Mel[(⋅)] is the B-dimensional vector corresponding to the vector (⋅). B=64 is satisfied. ln(⋅) represents an operation for replacing each element of the vector (⋅) with the natural logarithm of the element. That is, the operation result of ln(⋅) is a vector which has the natural logarithm of each element of the vector (⋅) as its element. In addition, due to restriction of description and notation, there are cases where the left side of Formula (9) is written as s{circumflex over ( )} t,D , and the left side of Formula (10) is written as n{circumflex over ( )} t,D .

In addition, the acoustic feature value ϕ t may also be obtained by the following procedure:

1. By using X t,f,D (m) (m∈{0, . . . , M}) obtained by down-sampling the observed signal X t,f (m) having a sampling frequency sf 1 (first frequency) to the observed signal having a sampling frequency sf 2 (second frequency), each of S{circumflex over ( )} t,f,D and N{circumflex over ( )} t,f,D , which is down-sampled so as to have the sampling frequency sf 2 , is calculated according to Formulas (2) and (7). Note that sf 2 <sf 1 is satisfied.

2. S{circumflex over ( )} t,f,D and N{circumflex over ( )} t,f,D are up-sampled to S{circumflex over ( )} t,f and N{circumflex over ( )} t,f each having the sampling frequency sf 1 .

3. In up-sampled states, by using S{circumflex over ( )} t,f and N{circumflex over ( )} t,f instead of S{circumflex over ( )} t,f,D and N{circumflex over ( )} t,f,D , s{circumflex over ( )} t and n{circumflex over ( )} t are calculated instead of s{circumflex over ( )} t,D and n{circumflex over ( )} t,D according to Formulas (9) and (10). Further, s{circumflex over ( )} t,L is obtained by extracting only an element in a frequency band equal to or lower than the Nyquist frequency from s{circumflex over ( )} t , and n{circumflex over ( )} t,L is obtained by extracting only an element in a frequency band equal to or lower than the Nyquist frequency from n{circumflex over ( )} t .

4. The acoustic feature value ϕ t is calculated according to Formula (8) by using s{circumflex over ( )} t,L and n{circumflex over ( )} t,L instead of s{circumflex over ( )} t,D and n{circumflex over ( )} t,D .

In this case, in the case where the sampling frequency sf 1 after up-sampling is 16 kHz, the number of dimensions of the acoustic feature value ϕ t is as follows:

40[points]×(1+5+5)[frames]×2[2channels consisting of near and distant channels]=880[dimensions]  (11)

As described above, in the case where the observed signal is used as the input to the neural network without being altered, the number of dimensions of the acoustic feature value corresponds to the number of microphones M+1 channels (33 channels in the example of Formula (6)), and the number of dimensions thereof has an extremely large value (93291 dimensions in the example of Formula (6)). In contrast to this, the number of dimensions of the acoustic feature value ϕ t obtained by associating the value corresponding to the estimated value S{circumflex over ( )} t,f,D of the short-distance acoustic signal with the value corresponding to the estimated value of the long-distance acoustic signal N{circumflex over ( )} t,f,D as shown in Formula (8) corresponds to two channels consisting of S{circumflex over ( )} t,f,D and N{circumflex over ( )}{circumflex over ( )} t,f,D irrespective of the number of microphones M+1, and has a relatively small value (880 dimensions in the example of Formula (11)). For example, when Formula (6) is compared with Formula (11), the number of dimensions of the acoustic feature value ϕ t of Formula (8) is reduced to 1/100 or less as compared with the case where the observed signal is used as the input to the neural network without being altered.

The parameter θ of the above-described Formula (5) is learned by using the acoustic feature value ϕ t obtained in the above manner as learning data. For example, by using the given short-distance acoustic signal S t,f (0) , the given observed signal X t,f (0) , and the acoustic feature value ϕ t obtained from the observed signal X t,f (m) as learning data, the parameter θ which minimizes the following function value J(θ) is learned.

[ Formula ⁢ ⁢ ⁢ 15 ] J ⁡ ( Θ ) = ∑ t = 1 T ⁢  S t ( 0 ) - M ⁡ ( φ t | Θ ) ∘ X t ( 0 )  2 ( 12 ) where [ Formula ⁢ ⁢ 16 ] S t ( 0 ) = ( S t , 1 ( 0 ) , … ⁢ , S t , F ( 0 ) ) T ( 13 ) [ Formula ⁢ ⁢ 17 ] X t ( 0 ) = ( X t , 1 ( 0 ) , … ⁢ , X t , F ( 0 ) ) T ( 14 )

αOβ represents an operation (multiplication for each element) for obtaining a vector which has an element obtained by multiplying an element of a vector α and an element of a vector β which are at the same positions together as its element. That is, when α=(α 1 , . . . , α F ) T and β=(β 1 , . . . , β F ) T are satisfied, αOβ=(α 1 β 1 , . . . , α F β F ) T is satisfied. In addition, ∥α∥ q is a L q norm.

By using the parameter θ obtained in the above manner, it becomes possible to perform acoustic signal separation on X t,f (m) (m∈{0, . . . , M}) which is newly obtained by being subjected to collection with M+1 microphones, sampling, and conversion to the time-frequency domain. That is, by using the parameter θ and the acoustic feature value ϕ t calculated from newly obtained X t,f (m) , G t =(G t,1 , . . . , G t,F ) T is obtained according to Formula (5), and S{circumflex over ( )} t,f can be calculated according to Formula (3).

First Embodiment

A first embodiment will be described.

<Configuration>

As illustrated in FIG. 1 , an acoustic signal separation system 1 of the present embodiment has a learning device 11 , an acoustic signal separation device 12 , and a spherical microphone array 13 .

«Learning Device 11 »

As illustrated in FIG. 2 , the learning device 11 of the present embodiment has a setting section 111 , a storage section 112 , a random sampling section 113 , down-sampling sections 114 - m (m∈{0, . . . , M}), function operation sections 115 and 116 , a feature value calculation section 117 , a learning section 118 , and a control section 119 .

«Acoustic Signal Separation Device 12 »

As illustrated in FIG. 3 , the acoustic signal separation device 12 of the present embodiment has a setting section 121 , a signal processing section 123 , down-sampling sections 124 - m (m∈{0, . . . , M}), function operation sections 125 and 126 , a feature value calculation section 127 , and a filter section 128 .

«Spherical Microphone Array 13 »

The spherical microphone array 13 has the 0-th microphone disposed at the center of a sphere having a radius r, and the first to M-th microphones disposed at regular intervals on the spherical surface of the sphere.

<Learning Processing>

Next, by using FIG. 4 , learning processing of the present embodiment will be described.

As preprocessing, the short-distance acoustic signal obtained by collecting the near sound emitted from a single or a plurality of any near sound sources with M+1 microphones of the spherical microphone array 13 is sampled with the sampling frequency sf 1 and the short-distance acoustic signal is converted to the short-distance acoustic signal in the time-frequency domain, and the short-distance acoustic signal S t,f (m) (m∈{0, . . . , M}) in the time-frequency domain is thereby obtained. A plurality of S t,f (m) are acquired while the near sound source is randomly selected, and the set S consisting of the plurality of S t,f (m) is obtained. Similarly, the long-distance acoustic signal obtained by collecting the distant sound emitted from a single or a plurality of any distant sound sources with M+1 microphones of the spherical microphone array 13 is sampled with the sampling frequency sf 1 and the long-distance acoustic signal is converted to the long-distance acoustic signal in the time-frequency domain, and the long-distance acoustic signal N t,f (m) (m∈{0, . . . , M}) in the time-frequency domain is thereby obtained. A plurality of N t,f (m) are acquired while the distant sound source is randomly selected, and the set N consisting of the plurality of N t,f (m) is obtained. In addition, various parameters p (e.g., M, F, T, C, B, r, sf 1 , sf 2 , and parameters required for learning) are set. S, N, and p obtained by the preprocessing are input to the setting section 111 of the learning device 11 ( FIG. 2 ). The sets S and N are stored in the storage section 112 , and various parameters p are set in the individual sections of the learning device 11 (Step S 111 ).

The random sampling section 113 randomly selects the short-distance acoustic signals {S t,f (0) , . . . , S t,f (M) } and the long-distance acoustic signals (N t,f (0) , . . . , N t,f (M) ) in T+2C or more time intervals (frames) t (f∈{1, . . . , F}) from the sets S and N stored in the storage section 112 , performs a simulation in which the observed signals {X t,f (0) , . . . , X t,f (M) } are obtained by superimposing the short-distance acoustic signals on the long-distance acoustic signals, and outputs the obtained observed signals X t,f (m) (m∈{0, . . . , M}) (Step S 113 ).

Each observed signal X t,f (m) obtained in Step S 113 is input to each down-sampling section 114 - m . The down-sampling section 114 - m down-samples the observed signal X t,f (m) to the observed signal X t,f,D (m) having the sampling frequency sf 2 (a second acoustic signal derived from signals collected by a plurality of microphones), and outputs the observed signal (Step S 114 ).

The observed signals X t,f,D (0) , . . . , X t,f,D (M) obtained in Step S 114 are input to the function operation section 115 . The function operation section 115 obtains the estimated value S{circumflex over ( )} t,f,D of the short-distance acoustic signal (the estimated value of the short-distance acoustic signal emitted from a position close to a plurality of microphones) from the observed signals X t,f,D (0) , . . . , X t,f,D (M) according to Formula (2) (a predetermined function), and outputs the estimated value (Step S 115 ).

The observed signal X t,f,D (0) obtained in Step S 114 and the estimated value S{circumflex over ( )} t,f,D of the short-distance acoustic signal obtained in Step S 115 are input to the function operation section 116 . The function operation section 116 obtains the estimated value N{circumflex over ( )} t,f,D of the long-distance acoustic signal (the estimated value of the long-distance acoustic signal emitted from a position far from a plurality of microphones) from X t,f,D (0) and S{circumflex over ( )} t,f,D according to Formula (7), and outputs the estimated value (Step S 116 ).

The estimated value S{circumflex over ( )} t,f,D of the short-distance acoustic signal obtained in Step S 115 and the estimated value N{circumflex over ( )} t,f,D of the long-distance acoustic signal obtained in Step S 116 are input to the feature value calculation section 117 . The feature value calculation section 117 calculates the above acoustic feature value ϕ t (the acoustic feature value obtained by associating the value s{circumflex over ( )} t,D corresponding to the estimated value S{circumflex over ( )} t,f,D of the short-distance acoustic signal with the value n{circumflex over ( )} t,D corresponding to the estimated value N{circumflex over ( )} t,f,D of the long-distance acoustic signal) according to the following Formulas (8), (9), and (10), and outputs the acoustic feature value ϕ t (Step S 117 ).

The acoustic feature value ϕ t obtained in Step S 117 and S t,f (0) and X t,f (0) (t∈{1, . . . , T}, f∈{1, . . . , F}) corresponding to the acoustic feature value ϕ t are input to the learning section 118 as learning data. The learning section 118 learns the parameter θ (information corresponding to a filter) so as to minimize the function value J(θ) of Formula (12) with the acoustic feature value ϕ t , and S t,f (0) and X t,f (0) by using a known learning method. As the learning method, for example, stochastic gradient descent or the like may be appropriately used, and its learning rate may be set to about 10 −5 (Step S 118 ).

The control section 119 performs a convergence determination to determine whether or not a convergence condition has been met. Examples of the convergence condition include a condition that learning has been repeated a specific number of times (e.g., one hundred thousand times), and a condition that the change amount of the parameter θ obtained by each learning has fallen within a specific range. In the case where the control section 119 determines that the convergence condition is not met, the processing returns to the processing in Step S 113 . On the other hand, in the case where the control section 119 determines that the convergence condition has been met, the learning section 118 outputs the parameter θ which has met the convergence condition. By using this parameter θ and Formula (5), it is possible to obtain the time-frequency masks G t,1 , . . . , G t,F corresponding to the unknown acoustic feature value ϕ t (Step S 119 ).

<Separation Processing>

Next, by using FIG. 5 , separation processing of the present embodiment will be described. As preprocessing, parameters p′ (identical to the above parameters p except parameters required for learning) are input to the setting section 121 , and the parameter θ output in Step S 119 is input to the filter section 128 . The parameters p′ are set in the individual sections of the acoustic signal separation device 12 , and the parameter θ is set in the filter section 128 . Thereafter, the following processing is executed for each time interval t.

The sound emitted from a single or a plurality of any sound sources is collected by M+1 (plural) microphones of the spherical microphone array 13 , and the signals obtained by the collection are sent to the signal processing section 123 (Step S 121 ). The signal processing section 123 samples the signal acquired by the m∈{0, . . . , M}-th microphone with the sampling frequency sf 1 and further converts the signal to the signal in the time-frequency domain to obtain the observed signal X′ t,f (m) (m∈{0, . . . , M}) in the time-frequency domain (a second acoustic signal derived from signals collected by a plurality of microphones), and outputs the observed signal (Step S 123 ).

Each observed signal X′ t,f (m) obtained in Step S 123 is input to each down-sampling section 124 - m . The down-sampling section 124 - m down-samples the observed signal X′ t,f (m) to the observed signal X′ t,f,D (m) having the sampling frequency sf 2 (the second acoustic signal derived from signals collected by a plurality of microphones), and outputs the observed signal (Step S 124 ).

The observed signals X′ t,f,D (0) , . . . , X′ t,f,D (M) obtained in Step S 124 are input to the function operation section 125 . According to

[ Formula ⁢ ⁢ 18 ] S ^ t , f , D ′ = X t , f , D ′ ⁡ ( 0 ) - ∑ m ′ = 1 M ⁢ 1 J 0 ⁡ ( kr ) ⁢ 1 M ⁢ X t , f , D ′ ⁡ ( m ′ ) ( 15 )

(a predetermined function), the function operation section 125 obtains the estimated value S{circumflex over ( )}′ t,f,D of the short-distance acoustic signal (the estimated value of the short-distance acoustic signal emitted from the position close to a plurality of microphones) from the observed signals X′ t,f,D (0) , . . . , X′ t,f,D (M) , and outputs the estimated value. Note that, due to restriction of description and notation, the left side of Formula (15) is written as S{circumflex over ( )}′ t,f,D (Step S 125 ).

The observed signal X′ t,f,D (0) obtained in Step S 124 and the estimated value S{circumflex over ( )}′ t,f,D of the short-distance acoustic signal obtained in Step S 125 are input to the function operation section 126 . According to

[ Formula ⁢ ⁢ 19 ] N ^ t , f , D ′ =  X t , f , D ′ ⁡ ( 0 )  -  S ^ t , f , D ′   X t , f , D ′ ⁡ ( 0 )  · X t , f , D ′ ⁡ ( 0 ) , ( 16 )

the function operation section 126 obtains the estimated value N{circumflex over ( )}′ t,f,D of the long-distance acoustic signal (the estimated value of the long-distance acoustic signal emitted from the position far from a plurality of microphones) from X′ t,f,D (0) and S{circumflex over ( )}′ t,f,D , and outputs the estimated value. Note that, due to restriction of description and notation, the left side of Formula (16) is written as N{circumflex over ( )}′ t,f,D (Step S 126 ).

The estimated value S{circumflex over ( )}′ t,f,D of the short-distance acoustic signal obtained in Step S 125 and the estimated value N{circumflex over ( )}′ t,f,D of the long-distance acoustic signal obtained in Step S 126 are input to the feature value calculation section 127 . According to the following Formulas (17), (18), and (19), the feature value calculation section 127 calculates the acoustic feature value ϕ′ t (the acoustic feature value obtained by associating the value s{circumflex over ( )}′ t,D corresponding to the estimated value S{circumflex over ( )}′ t,f,D of the short-distance acoustic signal with the value n{circumflex over ( )}′ t,D corresponding to the estimated value N{circumflex over ( )}′ t,f,D of the long-distance acoustic signal), and outputs the acoustic feature value ϕ′ t .

φ′ t =( ŝ′ t−C,D ,{circumflex over (n)}′ 1−C,D , . . . ,ŝ′ t+C,D ,{circumflex over (n)}′ t+C,D ) T (17)  [Formula 20]

ŝ′ t,D =ln( Mel [Abs[( Ŝ′ t,1,D ,Ŝ′ t,2,D , . . . ,Ŝ′ t,F,D )]]) (18)  [Formula 21]

{circumflex over (n)}′ t,D =ln( Mel [Abs[( {circumflex over (N)} t,1,D ,{circumflex over (N)}′ t,2,D , . . . {circumflex over (N)}′ t,F,D )]]) (19)  [Formula 22]

Note that, due to restriction of description and notation, the left sides of Formulas (18) and (19) are written as s{circumflex over ( )}′ t,D and n{circumflex over ( )}′ t,D , respectively (Step S 127 ).

Each observed signal X′ t,f (0) obtained in Step S 123 and the acoustic feature value ϕ′ t obtained in Step S 127 are input to the filter section 128 . The filter section 128 calculates the vector G t =(G t,1 , . . . , G t,F ) T obtained by vertically arranging the time-frequency masks G t,1 , . . . , G t,F by using the above-described parameter θ in the following manner:

G t =M (φ′ t |θ) (20)  [Formula 23]

Each of the time-frequency masks G t,1 , . . . , G t,F obtained in this manner is a filter (nonlinear filter) obtained by associating the value s{circumflex over ( )} t,D (s{circumflex over ( )}′ t,D ) corresponding to the estimated value S{circumflex over ( )} t,f,D (S{circumflex over ( )}′ t,f,D ) of the short-distance acoustic signal emitted from the position close to a plurality of microphones with the value n{circumflex over ( )} t,D (n{circumflex over ( )}′ t,D ) corresponding to the estimated value N{circumflex over ( )} t,f,D (N{circumflex over ( )}′ t,f,D ) of the long-distance acoustic signal emitted from the position far from a plurality of microphones. Further, by using the time-frequency mask G t,f (f∈{0, . . . , F}), the filter section 128 acquires the estimated value S{circumflex over ( )}′ t,f of the short-distance acoustic signal (a desired acoustic signal representing a sound emitted from a position close to a specific microphone) from the observed signal X′ t,f (0) (a first acoustic signal derived from a signal collected by a specific microphone) in the following manner, and outputs the estimated value:

Ŝ′ t,f =G t,f X′ t,f (21)  [Formula 24]

Note that, in the present embodiment, the sampling frequency of the time-frequency mask G t,f is still sf 2 , and hence, before the calculation of Formula (21) is performed, it is desirable to up-sample the sampling frequency of the time-frequency mask G t,f to the sampling frequency sf 1 or the sampling frequency in the vicinity of the sampling frequency sf 1 (Step S 128 ). The output S{circumflex over ( )} t,f may be converted to the signal in the time domain or may also be used in other processing without being converted to the signal in the time domain.

Modification 1 of First Embodiment

In Step S 128 in the first embodiment, the filter section 128 of the acoustic signal separation device 12 acquires the estimated value S{circumflex over ( )} t,f of the short-distance acoustic signal from the observed signal X′ t,f (0) by using the time-frequency mask G t,f , and outputs the estimated value (Formula (21)). However, the acoustic signal separation device 12 may include a filter section 128 ′ instead of the filter section 128 , and the filter section 128 ′ may acquire the estimated value N{circumflex over ( )}′ t,f of the long-distance acoustic signal (the desired acoustic signal representing the sound emitted from the position far from a specific microphone) from the observed signal X′ t,f (0) by using the time-frequency mask G t,f in the following manner, and output the estimated value:

{circumflex over (N)}′ t,f =(1− G t,f ) X′ t,f (22)  [Formula 25]

Alternatively, the acoustic signal separation device 12 may include the filter section 128 ′ in addition to the filter section 128 , the filter section 128 may acquire the estimated value S{circumflex over ( )} t,f of the short-distance acoustic signal according to Formula (21) as described above, and output the estimated value, and the filter section 128 ′ may acquire the estimated value N{circumflex over ( )}′ t,f of the long-distance acoustic signal according to Formula (22) as described above, and output the estimated value. Alternatively, it may be possible to select, based on the input, the acquisition and outputting of the estimated value S{circumflex over ( )}′ t,f of the distance acoustic signal by the filter section 128 or the acquisition and outputting of the estimated value N{circumflex over ( )}′ t,f of the long-distance acoustic signal by the filter section 128 ′ (Step S 128 ′).

Modification 2 of First Embodiment

In Step S 118 in the first embodiment, the learning section 118 of the learning device 11 learns the parameter θ (information corresponding to the filter) so as to minimize the function value J(θ) of Formula (12). However, the learning device 11 may include a learning section 118 ″ instead of the learning section 118 , and the learning section 118 ″ may use the acoustic feature value ϕ t obtained in Step S 117 , and N t,f (0) and X t,f (0) (t∈{1, . . . , T}, f∈{1, . . . , F}) corresponding to the acoustic feature value ϕ t as learning data, and learn the parameter θ (information corresponding to the filter) so as to minimize the function value J(θ) by using a known learning method in the following manner (Step S 118 ″):

In this case, the filter section 128 of the acoustic signal separation device 12 may acquire the estimated value N{circumflex over ( )}′ t,f of the long-distance acoustic signal from the observed signal X′ t,f (0) by using the time-frequency mask G t,f in the following manner and output the estimated value:

{circumflex over (N)}′ t,f =G t,f X′ t,f (25)  [Formula 28]

Alternatively, the filter section 128 ′ of the acoustic signal separation device 12 may acquire the estimated value S{circumflex over ( )}′ t,f of the short-distance acoustic signal from the observed signal X′ t,f (0) by using the time-frequency mask G t,f in the following manner and output the estimated value:

Ŝ′ t,f =(1− G t,f ) X′ t,f (26)  [Formula 29]

Alternatively, the acoustic signal separation device 12 may include the filter section 128 ′ in addition to the filter section 128 , the filter section 128 may acquire the estimated value N{circumflex over ( )}′ t,f of the long-distance acoustic signal according to Formula (25) as described above and output the estimated value, and the filter section 128 ′ may acquire the estimated value S{circumflex over ( )}′ t,f of the short-distance acoustic signal according to Formula (26) as described above and output the estimated value. Alternatively, it may be possible to select, based on the input, the acquisition and outputting of the estimated value N{circumflex over ( )}′ t,f of the long-distance acoustic signal by the filter section 128 or the acquisition and outputting of the estimated value S{circumflex over ( )}′ t,f of the short-distance acoustic signal by the filter section 128 ′.

Second Embodiment

A second embodiment will be described. The present embodiment is a modification of the first embodiment, and is different from the first embodiment only in that up-sampling is performed before the calculation of the acoustic feature value. In the following description, points different from the first embodiment will be mainly described, and the description of matters common to the first embodiment will be simplified by using the same reference numerals.

<Configuration>

As illustrated in FIG. 1 , an acoustic signal separation system 2 of the present embodiment has a learning device 21 , an acoustic signal separation device 22 , and the spherical microphone array 13 .

«Learning Device 21 »

As illustrated in FIG. 2 , the learning device 21 of the present embodiment has the setting section 111 , the storage section 112 , the random sampling section 113 , the down-sampling sections 114 - m (m∈{0, . . . , M}), the function operation sections 115 and 116 , a feature value calculation section 217 , the learning section 118 , and the control section 119 .

«Acoustic Signal Separation Device 22 »

As illustrated in FIG. 3 , the acoustic signal separation device 22 of the present embodiment has the setting section 121 , the signal processing section 123 , the down-sampling sections 124 - m (m∈{0, . . . , M}), the function operation sections 125 and 126 , a feature value calculation section 227 , and the filter section 128 .

<Learning Processing>

Next, learning processing of the present embodiment will be described by using FIG. 4 . The learning processing of the present embodiment is different from the learning processing of the first embodiment only in that Step S 117 is replaced with Step S 217 described below. The other points of the learning processing are the same as those of the learning processing of the first embodiment, Modification 1 of the first embodiment, or Modification 2 of the first embodiment.

«Step S 217 »

The estimated value S{circumflex over ( )} t,f,D of the short-distance acoustic signal obtained in Step S 115 and the estimated value N{circumflex over ( )} t,f,D of the long-distance acoustic signal obtained in Step S 116 are input to the feature value calculation section 217 . The feature value calculation section 217 up-samples S{circumflex over ( )} t,f,D and N{circumflex over ( )} t,f,D to S{circumflex over ( )} t,f and N{circumflex over ( )} t,f each having the sampling frequency sf 1 . Thereafter, in up-sampled states, the feature value calculation section 217 calculates s{circumflex over ( )} t and n{circumflex over ( )} t instead of s{circumflex over ( )} t,D and n{circumflex over ( )} t,D according to Formulas (9) and (10) by using S{circumflex over ( )} t,f and N{circumflex over ( )} t,f instead of S{circumflex over ( )} t,f,D and N{circumflex over ( )} t,f,D . Further, the feature value calculation section 217 obtains s{circumflex over ( )} t,L by extracting only an element in a frequency band equal to or lower than the Nyquist frequency from s{circumflex over ( )} t , and obtains n{circumflex over ( )} t,L by extracting only an element in a frequency band equal to or lower than the Nyquist frequency from n{circumflex over ( )} t . The feature value calculation section 217 calculates the acoustic feature value ϕ t (the acoustic feature value obtained by associating the value s{circumflex over ( )} t,L corresponding to the estimated value S{circumflex over ( )} t,f,D of the short-distance acoustic signal with the value n{circumflex over ( )} t,L corresponding to the estimated value N{circumflex over ( )} t,f,D of the long-distance acoustic signal) according to Formula (8) by using s{circumflex over ( )} t,L and n{circumflex over ( )} t,L instead of s{circumflex over ( )} t,D and n{circumflex over ( )} t,D , and outputs the acoustic feature value ϕ t .

<Separation Processing>

Next, separation processing of the present embodiment will be described by using FIG. 5 . The separation processing of the present embodiment is different from the separation processing of the first embodiment only in that Step S 127 is replaced with Step S 227 described below. The other points of the separation processing are the same as those of the separation processing of the first embodiment.

«Step S 227 »

The estimated value S{circumflex over ( )}′ t,f,D of the short-distance acoustic signal obtained in Step S 125 and the estimated value N{circumflex over ( )}′ t,f,D of the long-distance acoustic signal obtained in Step S 126 are input to the feature value calculation section 227 . The feature value calculation section 227 up-samples S{circumflex over ( )}′ t,f,D and N{circumflex over ( )}′ t,f,D to S{circumflex over ( )}′ t,f and N{circumflex over ( )}′ t,f each having the sampling frequency sf 1 . Thereafter, in up-sampled states, the feature value calculation section 227 calculates s{circumflex over ( )}′ t and n{circumflex over ( )}′ t instead of s{circumflex over ( )}′ t,D and n{circumflex over ( )}′ t,D according to Formulas (18) and (10) by using S′{circumflex over ( )} t,f and N′{circumflex over ( )} t,f instead of S{circumflex over ( )}′ t,f,D and N{circumflex over ( )}′ t,f,D . Further, the feature value calculation section 227 obtains s{circumflex over ( )}′ t,L by extracting only an element in a frequency band equal to or lower than the Nyquist frequency from s{circumflex over ( )}′ t , and obtains n{circumflex over ( )}′ t,L by extracting only an element in a frequency band equal to or lower than the Nyquist frequency from n{circumflex over ( )}′ t . The feature value calculation section 227 calculates the acoustic feature value ϕ′ t (the acoustic feature value obtained by associating the value s{circumflex over ( )}′ t,L corresponding to the estimated value S{circumflex over ( )}′ t,f,D of the short-distance acoustic signal with the value n{circumflex over ( )}′ t,L corresponding to the estimated value N{circumflex over ( )}′ t,f,D of the long-distance acoustic signal) according to Formula (17) by using n{circumflex over ( )}′ t,L and n{circumflex over ( )}′ t,L instead of s{circumflex over ( )}′ t,D and n{circumflex over ( )}′ t,D , and outputs the acoustic feature value ϕ′ t .

The learning device of each of the first and second embodiments and the modifications thereof uses the learning data (the acoustic feature value ϕ t ) in which the value corresponding to the estimated value S{circumflex over ( )} t,f,D of the short-distance acoustic signal which is obtained by using “the predetermined function” (Formula (2)) from the second acoustic signal (the observed signal X t,f,D (m) ) derived from the signals collected by “the plurality of microphones” and is emitted from the position close to “the plurality of microphones” is associated with the value corresponding to the estimated value N{circumflex over ( )} t,f,D of the long-distance acoustic signal which is emitted from the position far from “the plurality of microphone”, and learns the information (the parameter θ) corresponding to the filter (the time-frequency masks G t,1 , . . . , G t,F ) for separating the desired acoustic signal representing at least one of the sound emitted from the position close to “the specific microphone” and the sound emitted from the position far from the specific microphone from the first acoustic signal (the observed signal X′ t,f (0) ) derived from the signal collected by “the specific microphone”. Note that the distance represented by the expression “close to the microphone” is shorter than the distance represented by the expression “far from the microphone”. For example, the distance represented by the expression “close to the microphone” is a distance of 30 cm or less, and the distance represented by the expression “far from the microphone” is a distance of 1 m or more. For example, the estimated value S{circumflex over ( )} t,f,D of the short-distance acoustic signal is obtained by using the second acoustic signal and “the predetermined function” (Formula (2)), and the estimated value N{circumflex over ( )} t,f,D of the long-distance acoustic signal is obtained by using the second acoustic signal and the estimated value S{circumflex over ( )} t,f,D of the short-distance acoustic signal (Formula (7)).

In addition, in the acoustic signal separation device for separating the desired acoustic signal from the first acoustic signal (the observed signal X′ t,f (0) ), by using the filter (the time-frequency masks G t,1 , . . . , G t,F serving as the filter based on the information obtained by the learning which uses the learning data in which the value corresponding to the estimated value of the short-distance acoustic signal is associated with the value corresponding to the estimated value of the long-distance acoustic signal) which is obtained by associating the value corresponding to the estimated value (S{circumflex over ( )} t,f,D , S{circumflex over ( )}′ t,f,D ) of the short-distance acoustic signal which is obtained by using “the predetermined function” from the second acoustic signal (the observed signal X t,f,D (m) , X′ t,f (0) ) derived from the signals collected by “the plurality of microphones” and is emitted from the position close to “the plurality of microphones” with the value corresponding to the estimated value (N{circumflex over ( )} t,f,D , N{circumflex over ( )}′ t,f,D ) of the long-distance acoustic signal which is emitted from the position far from the plurality of microphone, the desired acoustic signal (S{circumflex over ( )}′ t,f and/or N{circumflex over ( )}′ t,f ) representing at least one of the sound emitted from the position close to “the specific microphone” and the sound emitted from the position far from “the specific microphone” is acquired from the first acoustic signal (the observed signal X′ t,f (0) ) derived from the signal collected by “the specific microphone”.

As described above, the number of dimensions of the acoustic feature value ϕ t used as the learning data in each embodiment is obtained by associating the value corresponding to the estimated value S{circumflex over ( )} t,f,D of the short-distance acoustic signal with the value corresponding to the estimated value N{circumflex over ( )} t,f,D of the long-distance acoustic signal, and corresponds to two channels consisting of S{circumflex over ( )} t,f,D and N{circumflex over ( )} t,f,D irrespective of the number of microphones M+1. Consequently, in each embodiment, as compared with the case where the observed signals by the microphones M+1 are used as the learning data without being altered, it is possible to significantly reduce the number of dimensions of the learning data. As a result, as compared with the case where the observed signals by the microphones M+1 are used as the learning data without being altered, it is possible to reduce the data amount of the learning data and significantly reduce the amount of learning time. The acoustic feature value ϕ t is obtained by using “the predetermined function”, and “the predetermined function” is the function which uses such an approximation that the sound emitted from the position close to “the plurality of microphones” is collected by “the plurality of microphones” as the spherical wave and the sound emitted from the position far from “the plurality of microphones” is collected by “the plurality of microphones” as the plane wave. The acoustic feature value ϕ t obtained in this manner includes the clue for distinguishing between the short-distance acoustic signal and the long-distance acoustic signal, and has the large mutual information amount with G t =(G t,1 , . . . , G t,F ). Accordingly, by using such an acoustic feature value ϕ t as the learning data, it is possible to estimate the filter (the time-frequency masks G t,1 , . . . , G t,F ) with high accuracy and separate the acoustic signal with high accuracy based on the difference in the distance from the sound source to the microphone. In addition, although only the acoustic feature value in the low frequency band can be used in the learning of the filter (the time-frequency masks G t,1 , . . . , G t,F ), it is possible to use the filter obtained by the learning in the high frequency band. Accordingly, it is also possible to use the acoustic signal separation obtained by using such a filter as preprocessing of an application which handles the acoustic signal in voice recognition or the like.

The sampling frequency of the first acoustic signal (the observed signal X′ t,f (0) ) is sf 1 (the first frequency), the sampling frequency of the second acoustic signal (the observed signal X t,f,D (m) ) is sf 2 (the second frequency), and sf 2 (the second frequency) is lower than sf 1 (the first frequency). In each of the second embodiment and its modification, while the sampling frequency of each of the estimated value S{circumflex over ( )} t,f,D of the short-distance acoustic signal and the estimated value N{circumflex over ( )} t,f,D of the long-distance acoustic signal is sf 2 (the second frequency), the sampling frequency of each of the value corresponding to the estimated value S{circumflex over ( )} t,f,D of the short-distance acoustic signal and the value corresponding to the estimated value N{circumflex over ( )} t,f,D of the long-distance acoustic signal is up-sampled to sf 1 (the first frequency). Consequently, it is possible to cause the sampling frequency of the filter (the time-frequency masks G t,1 , . . . , G t,F ) obtained based on the learning to coincide with that of the first acoustic signal (the observed signal X′ t,f (0) ), and simplify filtering processing. Note that the sampling frequency of each of the estimated value S{circumflex over ( )} t,f,D of the short-distance acoustic signal and the estimated value N{circumflex over ( )} t,f,D of the long-distance acoustic signal may be in the vicinity of sf 2 (the second frequency), and the sampling frequency of each of the value corresponding to the estimated value S{circumflex over ( )} t,f,D of the short-distance acoustic signal and the value corresponding to the estimated value N{circumflex over ( )} t,f,D of the long-distance acoustic signal may be up-sampled to a frequency in the vicinity of sf 1 (the first frequency).

Note that the present invention is not limited to the above-described embodiments. For example, learning and application of the filter may be performed by using a model other than DNN. In addition, a single device including the function of the learning device and the function of the acoustic signal separation device may also be provided. The above-described various processing may be executed in parallel or individually depending on the processing capability of a device which executes the processing or on an as needed basis as well as being executed time-sequentially according to the description. In addition, it will be easily appreciated that the present invention can be changed appropriately without departing from the spirit of the present invention.

A general-purpose or dedicated computer including, e.g., a processor (hardware processor) such as a CPU (central processing unit) and a memory such as a RAM (random-access memory) or a ROM (read-only memory) executes a predetermined program, and each device described above is thereby constituted. The computer may include one processor and one memory, or may also include a plurality of processors and a plurality of memories. The program may be installed in the computer or may also be recorded in the ROM or the like in advance. In addition, part or all of processing sections may be constituted by using electronic circuitry which implements processing functions without using the program instead of electronic circuitry which implements processing functions by reading the program such as the CPU. Electronic circuitry constituting one device may include a plurality of CPUs.

In the case where the above-described configuration is implemented by a computer, the processing contents of the functions of the individual devices are described using a program. By executing the program with the computer, the above processing functions are implemented on the computer. The program in which the processing contents are described can be recorded in a computer-readable recording medium. An example of the computer-readable recording medium includes a non-transitory recording medium. Examples of such a recording medium include a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory.

Distribution of the program is performed by selling, transferring, or lending a portable recording medium such as a DVD or a CD-ROM in which the program is recorded. Further, the program may be stored in a storage device of a server computer in advance, and the program may be distributed by transferring the program from the server computer to another computer via a network.

First, for example, the computer which executes such a program temporarily stores the program recorded in the portable recording medium or the program transferred from the server computer in a storage device of the computer. When processing is executed, the computer reads the program stored in its storage device, and executes the processing corresponding to the read program. As another execution mode of the program, the computer may read the program directly from the portable recording medium and execute the processing corresponding to the program. Further, every time the program is transferred to the computer from the server computer, the computer may execute the processing corresponding to the received program. A configuration may also be adopted in which the above processing is executed by what is called an ASP (Application Service Provider)-type service in which the transfer of the program to the computer from the server computer is not performed and the processing functions are implemented only by execution instructions and result acquisition.

Instead of implementing the processing functions of the present devices by causing the predetermined program to be executed on the computer, at least part of the processing functions may be implemented by hardware.

For example, in the case where the above-described technique for separating the sound emitted from the position far from the microphone is applied to a smart speaker or the like, even when the smart speaker or the like is placed at the side of a television set, it is possible to suppress the sound of the television set to clearly extract a distant sound or the like, and it is possible to improve the quality of voice recognition and a call.

For example, in the case where the above-described technique for separating the sound emitted from the position close to the microphone is applied to an abnormal sound detection device in a factory, and the abnormal sound detection device is disposed at the side of target equipment to be monitored, it becomes possible to suppress noise coming from another section to extract only the sound of the target equipment to be monitored, and it is possible to improve detection accuracy by the abnormal sound detection device.

1 Acoustic signal separation system

11 , 21 Learning device

12 , 22 Acoustic signal separation device