Signal separation system and method for automatically selecting threshold to separate sound sources
Granted 6 May 2014 · 1 office action
Assignee: Samsung Electronics
Law firm: Law firm · Log in to unlock
Attorney: Attorney · Log in to unlock
Inventors: Ki Wan Eom, Chan Woo Kim, Richard M. Stern, Jae Won Lee · Examiner: Vivian Chin · AU 2654 · TC 2600
Life of the application
8 dated eventsAbstract
A signal separation system and a method for automatically selecting a threshold to separate sound sources. The signal separation system calculates a power sequence for a target signal using a target mask, and a power sequence for an interference signal using a complementary mask, based on signals received from a plurality of microphones; applies a nonlinearity to the target signal power sequence and the interference signal power sequence; calculates a correlation coefficient of the nonlinear target signal power sequence and the nonlinear interference signal power sequence; and sets a noise masking threshold that minimizes the correlation coefficient.
Description
9 parts›CROSS-REFERENCE TO RELATED APPLICATIONS
This application claims the benefit under 35 USC 119(a) of Korean Patent Application No. 10-2010-0007751 filed on Jan. 28, 2010, in the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes.
›BACKGROUND
1. Field
The following description relates to a signal separation system and a method for automatically selecting a threshold to separate sound sources.
2. Description of Related Art
Accuracy of speech recognition generally degrades in noisy environments even though the performance of speech recognition technology has been considerably improved. Thus, there is a demand to effectively solve a problem where the accuracy of speech recognition is reduced in speech recognition systems actually employed in consumer products.
Accordingly, there is a desire for a system and a method for effectively separating a target sound from interference sound sources.
›SUMMARY
In one general aspect, a signal separation system includes a power sequence calculator to calculate a power sequence for a target signal using a target mask, and a power sequence for an interference signal using a complementary mask, based on signals received from a plurality of microphones; and a threshold setting unit to apply a nonlinearity to the target signal power sequence and the interference signal power sequence; calculate a correlation coefficient of the nonlinear target signal power sequence and the nonlinear interference signal power sequence; and set a noise masking threshold that minimizes the correlation coefficient.
The power sequence calculator may generate the target mask and the complementary mask based on at least one difference selected from an interaural time difference (ITD) of the received signals, an interaural phase difference (IPD) of the received signals, and an interaural intensity difference (IID) of the received signals.
The signal separation system may further include a difference calculator to apply a short-time Fourier transform (STFT) to each of the received signals; and calculate the at least one difference based on the STFT-transformed signals.
The threshold setting unit may calculate the correlation coefficient based on the nonlinear target signal power sequence, the nonlinear interference signal power sequence, and at least one difference selected from an interaural time difference (ITD) of the received signals, an interaural phase difference (IPD) of the received signals, and an interaural intensity difference (IID) of the received signals.
The threshold setting unit may set the at least one difference as the noise masking threshold that minimizes the correlation coefficient.
The nonlinearity may be a logarithmic nonlinearity or a power-law nonlinearity.
The target mask and the complementary mask may each be a binary mask or a continuous mask.
In another general aspect, a signal separation method includes calculating a power sequence for a target signal using a target mask, and a power sequence for an interference signal using a complementary mask, based on signals received from a plurality of microphones; applying a nonlinearity to the target signal power sequence and the interference signal power sequence; calculating a correlation coefficient of the nonlinear target signal power sequence and the nonlinear interference signal power sequence; and setting a noise masking threshold that minimizes the correlation coefficient.
In another general aspect, a signal separation system includes a masking unit to individually mask signals received from a plurality of microphones using a target mask and a complementary mask, and a threshold setting unit to set a noise masking threshold that minimizes a correlation between the masked signals.
In another general aspect, a signal separation method includes individually masking signals received from a plurality of microphones using a target mask and a complementary mask; and setting a noise masking threshold that minimizes a correlation between the masked signals.
In another general aspect, a signal separation system includes a masked spectrum generator to generate a masked target signal spectrum and a masked interference signal spectrum from signals received from a plurality of microphones using a target mask and a complementary mask; and a threshold setting unit to set a threshold of the target mask and the complementary mask based on a difference between the received signals so that the threshold minimizes a correlation between a nonlinearized target power sequence of the masked target signal spectrum and a nonlinearized interference power sequence of the masked interference signal spectrum.
In another general aspect, a signal separation method includes generating a masked target signal spectrum and a masked interference signal spectrum from signals received from a plurality of microphones using a target mask and a complementary mask; and setting a threshold of the target mask and the complementary mask based on a difference between the received signals so that the threshold minimizes a correlation between a nonlinearized target power sequence of the masked target signal spectrum and a nonlinearized interference power sequence of the masked interference signal spectrum.
Other features and aspects will be apparent from the following detailed description, the drawings, and the claims.
›BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 shows an example of a left microphone, a right microphone, a target sound source, and an interference sound source.
FIG. 2 shows an example of a process to select an optimum masking interaural time difference (ITD) threshold for sound source separation.
FIG. 3 shows an example of a signal separation system.
FIG. 4 shows an example of a signal separation method.
FIG. 5 shows an example of a signal separation system.
FIG. 6 shows an example of a signal separation method.
Throughout the drawings and the detailed description, unless otherwise indicated, the same drawing reference numerals will be understood to refer to the same elements, features, and structures. The relative size and depiction of these elements may be exaggerated for clarity, illustration, and convenience.
›DETAILED DESCRIPTION · 1 of 5
The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatuses, and/or systems described herein. Accordingly, various changes, modifications, and/or equivalents of the methods, apparatuses, and/or systems described herein will be suggested to those of ordinary skill in the art. Also, descriptions of well-known functions and constructions may be omitted for increased clarity and conciseness.
The human binaural system has the ability to separate a desired sound even in noisy environments where a variety of sounds are mixed. This is sometimes referred to as the binaural cocktail party effect.
In techniques used for separation of sounds, sounds may be separated based on a unique frequency for each sound, information on a direction from which a sound comes, and an auditory characteristic for masking sounds other than a desired sound.
Various methods of separating signals based on information on a sound generation direction have been developed using an interaural time difference (ITD), an interaural phase difference (IPD), and an interaural intensity difference (IID). The interaural intensity difference (IID) is also known as an interaural level difference (ILD). Phase information may be widely used in binaural processing since it is easy to acquire the phase information through frequency analysis.
In many algorithms based on the techniques described above, a binary masking scheme or a continuous masking scheme may be used to select a time-frequency bin dominated by a target sound source. The continuous masking scheme typically exhibits a superior performance compared to the binary masking scheme, but usually requires that the location of a noise source be known. However, the binary masking scheme may be used in the case of an omnidirectional noise environment or when there is no prior information about the location or characteristics of a noise source. However, the performance of the binary masking scheme depends on a threshold that is selected, and the optimal threshold depends on the location and strength of the noise source, which may not be known. Also, if the location and strength of the noise source is variable, the optimal threshold may vary over time.
Described below is a binary masking scheme in which the ITD, among the ITD, the IPD, and the IID, is set as a threshold. Generally, an appropriate ITD threshold may be selected from a set of potential ITD candidates. However, the optimum ITD threshold will depend on the number of noise sources and the location of the noise sources, and may vary over time. For example, when a direction of a sound from a noise source differs greatly from a direction of a sound from a target sound source, an ITD threshold encompassing a wider range of ITDs might provide better results. However, if such an ITD threshold encompassing a wider range of ITDs is used when the noise source is located very close to the target sound source, interference sound source signals as well as target sound source signals may be passed by the ITD threshold. This problem may become more complicated when there is more than one noise source and/or when a noise source moves.
Thus, as described below, two complementary masks employing a binary threshold may be used. When the two complementary masks are used, two different spectra may be obtained, i.e., a spectrum for a target sound source and a spectrum for an interference sound source. Short-time powers for the target sound source and the interference sound source may be obtained from the two spectra as short-time power sequences. A nonlinearity may be applied to the short-time power sequences. A correlation coefficient may be calculated from the power sequences with the applied nonlinearity, and an ITD threshold that minimizes the correlation coefficient may be selected.
A process of acquiring an ITD from phase information is described below. It is assumed that x L [n] and x R [n] denote signals received from a left microphone and a right microphone, respectively.
FIG. 1 shows an example of a left microphone 101 , a right microphone 102 , a target sound source 103 , and an interference sound source 104 . As shown in FIG. 1 , the target sound source 101 is placed on a perpendicular bisector 105 between the two microphones, and the interference sound source is placed on a line 106 rotated by an angle θ from the perpendicular bisector 105 in the clockwise direction. The two microphones are separated by a distance Δ. The distance from the interference sound source 104 to the left microphone 101 is longer than the distance from the interference sound source 104 to the right microphone 102 , which causes a sound from the interference sound source 104 to reach the right microphone 102 earlier than it reaches the left microphone 101 , producing an interaural time difference (ITD) and an interaural phase difference (IPD). The difference between the distances from the interference sound source 104 to the left microphone 101 and the right microphone 102 is Δ sin θ. Since the intensity of a sound diminishes with distance, this difference in distances causes the intensity of the sound at the right microphone 102 to be greater than the intensity of the sound at the left microphone 101 , thereby producing an interaural intensity difference (IID). When a total number of interference sound sources is S, individual sound sources s have respective ITDs δ(s). Both S and δ(s) are typically unknown. With the above formulations, the signals received from the left microphone 101 and the right microphone 102 , as denoted by x L [n] and x R [n], respectively, may be represented by the following Equation 1:
where x o [n] denotes a target signal, and x s [n] denotes signals received from each interference sound source s, where s ranges from 1 to S.
To perform spectral analysis, Equation 1 is multiplied by a Hamming window w[n] to obtain short-time signals represented by the following Equation 2:
›DETAILED DESCRIPTION · 2 of 5
x L [n;m]=x L [n−mL fp ]w[n]
x R [n;m]=x R [n−mL fp ]w[n]
for 0 ≦n≦L fl −1 (2)
where m denotes a frame index, L fp denotes a frame period, L fl denotes a frame length, and w[n] denotes a Hamming window having a length L fl . The Hamming window is well known in the art, and thus will not be described in detail here. Additionally, n denotes a sample index in a digital signal, and x L [n;m] and x R [n;m] denote signals that are an n-th sample in an m-th frame among signals received through the left microphone 101 and the right microphone 102 . In other words, since n and m have different characteristics, a semicolon is used instead of a comma to classify n and m.
FIG. 2 shows an example of a process to select an optimum masking ITD threshold for sound source separation. In operations 201 a and 201 b , a short-time Fourier transform (STFT) is performed using the following Equation 3 on the short-time signals obtained using Equation 2 from the signals received from the left microphone 101 and the right microphone 102 , which are represented by Equation 1. In other words, the STFT corresponding to Equation 1 may be represented by the following Equation 3:
X L [ m , ⅇ jω k ) = ∑ s = 0 S X s [ m , ⅇ jω k )
X R [ m , ⅇ jω k ) = ∑ s = 0 S ⅇ - jω k d s [ m , k ] X s [ m , ⅇ jω k ) ( 3 )
where ω k =2πk/N (0≦ω k ≦N/2−1) denotes a Fast Fourier Transform (FFT) size, [m,k] denotes a specific time-frequency bin, and k denotes one of N frequency bins, with positive frequency samples corresponding to ω k . Additionally, in ‘[m,e jω k )’, ‘[’ may indicate that m denotes a discrete signal, and ‘)’ may indicate that e jω k denotes a continuous signal.
Assuming that s*[m,k] is the strongest sound source for a specific time-frequency bin [m,k], the following Equation 4 may be derived from Equation 3:
X L [m,e jω k )≈ X s*[m,k] [m,e −jω k )
X R [m,e jω k )≈ e −jω k d s*[m,k] [m,k] ×X s*[m,k] [m,e −jω k ) (4)
The strongest sound source s*[m,k] may be either 0, indicating a target sound source, or 1≦s≦S, indicating any of the interference sound sources.
In operation 202 , from Equation 4, the ITD from the phases of the signals X L [m,e jω k ) and X R [m,e jω k ) for a particular time-frequency bin [m,k] is given by the following Equation 5:
d s * [ m , k ] [ m , k ] ≈ 1 ω k min r ∠X R [ m , ⅇ - jω k ) - ∠X L [ m , ⅇ - jω k ) - 2 π r ( 5 )
where r denotes a smallest integer multiple.
Thus, based on whether the obtained ITD from Equation 5 is within a certain range of the target ITD (which is zero), determination is made on whether the time-frequency bin [m,k] is likely to belong to the target speaker or not.
In operation 203 , the estimated ITD is smoothed. Smoothing over all frequency channels may be useful. The smoothing is well known in the art, and thus will not be described in detail here.
Next, two complementary binary masks may be obtained. One of the two complementary binary masks may identify time-frequency components that are believed to belong to the target signal, and the other may identify the components that are believed to belong to the interfering signals (i.e., everything except the target signal). The two complementary binary masks may be used to construct two different spectra corresponding to the power sequences representing the target and the interfering sources. A compressive nonlinearity may be applied to the power sequences, and the optimal ITD threshold may be defined as a threshold that minimizes the cross-correlation between these two output sequences (after the nonlinearity).
One element τ 0 of a finite set T of potential ITD threshold candidates may be considered to be an optimum ITD threshold. This element τ 0 may be used to obtain a target mask μ T [m,k] and a complementary mask μ I [m,k] as represented by the following Equation 6 for 0≦k≦N/2:
For N/2≦k≦N−1, a symmetry condition may be used as represented by the following Equation 7:
μ T [m,k]=μ T [m,N−k],N/ 2 ≦k≦N− 1
μ I [m,k]=μ I [m,N−k],N/ 2 ≦k≦N− 1 (7)
In other words, only time-frequency bins having |d[m,k]|≦τ 0 are considered to belong to a target sound source, and only time-frequency bins having |d[m,k]|>τ 0 are considered to belong to a noise source.
In operations 204 a and 204 b , a target time-frequency bin and a complementary time-frequency bin are selected, respectively, using the masks described by Equations 6 and 7. For time-frequency bins belonging to the noise source, i.e., the interference sound source, the interference sound may be removed by multiplying the time-frequency bins by a value of 0. However, since an interference sound spectrum typically contains some portion of the target sound spectrum, a floor constant η having a very small value may be used to preserve the portion of the target sound spectrum in the interference sound spectrum. For example, a value of 0.01 may be used for the floor constant η, although other values may also be used. The target mask μ T [m,k] and the complementary mask μ I [m,k] described by Equations 6 and 7 are applied to X [m,e jω k ), which is an average signal spectrogram of the left and right channels. The average signal spectrogram may be represented by the following Equation 8:
Using the procedure described above, a target spectrum X T [m,e jω k |τ 0 ) and an interference spectrum X I [m,e jω k |τ 0 ) may be represented by the following Equation 9:
X T [m,e jω k |τ 0 )= X [m,e jω k )μ T [m,e jω k )
X I [m,e jω k |τ 0 )= X [m,e jω k )μ I [m,e jω k ) (9)
Equation 9 explicitly includes the ITD threshold τ 0 to indicate that the target spectrum and the interference spectrum will depend on the ITD threshold τ 0 .
In operations 205 a and 205 b , frame powers of the target spectrum X T [m,e jω k ) and the interference spectrum X I [m,e jω k ) may be obtained as represented by the following Equation 10:
P T [ m ❘ τ 0 ) = ∑ k = 0 N - 1 X T [ m , ⅇ jω k ) 2
P I [ m ❘ τ 0 ) = ∑ k = 0 N - 1 X I [ m , ⅇ jω k ) 2 ( 10 )
where P T [m|τ 0 ) denotes a power for the target signal, and P I [m|τ 0 ) denotes a power for the interference signal.
›DETAILED DESCRIPTION · 3 of 5
In operations 206 a and 206 b , a nonlinearity is applied to each of the powers calculated in operations 205 a and 205 b . It is well known that the perceived loudness of a sound source is not proportional to the intensity of the sound source. Many nonlinearity models have been proposed to express a relationship between the perceived loudness and the intensity of the sound source. A logarithmic nonlinearity and a power-law nonlinearity are widely used as nonlinearity models. The results of applying the power-law nonlinearity to the powers calculated in operations 205 a and 205 b may be represented by the following Equation 11:
R T [m|τ 0 )= P T [m|τ 0 ) α 0
R I [m|τ 0 )= P I [m|τ 0 ) α 0 (11)
where α 0 denotes a power coefficient and may have, for example, a value of 1/15.
In operation 207 , a correlation coefficient is calculated from the results obtained using Equation 11. The correlation coefficient may be represented by the following Equation 12:
ρ T , I ( τ 0 ) = 1 N ∑ m = 1 M R T [ m ❘ τ 0 ) R I [ m ❘ τ 0 ) - μ R T μ R I σ R T σ R I ( 12 )
where σ R T and σ R I denote standard deviations of R T [m|τ 0 ) and R I [m|τ 0 ), respectively, and μ R T and μ R I denote averages of R T [m|τ 0 ) and R I [m|τ 0 ), respectively.
Then, the ITD threshold {circumflex over (τ)} 0 that minimizes the correlation coefficient ρ T,I (τ 0 ) expressed by Equation 12 is determined using the following Equation 13:
In operation 208 , an inverse fast Fourier transform (IFFT) is applied to a power per frequency unit using the target time-frequency bin selected in operation 204 a and the ITD threshold {circumflex over (τ)} 0 that minimizes the correlation coefficient obtained in operation 207 to generate a separated target signal that is substantially free of interference signals.
In operation 209 , an overlap-addition (OLA) method is performed on the separated target signal obtained in operation 208 to enhance the quality of the separated target signal. The OLA method is well known in the art, and thus will not be described in detail here.
FIG. 3 shows an example of a signal separation system 300 . In FIG. 3 , the signal separation system 300 includes a difference calculator 310 , a power sequence calculator 320 , and a threshold setting unit 330 .
The difference calculator 310 applies an STFT to each of a plurality of signals received from a plurality of microphones, and calculates at least one of three differences, an ITD, an IPD, and an IID. While an example of using the ITD has been described above with reference to FIGS. 1 and 2 , a threshold for noise masking may be automatically set based on a noise environment using the IPD, or the IID, or any two of the ITD, the IPD, and the IID, or all three of the ITD, the IPD, and the IID. An example of obtaining an ITD using Equation 5 has been described above. The IPD or the IID may also be applied to the examples in a similar manner to the ITD. The examples relate to how to use the calculated difference to set an optimum threshold, and thus how to obtain the IPD or the IID will not be described in detail here.
The power sequence calculator 320 calculates two power sequences from the received signals, one for a target signal and the other for an interference signal, using a target mask and a complementary mask. The target mask and the complementary mask are generated based on the difference calculated by the difference calculator 310 . For example, a power for the target signal and a power for the interference signal are calculated based on the ITD using Equation 10 as described above. Each of the target mask and the complementary mask may be a binary mask or a continuous mask.
The threshold setting unit 330 sets a threshold for noise masking so that a correlation coefficient has a minimum value. The correlation coefficient is calculated after applying a nonlinearity to the two power sequences. Specifically, the correlation coefficient is calculated from the two power sequences to which the nonlinearity is applied, and the difference calculated by the difference calculator 310 . A difference that minimizes the correlation coefficient is set as a threshold by the threshold setting unit 330 . The nonlinearity may be a logarithmic nonlinearity or a power-law nonlinearity. For example, using Equations 11 to 13 described above, the power-law nonlinearity may be applied to the two power sequences and an ITD may then be determined so that the correlation coefficient has a minimum value. The determined ITD is set as the optimum threshold for noise masking. After setting the optimum threshold in an initial sound period, whether to use the optimum threshold in a sound period subsequent to the initial sound period may be determined, or a search range may be changed, based on a variation pattern of the threshold since there is no radical change in a threshold for masking.
FIG. 4 shows an example of a signal separation method. The signal separation method of FIG. 4 may be performed by the signal separation system 300 of FIG. 3 . The signal separation method is described below with reference to FIG. 4 .
In operation 410 , the signal separation system 300 applies the STFT to each of a plurality of signals received from a plurality of microphones, and calculates at least one of three differences, an ITD, an IPD, and an IID. The operation of obtaining the ITD using Equation 5 has been described above, and thus will not be described in detail here.
In operation 420 , the signal separation system 300 generates a target mask and a complementary mask based on the difference calculated in operation 410 . Each of the target mask and the complementary mask may be a binary mask or a continuous mask.
In operation 430 , the signal separation system 300 calculates two power sequences, one for a target signal and the other for an interference signal, using the target mask and the complementary mask, respectively, with respect to the received signals. The target mask and the complementary mask are generated based on the difference calculated in operation 410 . For example, a power for the target signal and a power for the interference signal may be calculated based on the ITD using Equation 10 as described above.
›DETAILED DESCRIPTION · 4 of 5
In operation 440 , the signal separation system 300 sets a threshold for noise masking so that a correlation coefficient has a minimum value. The correlation coefficient is calculated after applying a nonlinearity to the two power sequences. Specifically, the correlation coefficient is calculated based on the two power sequences to which the nonlinearity is applied, and the difference calculated in operation 410 . A difference that minimizes the correlation coefficient is set as a threshold by the signal separation system 300 . The nonlinearity may be a logarithmic nonlinearity or a power-law nonlinearity. For example, using Equations 11 to 13 described above, the power-law nonlinearity may be applied to the two power sequences and an ITD may then be determined so that the correlation coefficient has a minimum value. The determined ITD is set as the optimum threshold for noise masking. After setting the optimum threshold in an initial sound period, whether to use the optimum threshold in a sound period subsequent to the initial sound period may be determined, or a search range may be changed, based on a variation pattern of the threshold since there is no significant change in a threshold for masking.
FIG. 5 shows an example of a signal separation system 500 . In FIG. 5 , the signal separation system 500 includes a masking unit 510 and a threshold setting unit 520 .
The masking unit 510 individually masks signals received from a plurality of microphones using a target mask and a complementary mask. Each of the target mask and the complementary mask may be a binary mask or a continuous mask. The target mask and the complementary mask have been described above in detail with reference to Equations 6 and 7, and thus will not be described in detail here.
The threshold setting unit 520 sets a threshold for noise masking so that a correlation between the masked signals is minimized. Specifically, the signals received from the plurality of microphones may be masked with the target mask and the complementary mask to obtain a signal for a target signal and a signal for an interference signal, respectively. Subsequently, a threshold that minimizes a correlation between the two signals may be set for noise masking. For example, the threshold setting unit 520 may set the threshold so that a correlation coefficient calculated after applying a nonlinearity to each of the masked signals has a minimum value. Alternatively, the threshold setting unit 520 may set a threshold that minimizes mutual information between the two signals to perform noise masking. Here, the mutual information pertains to a statistical ratio of a probability of an independent occurrence of two factors to a probability of a simultaneous occurrence of two factors. In other words, the threshold for minimizing the mutual information may refer to a threshold for minimizing a ratio indicating a mutual dependency between the two signals.
FIG. 6 shows an example of a signal separation method. The signal separation method of FIG. 6 may be performed by the signal separation system 500 of FIG. 5 . The signal separation method is described below with reference to FIG. 6 .
In operation 610 , the signal separation system 500 individually masks signals received from a plurality of microphones using a target mask and a complementary mask. Each of the target mask and the complementary mask may be a binary mask or a continuous mask. The target mask and the complementary mask have been described above in detail with reference to Equations 6 and 7, and thus will not be described in detail here.
In operation 620 , the signal separation system 500 sets a threshold for noise masking so that a correlation between the masked signals is minimized. Specifically, the signals received from the plurality of microphones are masked with the target mask and the complementary mask to obtain a signal for a target signal and a signal for an interference signal, respectively. Subsequently, a threshold that minimizes a correlation between the two signals may be set for noise masking. For example, the signal separation system 500 may set the threshold so that a correlation coefficient calculated after applying a nonlinearity to each of the masked signals may have a minimum value. Alternatively, the signal separation system 500 may set a threshold that minimizes mutual information between the two signals to perform noise masking. Here, the mutual information pertains to a statistical ratio of a probability of an independent occurrence of two factors to a probability of a simultaneous occurrence of two factors. In other words, the threshold for minimizing the mutual information may refer to a threshold for minimizing a ratio indicating a mutual dependency between the two signals.
According to the examples described above, in the signal separation system and the signal separation method based on a plurality of microphones, a threshold for noise masking may be automatically set based on a noise environment, and thus it is possible to adaptively respond to a change in the environment in which the system and method are used.
The signal separation methods described above may be recorded, stored, or fixed in one or more non-transitory computer-readable storage medium that includes program instructions to be implemented by a computer to cause a processor to execute or perform the program instructions. The non-transitory computer-readable storage medium may also include, alone or in combination with the program instructions, data files, data structures, and the like. The non-transitory computer-readable storage medium and program instructions may be those specially designed and constructed, or they may be of the kind that are well known and available to those having skill in the computer software arts. Examples of a non-transitory computer-readable storage medium include magnetic media, such as hard disks, floppy disks, and magnetic tapes; optical media, such as CD-ROM/±R/±RW, DVD-ROM/RAM/±R/±RW, and BD (Blu-ray)-ROM/−R/−RW; magneto-optical media; and hardware devices that are specially configured to store and perform program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, and the like. Examples of program instructions include machine code, such as produced by a compiler, and files containing higher level code that may be executed by the computer using an interpreter. The described hardware devices may be configured to act as one or more software modules in order to perform the operations and methods described above, or vice versa. In addition, a non-transitory computer-readable storage medium may be distributed among computer systems connected through a network, and computer-readable codes or program instructions may be stored and executed in a decentralized manner.
›DETAILED DESCRIPTION · 5 of 5
Several examples have been described above. Nevertheless, it will be understood that various modifications may be made. For example, suitable results may be achieved if the described techniques are performed in a different order and/or if components in a described system, architecture, device, or circuit are combined in a different manner and/or replaced or supplemented by other components or their equivalents. Accordingly, other implementations are within the scope of the claims and their equivalents.
Claims as granted
32 claimsLog in to read the claims of this application.
Log in to unlockClassifications
9 codes- G10L25/90
- G10L21/0272
- G10L21/02
- G10L21/0216
- G06F17/00
- G10L21/0232
- H04R3/02
Claim changes
SoonSee which claims were amended, added or cancelled during examination, with every added and removed word marked.
The published claims of this application are not paired with the granted ones in what we hold.
File wrapper
See the full prosecution history — every USPTO and applicant action on this file, in order.
Log in to unlockDocuments
Log in to open the documents of this file: the application as filed, every office action and response, the notice of allowance.
Log in to unlockChain of title
See the full assignment history — every owner this patent has passed through, with recordation dates and reel/frame numbers.
Log in to unlock