Method of processing auditory data

This invention was made with government support under Grant No.: 1 R43NS37944-01 awarded by the Small Business Innovation Research Program of the Department of Health and Human Services. The Government has certain rights in the invention.

The present invention is related to a method of transforming an electrical signal representative of a sound wave as a step in the electrical stimulation of a mammalian cochlea or for the purpose of effecting data compression of the electrical signal.

The human cochlea is a complex biochemical-electrical organ of the inner ear that translates sound waves into electrochemical impulses in the auditory nerve. Physically, the human cochlea is a coil having a wound, sound receiving surface, known as the basilar membrane, of approximately 32 mm in length. Over the past twenty years, research in the fields of bioengineering and psychoacoustics have led to enhanced access to the cochleas of hearing impaired individuals and to a better understanding of the critical elements in sound necessary for restoring hearing through the direct electrical stimulation of the cochlea in the hearing impaired. This knowledge has also led to advancement in the compression of sound representation in digital files.

Sound at a particular frequency impinging on the eardrum causes a traveling wave to exist in the cochlea, at the sound frequency with its maximum at a location corresponding to the frequency Sounds with multiple spectral components stimulate different portions of the cochlea, with higher frequency sounds stimulating cochlear loci near the initial (basal) portions of the basilar membrane and lower frequency sounds stimulating the more inner (apical) portions of the coil. Nerve fibers emanating from the various regions of the cochlea are associated with the frequencies that most efficiently stimulate those regions, and the brain, which receives neural impulses from the distributed fibers, maps those frequencies in accord with this association. The nerve stimulated by this traveling wave is associated, in the brain, with the frequency of the sound both due to this mapping of the locus associated with frequency and due to the timing of nerve impulses which tend to reflect the periodicities of lower frequencies. These time patterns of impulses carry information about single frequencies and about the relative magnitudes and phases of multiple frequency components in sounds. For this reason both the spatial mapping of frequencies and the complex timing relationships of the nerve impulses they evoke contribute to the full perception of sounds including speech.

In addition, the relative timing of auditory events at the two ears provides crucial information to a listener. For example, the difference in the times of arrival for sound vibrations at the two ears provides the listener with information about the direction in which the sound has traveled. Until now, the signal processing mechanisms of cochlear implants did not stimulate the cochlea in conformity with the timing of the arriving sound to the point where, even for those patients who were equipped with binaural implants, patients could determine the direction from which sound was arriving.

In addition to hearing restoration, this patent application addresses some problems encountered in the field of data compression of electrical signals representative of sound waves for the purposes-of efficient storage, transmission, and reproduction. One currently popular form of data compression of sound wave signals is included in the “Motion Picture Experts Group Layer 3 Audio Coding” or more simply “MPEG Layer 3 .” Advances in the field of psychoacoustics, specifically an understanding that much of a sound signal is unperceived by a human listener because it is masked by other portions of the sound signal or is redundant because sound energies within a restricted range of frequencies are not distinguished by human hearing, permit MPEG Layer 3 to achieve a data compression ratio of slightly better than ten-to-one. Unfortunately, the creation of an MPEG Layer 3 signal is not a real time process. Because of this, it is not suitable for use in telephony or other real time processes.

The first aspect of the present invention is a method for the real-time transformation of an electrical signal representative of a sound wave that includes the steps of providing an electrical signal representative of a sound wave passing said electrical signal, in parallel, through a number of bandpass filters to create a set of time domain real and imaginary band limited signals. Next, a stream of instantaneous phase angle and magnitude values for each of said set of time domain real and imaginary band limited signals is computed. Thirdly, a stream of electrical pulses or other digital representation of the phase and magnitude information is computed for delivery to a cochlear implant or transmission for decoding and synthesis of the original sound.

In a separate aspect the present invention is a method for effecting hearing restoration by the electrical stimulation of a human cochlea, comprising providing a cochlear implant assembly, including a microphone, a signal processing assembly connected to the microphone and a set of electrodes contacting the cochlea and being operatively connected to the signal processing assembly. Also, the microphone receives sound waves and translates them into an electrical signal and the signal processing assembly detects predefined events in the electrical signal in each frequency band out of a set of frequency bands and emits a set of signals in response to each detection of a predefined event. Additionally, at least one of the set of electrodes electrically stimulates the cochlea in response to each set of signals.

In a further separate aspect the present invention is a method for effecting hearing restoration by the electrical stimulation of the cochlea of a human, comprising providing a cochlear implant assembly, including a microphone, a signal processing assembly connected to the microphone, a set of electrodes contacting the cochlea and being operatively connected to the signal processing assembly. Also, the microphone receives sound waves and translates them into an electrical signal and the signal processing assembly iteratively chooses a frequency-magnitude pair in each frequency band out of a predefined set of frequency bands, each frequency-magnitude pair being representative of the sound in the frequency band. Additionally, the electrodes are stimulated in response to the frequency magnitude pairs.

In a yet further separate aspect the present invention is a method for effecting hearing restoration by the electrical stimulation of a human cochlea, comprising providing a cochlear implant assembly, including a microphone a signal processing assembly connected to the microphone and a set of electrodes contacting the cochlea and being operatively connected to the signal processing assembly. The microphone and the signal processing assembly form a set of abstracted frequency-magnitude pairs based on a sound signal received by the microphone A plurality of the electrodes cooperatively simulate the sound of all magnitude-frequency pairs.

In a still further separate aspect the present invention is a method for the real time data compression of an auditory signal, comprising the steps of converting the auditory signal into a digital electronic signal having an initial sampling rate, in real time and forming a time sequence of abstracted parameter values, representative of the auditory signal, in real time. Additionally, the time sequence is encoded to form an encoded time sequence that includes a full representation of the abstracted parameter values less often than the initial sampling rate of the auditory signal.

The foregoing and other objectives, features, and advantages of the invention will be more readily understood upon consideration of the following detailed description of the invention, taken in conjunction with the accompanying drawings.

FIG. 1 is a block diagram of an assemblage of hardware that may serve as a host for the present invention.

FIG. 2 is a block diagram of a flow of signal processing according to preferred embodiment of the present invention.

FIG. 3 is a table showing the division of a portion of the auditory spectrum into frequency bands according to a preferred embodiment of the present invention.

FIG. 4 is a pair of graphs in the signal amplitude versus frequency domain showing the spectrum of an auditory signal (top) and a set of events (bottom) consisting of an instantaneous frequency and amplitude pair that may be derived from the auditory signal.

FIG. 5 is a graph in the real and imaginary coordinate system showing a vector comprised of the instantaneous phase and magnitude of a signal time sample from one bandpass filter.

FIG. 6 is a graph in the real and imaginary coordinate system showing a time sequence of vectors, each comprised of the instantaneous phase and magnitude of a signal time sample.

FIG. 7 is a pair of graphs, the top graph shows the instantaneous frequency and magnitude of a signal time sample and the bottom graph the electrical current applied to a set of cochlear implant electrodes to represent the instantaneous frequency and magnitude on the basilar membrane of the cochlea.

FIG. 1 represents the set of physical elements that perform the signal processing that is the subject matter of the present invention, and is presented to help the reader understand the context of the present invention. A microphone 10 creates an analogue signal that the A/D convertor 12 changes to a digital stream. This digital stream is sent to a digital signal processing (DSP) chip 14 and a microprocessor 16 , which together determine the amplitude and timing for each electrode to stimulate the cochlea and formats and outputs this information in a predetermined serial format. The DSP chip may incorporate the functions of the microprocessor in its computational load. For implementation of a cochlear implant processor, a modulator 18 places the information into a transcutaneous medium, such as radio frequency (RF), and transmits it to transcutaneous demodulator 20 . A demultiplexor 22 divides the serial stream of information yielded by demodulator 20 into a set of pulse magnitude and timing commands for the drivers 24 , each of which produces the electrical pulses to drive a particular electrode 26 .

Referring to FIG. 2 the digital signal processing in a preferred embodiment begins with the Hilbert transforming to produce an analytic signal and bandpass filtering (block 50 ) of this digitized analytic signal, into a set of frequency bands. This step is performed in the DSP chip 14 . The dimensions of the resultant bands are indicated by the table of FIG. 3 . Other sets of bands may also be desirable and would fall within the scope of the present invention. It may be desirable to tailor the bandwidth set to the individual patient. FIG. 4 (bottom) provides an indication of this division, with one vertical line lying within the frequency scope of each bandpass filter. As those skilled in the art of digital signal processing will readily recognize, the product of each bandpass filter is a time domain set of complex samples comprising an analytic signal, each having a real and imaginary component. Referring to FIG. 5, each real and imaginary filter time sample is mapped to a single complex vector whose successive time samples have magnitude and frequency properties. The vector magnitude (henceforth referred to as “instantaneous magnitude”) and real/ imaginary phase (whose rate of change is “instantaneous frequency”) are computed in the microprocessor 16 . The phase difference between neighboring time samples is then computed to determine the instantaneous frequency of each band pass filtered signal (block 52 in FIG. 2 ). FIG. 4 shows a set of lines 66 having a height equal to the instantaneous magnitude and positioned at the instantaneous frequency points.

This information now permits considerable data compression for stimulation through a cochlear implant or for the transmission or storage of a compressed sound representation. A code could be constructed for including the instantaneous frequency/magnitude information for each frequency band to meet criteria specific to the end use. Perceptual criteria including masking and other known factors eliminate the need to transmit many samples. Furthermore, since lower frequencies are sampled more than needed to accurately represent them under the Nyquist criteria, updates at significantly longer intervals relative to the original digitized signal are possible. Furthermore, the elimination of signal events degrades the available information in a continuous fashion in contrast to the generation of large amounts of distortion as samples of the original signal are eliminated. Various compression schemes known to those familiar with signal processing would provide a means for optimally representing the information in a serial or parallel bit stream.

FIG. 6 shows a sequence of real/imaginary vectors numbered according to each sample's relative time position in a sequence of samples. A similar sequence is constructed for each of the frequency bands shown in FIG. 4 . The real axis crossing 60 , where I= 0 , occurs between time samples 11 and 12 . In one preferred embodiment, the cochlea is electrically stimulated at a time that is a uniform time delay from each such real axis crossing. The exact timing of the real axis crossing is determined through standard interpolation and is well within the limits of human discrimination for timing sound events in one ear or between the ears for low frequencies. Among advantages of event based cochlea stimulation is that for patients fitted with a cochlear implant in both ears (a binaural implant) the precisely timed binaural stimulation would give the patient a cue for the direction from which the sound was coming. Although, of course, the cochlear stimulation is delayed in both ears, it is the relative timing of stimulation that permits this determination to be made. Other events could be used for cochlear stimulation without diverging from the invention. Among events that could be used are imaginary axis crossings or combinations of axis crossings and criteria for successive instantaneous frequency occurrences.

At this point, the instantaneous frequency and magnitude at the moment of the event must, for each frequency band, be translated into a set of electrode stimulating pulses (block 54 ). The basic goal is to create a flow of electricity through the basilar membrane that will electrically stimulate the auditory nerve endings in a close proximity to the way they would be stimulated by a sound wave having the computed instantaneous magnitude and instantaneous frequency. It should be noted that the response of the cochlea to a sound wave at a single frequency is not limited to a single point on the cochlea. Rather, the traveling wave that is created has a significant effect over about 1 mm of cochlear length. To create a flow of electricity that stimulates the auditory nerve endings in a similarly restricted fashion, it is desirable to use a single electrode very near the target nerve endings or multiple electrodes to minimize the effective stimulating electrical field. For example, two electrodes, one being charged negatively and the other being charged positively, will concentrate current flow in the region between them By using three electrodes it is possible to concentrate the effective region of stimulation to an even greater extent.

A feature of the present invention is the continuous mapping to cochlear loci of stimulus frequency. This enables the support of large numbers of electrode contacts located in high density along the basilar membrane. However, even large numbers of contacts (e.g., 40 to 100) may not actualize an exact mapping, so the calculation of pulse delivery, in addition to the restriction of current spread mentioned above, would include the selection of currents to maximally stimulate at a desired cochlear location even though it lies between adjacent electrodes. For example, if the instantaneous frequency translated to a location 15.2 mm from the beginning of the basilar membrane and electrodes were available at 14.5 and 15.3 mm, the following equation could be solved for relative current flow magnitudes:

( a *14.5 +b *15.3)/( a +b )=15.2

where a and b are the relative currents at electrodes 1 and 2 relative to a distant reference. Current weighting of this nature can be extended to 3 or more contacts but is dependent on the homogeneity of the cochlear electroanatomy in the region of stimulation and on the distance from the stimulating electrodes to the excitable nerve cells. Adjustment of the currents can be used to accommodate inhomogeneities with perceptual feedback from the patient. FIG. 7 shows a single frequency and its magnitude. The frequency would be located at a position near to one of the electrodes drawn below it. The nearest electrode is to the left of the frequency line. That electrode is driven along with its two neighboring electrodes. In case a, the shaded electrodes are driven with the currents described by the equation on page 7 . The central electrode carries a+b and the flanking electrodes carry -a and -b. To steer the current toward the right hand flanking electrode, b<a. The purpose is to attempt to steer electric current nearer to the nerves that encode that frequency. The technique works in some cases, but not in all. In case b, the shaded electrodes are driven in a quadruple configuration. The central electrode carries 2 a and the flanking electrodes each carry -a.

In a cochlear implant 110 having two rows of transversely spaced electrodes 112 and 114 over the active length of the implant, the primary stimulus may be achieved by passing current between an electrode in row 112 and its corresponding electrode in row 114 -. By restricting the spread of the current field this may achieve a stronger stimulation of the auditory nerves of interest.

This paragraph describes one method of stimulating the electrodes of the cochlear implant in conformity with the pulse magnitude and time values for each electrode that are determined as described above. There are many different possible ways of doing this, however, and the invention, specifically, is not in any way limited by or to the following described mechanism or assembly. Referring to FIGS. 1 and 2, the signals representing the pulse magnitude to be delivered to each electrode are multiplexed into a serial signal having a predetermined format (block 56 ) by the microprocessor 16 . The signal is then modulated onto a medium, such as RF at encoder 18 (block 58 ) and transmitted into the body. In an alternative preferred embodiment the signal is sent by way of a percutaneous connector, eliminating the need for elements 18 and 20 . A subcutaneous receiver and demodulator 60 , receives and demodulates the signal from transmitter 18 . The signal is then demultiplexed by a demultiplexor 24 , into a set of channels equal to the number of electrodes 26 (block 62 ) and used to stimulate a set of electrode drivers 24 (block 54 ), which in turn stimulate the electrodes 26 that contact the cochlea.

As used in this patent application the term “set” may refer to a set containing a single element only.

The terms and expressions which have been employed in the foregoing specification are used therein as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding equivalents of the features shown and described or portions thereof, it being recognized that the scope of the invention is defined and limited only by the claims which follow.