Closed caption signal processing apparatus and method

This application is based upon and claims the benefit of priority from prior Japanese Patent Application No. 2003-202554, filed Jul. 28, 2003, the entire contents of which are incorporated herein by reference.

1. Field of the Invention

The present invention relates to a closed caption signal processing apparatus which outputs a speech and a closed caption based on a speech signal and a closed caption signal which express contents corresponding to each other, and a method therefor.

2. Description of the Related Art

Closed caption information is sent by a television broadcasting radio signal. However, the closed caption information does not always coincide with the speech signal. Particularly, when the closed caption is made up at a real time in the broadcasting station, the closed caption information may delay by 5 to 10 seconds with respect to the speech signal. Accordingly, when the television broadcasting radio signal is received with a television receiver, the closed caption is not displayed at the same timing as a speech. This makes a TV viewer to be hard to watch TV programming. It is desired that the closed caption is displayed at the same timing as the speech.

The object of the present invention is to provide a closed caption signal processing apparatus to detect a timing deviation between utterance and display of the closed caption, and compensate the deviation, and a method therefor.

An aspect of the present invention provides a closed caption signal processing apparatus using a speech signal and a closed caption signal, the apparatus comprising: a speech recognition unit configured to recognize speech contents from the speech signal; a speech timing detecting unit configured to detect a speech timing of the speech contents; a closed caption contents acquisition unit configured to acquire closed caption contents from the closed caption signal associated with the speech signal; a presentation timing detecting unit configured to detect a presentation timing of the closed caption contents; an agreement detecting unit configured to detect an agreement between the speech contents and the closed caption contents; and a time difference calculating unit configured to calculate a time difference between the speech timing and the presentation timing that are detected on the speech contents and the closed caption contents that agree with each other, respectively.

Another aspect of the present invention provides a method for processing a closed caption signal, comprising: recognizing speech contents from a speech signal; detecting a speech timing of the speech contents; deriving closed caption contents from a closed caption signal associated with the speech signal; detecting a presentation timing of the closed caption contents; detecting agreement between the speech contents and the closed caption contents; calculating a time difference between the speech timing and the presentation timing that are detected on the speech contents and the closed caption contents that agree with each other, respectively.

FIG. 1 is a block diagram illustrating a configuration of a hard disk recorder related to an embodiment of the present invention.

FIG. 2 is a block diagram illustrating an internal structure of a speech recognition unit 141 during FIG. 1 .

FIG. 3 is a flowchart of a process of a closed caption deviation quality determination unit 14 of FIG. 1 .

FIG. 4 is a diagram of a concrete example for speech recognition and closed caption analysis.

FIG. 5 is a diagram showing an example of correction of a presentation timing of a closed caption.

FIG. 6 is a diagram showing an example of correction of a presentation timing of a closed caption.

There will now be described one embodiment of the present invention referring to the drawing.

FIG. 1 is a block diagram illustrating a configuration of a hard disk recorder (referred to as HDD recorder) 1 related to the present embodiment.

The HDD recorder 1 includes a television receiver circuit 11 , a video recording controller 12 , a hard disk drive (referred to as HDD) 13 , a closed caption deviation quantity determination unit 14 , a deviation information storage unit 15 , a meta data memory 16 , a playback controller 17 , a closed caption signal correction unit 18 and a closed caption synthesizer 19 .

The television receiver circuit 11 receives a television broadcasting through an antenna 2 . The television broadcasting to be received by the HDD recorder 1 may be any system, but assumes an NTSC system here. Accordingly, the television receiver circuit 11 outputs a signal based on the NTSC system (referred to as NTSC signal).

The video recording controller 12 subjects video information and audio information of a baseband signal output from the television receiver circuit 11 to a compression process according to a given format and writes it in the HDD 13 . The video recording controller 12 extracts closed caption information multiplexed in the vertical blanking interval of the NTSC signal and writes it in the HDD 13 .

The closed caption deviation quantity determination unit 14 includes a speech recognition module 141 , a closed caption analysis module 142 , an agreement search module 143 , a deviation quantity determination module 144 , a deviation information generation module 145 , and a meta data generation module 146 .

This closed caption deviation quantity determination module 14 can be configured mainly by a processor, for example. The speech recognition module 141 , closed caption analysis module 142 , agreement search module 143 , deviation quantity determination module 144 , deviation information generation module 145 , and meta data generation module 146 can be realized by causing a processor to execute a program. In this time, the closed caption deviation quantity determination module 14 may be realized by a program installed in a built-in memory (not shown). Alternatively, it may be realized by a program stored in a removable recording medium such as a compact disk-read only memory, or a program distributed through a network and installed in the memory (not shown) included in the closed caption deviation quality determination module 14 .

The speech recognition module 141 subjects the speech information stored in the HDD 13 to speech recognition to derive speech contents (notation data, pronunciation data and various morpheme information), and speech timing information indicating a timing at which the speech contents are uttered.

The closed caption analysis module 142 decodes the closed caption information stored in the HDD 13 to derive text data indicating a character string of the closed caption. The closed caption analysis module 142 derives presentation timing information representing a presentation timing of the closed caption. The closed caption analysis module 142 subjects the text data to a morphological analysis to read each morpheme included in the text data and derive part-of-speech information.

The agreement search module 143 searches for an agreement between speech contents derived by the speech recognition module 141 and the text data derived by the closed caption analysis module 142 . The agreement search module 143 outputs text data agreed with the speech contents, presentation timing information of the text data, and speech timing information of the speech contents agreed with the text data to the deviation quality determination unit 144 .

The deviation quantity determination unit 144 determines a deviation quantity of a presentation timing with respect to the text data output from the agreement search module 143 as a time difference between the timings indicated respectively by the speech timing information and presentation timing information which are supplied at the same time. The deviation quantity determination module 144 outputs the determined deviation quantity along with the text data and presentation timing information to the deviation information generation module 145 and the meta data generation module 146 .

The deviation information generation module 145 generates the deviation information that made it reflect the deviation quantity output from the deviation quantity determination module 144 , and write the deviation information in the information storage unit 15 .

When the text data output from the deviation quantity determination module 144 matches a given meta registration condition, the meta data generation module 146 generates meta data which reflects the deviation quantity output from the deviation quantity determination module 144 , and write the meta data in the meta data memory 16 .

It is desirable that the deviation information storage unit 15 has a high capacity and is easy to access. It comprises, for example, a semiconductor memory or HDD. The deviation information storage unit 15 stores deviation information.

It is desirable that the meta data storage unit 16 has a high capacity and is easy to access. It comprises, for example, a semiconductor memory or HDD. The meta data storage unit 16 stores meta data.

The playback controller 17 reads video information, speech information and closed caption information which are stored in the HDD 13 , and plays back an NTSC signal or AV signal which indicates these information. The playback controller 17 outputs a baseband signal to the closed caption signal correction unit 18 , and outputs a video signal and closed caption information of the AV signal to the closed caption synthesis unit 19 .

The closed caption signal correction unit 18 corrects the closed caption information included in the NTSC signal output from the playback controller 17 referring to the deviation information stored in the information storage unit 15 . The NTSC signal from the closed caption signal correction unit 18 is sent via an NTSC terminal T 1 to the outside of the HDD recorder 1 . A television set (referred to as a closed caption broadcast compatible television) compatible with the closed-caption broadcasting, for example, is connected to the NTSC terminal T 1 .

The closed caption synthesis unit 19 synthesizes the closed caption according to the closed caption information output from the playback controller 17 with the video signal output from the playback controller 17 . In this time, the closed caption synthesis unit 19 can shift a synthesis timing of closed caption information with respect to the video signal referring to the deviation information stored in the deviation information storage unit 15 . The video signal from the closed caption synthesis unit 19 is output via a video terminal T 2 to the outside of the HDD recorder 1 . For example, a television receiver 4 with an AV terminal is connected to the video terminal T 2 .

FIG. 2 is a block diagram illustrating an internal structure of the speech recognition module 141 . The speech recognition unit 141 includes a speech feature extraction module 141 a , a pronunciation information estimation module 141 b , and a speech contents estimation module 141 c.

The speech recognition module 141 extracts a characteristic quality inherent to a speech from a speech signal with the speech feature extraction module 141 a . The speech recognition module 141 compares the characteristic quantity with a model of the speech prepared beforehand by means of the pronunciation information estimation module 141 b to estimate pronunciation information. Further, the speech recognition unit 141 estimates speech contents with linguistic meaning based on the pronunciation information by means of the speech contents estimation module 141 c . A difference between kannji character notation and hiragana character notation, a homonym, and a recognition candidate are acquired at the same time, and output as a speech recognition result.

The operation of the HDD recorder 1 configured as above will be described hereinafter. The operation for normal video recording and playback is omitted since it is similar to that of a conventional record-playback machine. The operation concerning a process of the closed caption will be described in detail hereinafter.

This HDD recorder 1 records a broadcasted program on HDD 13 , once, and then plays back the recorded program at an arbitrary timing required by a user or at a substantially real time.

In this time, if the program recorded on the HDD 13 has closed caption information, the closed caption deviation quantity determination unit 14 generates deviation information and meta data referring to the closed caption information and video information as follows.

FIG. 3 shows a flowchart of a process executed by the closed caption deviation quantity determination unit 14 . The speech recognition module 141 subjects the speech information stored in the HDD 13 to speech recognition to derive speech contents and speech timing information indicating a speech timing at which the speech contents are uttered. The speech recognition module 141 derives notation data and pronunciation data for the speech contents. The speech recognition module 141 takes a correlation time using, for example, the head of the program as a reference time to be speech timing information.

If information such as a time code indicating the absolute position of a speech signal is added to the speech signal, this information can be used as speech timing information. The speech contents of one of a series of speeches may be delimited at an arbitrary level. However, in this embodiment, it is delimited every morpheme.

Concretely, the speech recognition module 141 derives four speech contents of, for example, “ASU”, “WA”, “AME”, “DESU” as shown in FIG. 4 , from speech data representing the speech “ASU WA AME DESU”. The speech recognition module 141 acquires times t 1 , t 2 , t 3 , t 4 shown in FIG. 4 as speech timing information of the speech contents.

The closed caption analysis module 142 analyzes the closed caption information stored in the HDD 13 in step ST 2 . The closed caption analysis module 142 decodes the closed caption information stored in the HDD 13 and acquires the text data representing a closed caption sentence. The closed caption analysis module 142 subjects the text data to a morphological analysis to derive reading information of each morpheme and part-of-speech information which are included in the text data.

The closed caption analysis module 142 acquires presentation timing information representing a presentation timing of the closed caption sentence expressed by the text data. The presentation timing of the closed caption sentence assumes a timing at which the closed caption information is derived. Accordingly, the closed caption analysis module 142 sets a relative time defining, as a reference time, the position of the NTSC signal at which the closed caption information is multiplexed, for example, the head of the program, to presentation timing information.

When the presentation timing of the closed caption sentence is designated by control information superposed on the NTSC signal, this control information can be used as presentation timing information.

FIG. 4 shows an example that the text data representing the closed caption sentence of “ASU WA AME DESU” is received after the speech of “ASU WA AME DESU”. In this case, the closed caption analysis module 142 acquires a time t 5 indicating an arrival timing of the text data as presentation timing information, as well as the text data of “ASU WA AME DESU”. The closed caption analysis module 142 derives reading information of four morphemes of “ASU”, “WA”, “AME”, “DESU” and part-of-speech information from the text data.

In step ST 3 , the agreement search module 143 searches for an agreement between the character string expressing one or more speech contents which are acquired by the speech recognition module 141 and the character string represented by the text data acquired by the closed caption analysis module 142 .

When the speech contents and the text data do not agree with each other due to a difference between hiragana notation and kanji notation and so on, the agreement search unit 143 compares data of the next candidate for speech recognition and pronunciation data with the text data and the reading data acquired by the morphological analysis. The agreement search module 143 performs not only comparison between the speech contents and the text data that are identical in the speech timing and the presentation timing, but also comparison between the speech contents and the text data that are deviated in timing from each other in a fixed range.

In the example of FIG. 4 , the character string representing the speech contents and the character string representing the text data include a character string of “ASU WA AME DESU” together. Such a character string is detected by the agreement search module 143 . In performing a character string agreement search, use of various morphological information, for example, part-of-speech information permits more precise agreement.

In step ST 4 , the agreement search module 143 ensures whether an agreed character string is found by the search. If the agreed character string is found, the process advances to step ST 5 .

In step ST 5 , the text data corresponding to the found character string, the morpheme information of this text data, the presentation timing information, and the speech timing information of each speech content including the found character string are input to the deviation quantity determination module 144 from the agreement search module 143 . In step ST 5 , the deviation quantity determination module 144 calculates a time difference between the speech time of the head indicated by the speech timing information and the presentation time indicated by the presentation timing information, and determines it as a deviation quantity of the closed caption sentence indicated by the text data with respect to the speech timing.

The deviation quantity determination module 144 determines a deviation quantity based on a time difference between the speech time indicated by the speech timing information of agreed speech contents and the presentation time indicated by the presentation timing information in regard to each of morphemes included in the text data.

In the example of FIG. 4 , the speech times indicated by the speech timing information of speech contents agreeing with the closed caption sentence “ASU WA AME DESU” are t 1 , t 2 , t 3 , t 4 . Because the head is t 1 , the deviation quantity determination module 144 determines that the deviation quantity of the closed caption sentence expressing “ASU WA AME DESU” is [t 5 -t 1 ]. The deviation quantity determination module 144 determines deviation quantities of the morphemes of “ASU”, “WA”, “AME”, “DESU” as [t 5 -t 1 ], [t 5 -t 2 ], [t 5 -t 3 ], [t 5 -t 4 ], respectively.

In step ST 6 , the text data, the morpheme information, the presentation timing information, and the deviation quantity determined on the text data and morpheme information are input from the deviation quantity determination module 144 to the deviation information generation module 145 . The deviation information generation module 145 corresponds the text data, the presentation timing information and the deviation quantity to each other, and generates deviation information obtained by corresponding the morpheme information to the deviation quantity. This deviation information is stored in the deviation information storage unit 15 .

In step ST 7 , the morpheme information, the presentation timing information, and the deviation quantity determined on the morpheme information are input to the meta data generation module 146 from the deviation quantity determination module 144 . The meta data generation module 146 ensures whether a morpheme according to a predetermined meta registration condition exists in the morphemes represented by the input morpheme information. The meta registration condition may be arbitrary, but it may be, for example, “nouns”. If an appropriate morpheme exists, the process advances to step ST 8 .

In step ST 8 , the meta data generation module 146 generates the meta data by corresponding the morpheme matching the meta registration condition to the presentation timing information and the deviation quantity of the morpheme with respect to the presentation timing information, and stores the meta data in the meta data memory 16 .

If the closed caption deviation quantity determination unit 14 completes step ST 8 , the process returns to step ST 3 . If there is no morpheme matching the meta registration condition, the closed caption deviation quantity determination unit 14 passes step ST 8 , and returns the process to step ST 3 . The closed caption deviation quantity determination unit 14 repeats steps ST 3 -ST 8 till it is determined that no agreed character string is found in step ST 4 . If it is determined that no agreed character string is found in step ST 4 , the closed caption deviation quantity determination unit 14 finishes the process.

When the program recorded on the HDD 13 is played back, the closed caption signal correction unit 18 corrects the closed caption information included in the NTSC signal output from the playback controller 17 referring to the deviation information stored in the deviation information storage unit 15 .

The closed caption synthesis unit 19 synthesizes the closed caption corresponding to the closed caption information output from the playback controller with the video signal output from the playback controller 17 . In this time, the closed caption synthesis unit 19 deviates a synthesis timing of the closed caption information with respect to the video signal, referring to the deviation information stored in the information storage unit 15 .

The closed caption signal correction unit 18 and the closed caption synthesis unit 19 each have four compensation modes. In the first compensation mode, the closed caption signal correction unit 18 and the closed caption synthesis unit 19 decreases the deviation quantity of the closed caption sentence to correct the closed caption signal to approximate the presentation timing of the closed caption to the timing of the head of the agreed speech. For example, the presentation timing of the closed caption shown in FIG. 4 is corrected to a presentation timing as shown in FIG. 5 .

In the second compensation mode, the closed caption signal correction unit 18 and the closed caption synthesis unit 19 decreases the deviation quantity included in the closed caption sentence every morpheme to correct the closed caption signal to approximate the speech timing of the agreed morpheme to the presentation timing of each morpheme. For example, the presentation timing of the closed caption shown in FIG. 4 is corrected to the presentation timing as shown in FIG. 6 .

In the third compensation mode, the closed caption signal correction unit 18 and the closed caption synthesis unit 19 correct the closed caption signal to set the presentation timing of the closed caption to a timing deviated by a given time from the timing of the head of the agreed speech. For example, the presentation timing of the closed caption is again set to a timing further deviated by a give time from the state of FIG. 5 . In this time, the given time is determined according to a predefined rule such as “present only a noun at three seconds late”.

In the fourth compensation mode, the closed caption signal correction unit 18 and the closed caption synthesis unit 19 correct the closed caption signal to set the presentation timing of each morpheme of the closed caption to a timing deviated by a given time from the speaking timing of the agreed morpheme. For example, the presentation timing of the closed caption is again set to a timing further deviated by a given time from the status of FIG. 6 . In this time, the given time is determined according to a predefined rule such as “present only a noun at three seconds late”.

According to the present embodiment as described above, the deviation quantity of the presentation timing of the closed caption with respect to the speech timing can be determined.

According to the present embodiment, the closed caption presentation can be realized at an appropriate timing by performing compensation of the closed caption signal and adjustment of a synthesis timing of the closed caption to decrease the deviation of the presentation timing of the closed caption with respect to the speech timing in consideration with the determined deviation quantity.

According to the present embodiment, the closed caption presentation can be realized in a uniform deviation by performing compensation of the closed caption signal or adjustment of a synthesis timing of the closed caption to set the deviation of the presentation timing of the closed caption with respect to the speech timing at a constant time, in consideration with the determined deviation quality. This is convenient for cases, for example, to train a listening comprehension of a foreign language by a foreign language program.

According to the present embodiment, the meta data which reflects the determined deviation quantity is generated. Consequently, it is possible to identify the part of an appropriate program that a closed caption including a certain word and phrase should be presented by referring to the meta data. For this reason, if, for example, the playback controller 17 uses this meta data for searching for a playback start point, it is possible to locate the start of the program appropriately.

Selection and presentation of the word and phrase that may be more important than the meta data is a help to understand the program information. For example, providing such a keyword at the time of a fast-forward playback can provide various usages to make it possible to do a fast-forward while confirming the contents.

According to the present embodiment, since a morpheme to register with meta data is only one according with a morpheme included in the speech contents, the important word and phrase shown in both of the speech and the closed caption are extracted to generate meta data. Therefore, it is possible to generate the appropriate meta data including no meaningless information.

This embodiment permits the following various kinds of modifications.

The closed caption deviation quantity determination unit 14 may be realized as an independent device, and circulated independently. The closed caption deviation quantity determination unit 14 does not need to include the deviation information generation module 145 and the meta data generation module 146 . In this case, the deviation quantity determined by the deviation quantity determination module 144 is output to an external computer to use the determined deviation quantity with the computer.

The deviation information stored in the deviation information storage unit 15 or the meta data stored in the meta data memory 16 may be output to an external computer to use it with the computer.

Additional advantages and modifications will readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described herein. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents.