Method and apparatus for generating encoded content using dynamically optimized conversion for 3D movies

This application claims priority to U.S. application Ser. No. 13/205,720, which, in turn, claims priority to U.S. provisional application Ser. No. 61/371,901 filed Aug. 9, 2010, and is a continuation-in-part of U.S. application Ser. No. 13/173,671 filed on Jun. 30, 2011, entitled METHOD AND APPARATUS FOR GENERATING ENCODED CONTENT USING DYNAMICALLY OPTIMIZED CONVERSION and claiming priority to U.S. Provisional patent application Ser. No. 61/360,088 filed on Jun. 30, 2010, all applications being incorporated herein by reference in their entirety.

A. Field of Invention

This invention pertains to a method and apparatus of enhancing a 3D movie by adding to it new content, such as subtitles, at a particular depth.

B. Description of the Prior Art

In the past, 3D movies or other episodic 3D content were recorded optically on film and then distributed in analog formats. However, advances in digital signal processing make it possible to provide such content in digital formats which have several well-known advantages over the analog formats.

In the present application, the term ‘digital 3D movie’ is used to refer to 3D movies, episodic content, or other 3D multi-media content recorded, produced and/or converted into a digital format. This includes content produced in 2D and then post-produced from 2D to 3D, as well as rendered from 3D animation systems.

Digital 3D digital movies may be produced by converting the analog content (e.g., content recorded on film or analog video tape) into a desired digital format. Alternatively, digital video cameras are used to generate content in a raw digital format, which is then edited and converted into a format suitable for viewing or distribution. Different venues or distribution channels are used for distributing the digital 3D movie electronically, including:

A. Digital communication channel to movie house for theatrical presentation;

B. Satellite transmission to subscribers, such as DirecTV;

C. Blu-ray Disc;

D. DVD;

E. Various hand-held devices such as Nintendo 3DS® receiving streamed media over a Wi-Fi network or other private or public channels.

Each of these venues requires typically its own format(s). For example, a satellite communication channel for digital projection in a movie theater the digital 3D movie in a JPEG 2000 format.

A Blu-ray 3D Disc normally uses MVC formatted files.

A DVD may require MPEG-2 format files.

Each of these digital 3D movie distribution channels may use different forms of 3D encoding, resolution, compression, and/or authoring. For example, theatrical presentation may require 4K resolution using JPEG2000 compression with ‘burned in’ subtitles for digital theatrical exhibition. A Blu-ray disc may require an image with 1920×1080 p HD resolution using MVC compression with subtitles presented in a ‘one plane+offset’ manner for Blu-ray Disc. DirecTV may require a frame compatible spatial compression (e.g. ‘side-by-side’) and AVC compression with ‘burned in’ subtitles for DirecTV satellite broadcast. A small, hand held device may require a 800×240 pixel image (400×240 for each eye) resolution using Mobiclip compression with ‘burned in’ subtitles for Nintendo 3DS autostereo (no glasses required) viewing.

Moreover, as previously mentioned, digital 3D movies in some of the formats may incorporate additional content, e.g., menus, subtitles in one or more languages, etc., using an appropriate authoring process.

Typically, once a digital 3D movie is made and recorded in a raw format, it is converted into a format suitable for a particular distribution channel, such as the channels described above. Depending upon the order in which a digital 3D movie is prepared for distribution, e.g. theatrical, Blu-ray Disc, DVD, satellite broadcast, there may be relevant 3D information determined by encoding or authoring software and/or an operator on a frame by frame, group of frames, or scene by scene basis and recorded in a respective log. For example, a Z-axis depth map of whole frames' content, or of objects of interest within frames, may be determined when preparing to position subtitles or other graphics in 3D space over the background video.

One problem with the existing techniques described above is that they are not used systematically to add new information, such as subtitles and/or other relevant graphics, to an existing 3D movie

The present invention pertains to an apparatus, including an authoring tool, for adding a graphic element to a 3D movie. The authoring tool receives a depth map indicating the position of various objects in the frames of the 3D movie along a Z-axis. The authoring device then designates a position for additional graphic elements in each frame, these positions being determined in relation either to the positions of the objects or the position of the screen along at least said Z-axis. An encoder uses parameters from the authoring tool to reauthor the 3D content by adding the graphic element at the positions designated by the parameters. The graphic element may include subtitles and other graphic elements.

FIG. 1 shows a block diagram of a system constructed in accordance with the present invention;

FIG. 2 shows a frame of a scene of a movie captured in 3D;

FIG. 3 shows a depth map with a first title position for the frame of FIG. 2 ; and

FIG. 4 shows a depth map with a second title position for the frame of FIG. 2 .

The present invention provides a solution for the above-described problem and it operates as follows. More particularly, the present invention presents a system and method for adding subtitles and other relevant graphics to a 3D movie. It should be understood that for the purposes of brevity and clarity, in the following description the term “subtitles” is intended not only standard subtitles (for example, dialogue for a foreign movie) but other relevant graphic content as well.

Referring to the FIG. 1 , an uncompressed digital 3D movie 60 is analyzed and encoded by a first 3D video encoder 10 using a first set of encoding coefficients 10 A. For example, the digital 3D movie could be in a 1920×1080 p uncompressed image sequence for each left/right eye at the same native resolution as the source material (i.e. 1920×1080 p).

As part of the original analysis, a Z-axis depth map is generated either based on the content of whole frames or based on objects of interest within the frames. For example, as shown in FIG. 2 , a frame of scene of a movie captured in 3D may include three trees A, B and C disposed at different distances from the viewer, as well as a cloud D. As part of the encoding process, the encoder generates a Z-axis depth map. This depth map may be generated in various different forms. One typical technique is to generate a gray-scale version of the frame in which each object of interest is assigned a gray scale value dependent on its distance from the viewer. However, for the sake of clarity, the depth map presented in FIG. 3 as a linear axis, with the relative positions of objects of interest being represented by arrows A-D. Arrow S represents the position of the screen.

In one embodiment of the invention, the Z-axis depth map is generated by the encoder 10 and stored in a log 20 together with other compression or encoding parameters generated during the conversion process. The encoder also generates a first 3D encoded content which may be stored, recorded or distributed to customers via various distribution channels. The first 3D encoded content does not have subtitles.

The invention further includes an authoring tool 22 used to add subtitles to the 3D movie. For this purpose, the authoring tool 22 receives the subtitles and title position parameters indicative of where the subtitles should be positioned in the relevant frames. Preferably, subtitles can be positioned in 2D and 3D space with high quality while maintaining 3D viewing comfort. For this reason, not only 2D X and Y positions but also the position of the subtitles in the Z axis is fairly important. Various criteria or approaches can be used to accomplish this goal. Once this position is selected, then the authoring tool analyzes the Z depth map for each frame, allocates the position of the subtitles T and then generates appropriate reauthor control signals for superimposing the subtitles on the 3D frames at the designated location. The following is a partial list of some possible criteria dictating the position of the subtitles on the Z-axis.

a. At a fixed distance d 1 in front of the foremost object of interest. For example in FIG. 3 , the subtitles are positioned at T disposed at distance d 1 in front of tree A.

b. At a fixed distance in front of a particular object, unless, of course, the object or the subtitles are obstructed by other objects. For example, in FIG. 4 , the subtitles are positioned at a distance d 2 in front of tree B.

c. At a fixed distance from the screen S.

Moreover, the subtitles are positioned by the authoring tool not only along the Z-axis, but also anywhere along the X- and Y-axes. This position can be done either manually or systematically, by taking into consideration the positions of the other objects of interest.

A further consideration in determining the position of subtitles pertains to the actual screen size for which the 3D content is intended for. Generally, 3D content is originally composed and mastered for a specific screen size. However, when content created for one screen size is subsequently prepared, distributed, and viewed on a different screen size some qualitative anomalies can occur, e.g. miniaturization. Therefore when performing subsequent encoding of 3D movies by adding graphic subtitles, it may be desired to optimize the subtitles based on the target screen size. The original stereoscopic content (left/right video image pairs and/or left/right graphic overlay pairs) is fixed in its composition. The only parameter available during the re-encoding or re-authoring process is the horizontal shift, hereafter denoted as “offset,” between the left/right video image pairs or left/right graphic overlays.

For the purposes of describing this invention, it is assumed that the original offset corresponding to the original 3D content targeted for an initial screen size is zero. When that same 3D content is to be re-authored for a different target screen size a change in the offset value can be made. The value of the offset is a function of the original screen width (W 1 ), the new target screen width (W 2 ), the interoccular distance between the intended viewer's eyes e (e=2.5 inches for average human adult), and the horizontal resolution of the intended screen size (HorRes).

It is presently estimated that

Offset=(1− W 2/ W 1)× e ×(HorRes/ W 2)

This offset is the total horizontal shift required for the left and right stereoscopic images which will result in a shift of the subtitles along the Z-axis. Preferably, each image should be shifted in the respective direction, and therefore, each image should be shifted by ½ of the offset.

It should be understood that the above formula is a very basic formula and that it could be refined by using additional information that may be available to the authoring tool, including, but not limited to the intended viewer's distance to the screen, maximum 3D disparity desired, original set geometry used when acquiring the 3D content (e.g., distance of each object to camera), original camera parameters used to generate the 3D content (e.g. lens focal length, horizontal offset from optical axis), etc.

Returning to FIG. 1 , the depth map and other information is received from the log 20 by the authoring tool 22 . The authoring tool also receives the subtitle text and text positioning information (unless it is already preprogrammed in the authoring tool). If it is desired to tailor the position of the subtitles along the Z-axis to compensate for the intended screen size, then the authoring tool also receives the original screen size, the intended screen size and other information necessary for the shift along the Z-axis.

The authoring tool then generates a new set of parameters 23 for creating a second encoded 3D content.

A second encoder 70 (or the original encoder 10 ) receives either the original 3D uncompressed video master 60 or the first encoded content 12 and, using a second set of encoding coefficients 70 B to generate a second 3D encoded content with subtitles. This second encoded content is distributed by standard distribution.

In an alternate embodiment, the authoring tool generates encoding coefficients 70 B defining the position(s) of the subtitles and the text for the subtitles are then provided to the encoder 70 .

The second set of coefficients 70 B may include an original set of predetermined parameters, and/or some or all of the first encoding coefficients 10 A, and/or other parameters derived by the first encoder and stored in the log 20 . Importantly, the second set of coefficients 70 B also includes the parameters from the authoring tool 22 for superimposing the subtitles as discussed above.

Depending on the actual devices used as encoders 10 or 60 , the parameters from the authorizing device may not be directly compatible with the encoder 70 . In this case, the parameters 23 are fed to a mapping/processing element 24 for mapping the parameter into corresponding coefficients.

The parameters from the authoring tool ( 22 ) place the subtitles in the appropriate planes in 3D space for the specific release channel(s) at issue, e.g. Blu-ray Disc, or satellite broadcast, or Nintendo 3DS distribution.

The authoring tool is preferably one such as Sonic Solutions' Scenarist that has controls to set or calculate where to position the subtitles or other relevant graphics over the background video in 3D space using the depth map info depending upon the intended distribution channel.

In the embodiment described above, the depth map and other information derived from the log 20 is used to generate the parameters. Alternatively, if the log is not available, either the first 3D encoded content 12 or the 3D uncompressed video master 60 is analyzed by a separate analyzer to derive the information required by the authoring tool 22 and the optional mapping/processing element 24 .

In any case, the subtitles can be positioned in the X, Y and Z 3D space, either using one set of pre-set parameters for the entire duration of the video content, or dynamic positioning can be managed by setting different parameters for different scenes (or other relevant groups of frames), or manual positioning can be used to override pre-set parameters in exceptional cases. For example, prior to assigning a position for a title for a given scene, the apparent distance between relevant objects and the viewer is monitored during the scene and the shortest distance is determined. The position for the subtitle is set for that scene to be a predetermined, e.g., 2%, less than this shortest distance.

In this manner, video frames with subtitles or other relevant graphics are efficiently processed dynamically using the depth map obtained from the log of an original encoder or from a separate analyzer.

Numerous modifications may be made to the invention without departing from its scope as defined in the appended claims.