Apparatus and method for producing and storing multiple video streams

The present invention relates generally to digital imaging and more specifically to digital still cameras and digital camcorders.

Both digital camcorders and digital still cameras having a video mode produce a single digital video stream that may be readily edited and, if desired, stored on an optical medium such as a DVD. Some devices are also capable of operating in picture-in-picture (PIP) mode, in which an inset second view is embedded within a larger background view. In some devices, it is even possible to record video in this PIP mode. Thus, two different views of the scene are recorded simultaneously in a single video stream.

In some applications, however, it is advantageous to generate two or more different views as separate, independent video streams. Typically, this is accomplished through the use of multiple cameras. Using multiple cameras can be both expensive and cumbersome.

It is thus apparent that there is a need in the art for an improved apparatus and method for producing and storing multiple video streams.

A digital imaging device that produces and stores multiple video streams is provided. An associated method for capturing digital video in the digital imaging device is also provided.

Other aspects and advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the invention.

FIGS. 1A and 1B are functional block diagrams of a digital imaging device in accordance with an illustrative embodiment of the invention.

FIG. 2A is an illustration showing disjoint first and second regions of an imaging sensor array in accordance with an illustrative embodiment of the invention.

FIG. 2B is an illustration showing a first example of overlapping first and second regions of an imaging sensor array in accordance with an illustrative embodiment of the invention.

FIG. 2C is an illustration showing a second example of overlapping first and second regions of an imaging sensor array in accordance with an illustrative embodiment of the invention.

FIG. 3A is an illustration of sampling overlapping first and second regions of a CMOS imaging sensor array in accordance with an illustrative embodiment of the invention.

FIG. 3B is an illustration of sampling overlapping first and second regions of a CCD imaging sensor array in accordance with an illustrative embodiment of the invention.

FIG. 4 is a flowchart of the operation of the digital imaging device shown in FIGS. 1A and 1B in accordance with an illustrative embodiment of the invention.

Multiple video streams may be produced simultaneously in a digital imaging device by sampling multiple regions of a single imaging sensor array. Saving the multiple video streams in a memory provides a variety of post-production options. For example, portions of the video streams may be edited into a single combined video presentation, or two or more video streams may be combined in split-screen, picture-in-picture, or auto-fade fashion.

In a specific example, a parent records a child's piano recital using a digital camcorder. The digital camcorder produces and records two independent video streams simultaneously: (1) a wide-angle view showing the child playing the piano from 15 feet away and (2) a cropped view showing only the child's torso and hands at the keyboard. Since the two video streams are saved separately, video editing software may be used to create a single video presentation combining portions of both the wide-angle and close-up views for a more varied, interesting presentation. In some embodiments, this editing and combining of the two video streams may be performed manually using the video editing software. In other embodiments, the video editing software may automatically combine the two streams to create a single video stream. In automatically combining the two video streams, the video editing software may employ techniques well known in the art such as picture in picture, split screen, or auto-fade. In the case of auto-fade, for example, the video editing software may alternately fade between the first and second video streams at predetermined intervals.

FIGS. 1A and 1B are functional block diagrams of a digital imaging device 100 in accordance with an illustrative embodiment of the invention. Digital imaging device 100 may be a digital still camera with the capability of operating in video mode, a digital camcorder, or any similar device capable of capturing digital video. In FIG. 1A , controller 105 communicates over data bus 110 with imaging module 115 , memory 120 , and display 125 . Optical system 130 produces optical images that are converted to digital images by imaging module 115 . Memory 120 may further comprise random access memory (RAM) 135 and non-volatile memory 140 . Non-volatile memory 140 may be a CompactFlash™, Smart Media™, or Secure Digital™ card; magnetic tape; magnetic disk; or optical disc. As shown in FIG. 1B , imaging module 115 may further comprise an imaging sensor array 145 based on charge-coupled-device (CCD) or CMOS technology and sampling logic 150 . Sampling logic 150 controls the sequence and timing of reading (sampling) imaging sensor array 145 to produce digital video. Imaging module 115 typically also includes an analog-to-digital converter (A/D), a gain control, and a digital signal processor (DSP), as is well known in the art (not shown in FIG. 1B ). Display 125 may, among other things, provide a preview of the scene currently emanating from optical system 130 prior to recording, a view of what is being recorded during recording, and an on-screen user interface for controlling the operating modes and options of digital imaging device 100 .

FIGS. 2A-2C are illustrations showing examples of ways in which a first region 205 and a second region 210 of a single imaging sensor array 145 may be sampled to produce two independent video streams that may be stored in memory 120 . Two regions ( 205 and 210 ) are shown in FIGS. 2A-2C for simplicity. In other embodiments, more than two regions of imaging sensor array 145 may be sampled to produce more than two independent video streams. First region 205 may be sampled at a first resolution R 1 ; second region 210 , at a second resolution R 2 . R 1 and R 2 may be different or the same, depending on the application.

FIG. 2A shows an embodiment in which first region 205 and second region 210 are disjoint. FIG. 2B shows another embodiment in which first region 205 comprises a central portion of imaging sensor array 145 and second region 210 comprises a background view that spans the full dimensions of imaging sensor array 145 . FIG. 2C shows yet another embodiment in which first region 205 and second region 210 partially overlap.

FIGS. 2A-2C are merely illustrative. A variety of other configurations are possible in which both a first region 205 and a second region 210 of imaging sensor array 145 are sampled and the resulting video streams are stored in memory 120 .

Implementation of the invention may differ somewhat depending on whether imaging sensor array 145 is of the CCD or CMOS type. One advantage of a CMOS imaging sensor array is that sampling logic 150 may address it directly at the picture element (pixel) level, much like a RAM. However, some CMOS imaging sensor arrays may be read at a given pixel only once per exposure (destructive read).

FIG. 3A is an illustration showing how first and second regions 205 and 210 , respectively, may be sampled from a CMOS imaging sensor array 145 in a situation in which the two regions partially overlap. The portion of imaging sensor array 145 marked with “X's” may be ignored (not sampled). Within first region 205 , imaging sensor array 145 may be sampled at resolution R 1 . Within second region 210 , imaging sensor array 145 may be sampled at resolution R 2 . In the case of a destructive read, additional care must be taken with the overlapping portion of the two regions 205 and 210 . In the example shown in FIG. 3A , R 1 is assumed to be greater than R 2 . As those skilled in the art will recognize, a destructive read necessitates downsampling or scaling, in memory, the overlapping portion of first region 205 to produce the remainder of the video stream associated with second region 210 at (lower) resolution R 2 . In general, in the case of a destructive read and overlapping regions, the region to be sampled at higher resolution should be sampled before the region to be sampled at lower resolution.

FIG. 3B illustrates how the sampling of first region 205 and second region 210 differs if imaging sensor array 145 is a CCD. Since CCD's are typically sampled by shifting a row of sensor elements at a time into a set of horizontal shift registers (not shown in FIG. 3B ) and then reading out a row of pixels in succession from the shift registers, imaging sensor array 145 may be logically divided into “strips,” as shown in FIG. 3B . In this example, it is assumed that the rows of pixels are shifted downward in FIG. 3B into the set of horizontal shift registers. The top and bottom strips marked with “X's” are shifted into the set of horizontal shift registers but are skipped (not read into memory by sampling logic 150 ). The strip beside which “R 1 ” has been placed in FIG. 3B may be read out at resolution R 1 ; the strip beside which “R 2 ” has been placed, at resolution R 2 , where R 1 is greater than R 2 . The portion of the strip read out at R 1 and containing a portion of second region 210 may be downsampled to produce the remainder of the video stream associated with second region 210 , as explained above in connection with CMOS imaging sensor arrays. The pixels lying outside first region 205 and second region 210 in any given row of pixels may be skipped when pixels from the applicable row are read from the set of horizontal shift registers.

FIG. 4 is a flowchart of the operation of the digital imaging device 100 shown in FIGS. 1A and 1B in accordance with an illustrative embodiment of the invention. At 405 , sampling logic 150 samples first region 205 of imaging sensor array 145 at a first resolution to produce a first video stream. At 410 , sampling logic 150 samples second region 210 of imaging sensor array 145 at a second resolution to produce a second video stream. As mentioned previously, the first and second resolutions may, in some embodiments, be the same. At 415 , the first and second video streams are stored in memory 120 , which may, depending on the embodiment, be RAM 135 or non-volatile memory 140 , or both. At optional step 420 , the first and second video streams may be post-processed using suitable video editing software, as explained above. The process terminates at 425 .

In some embodiments, the first and second video streams may, at 415 , be stored in a single data file organized such that the two video streams are separable (separately readable). In other embodiments, the two video streams may be stored in separate data files at 415 . In either case, the resulting video streams may be post-processed using suitable video editing software.

The foregoing description of the present invention has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed, and other modifications and variations may be possible in light of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application to thereby enable others skilled in the art to best utilize the invention in various embodiments and various modifications as are suited to the particular use contemplated. It is intended that the appended claims be construed to include other alternative embodiments of the invention except insofar as limited by the prior art.