Event management for camera systems

Conventional film and more recently, digital cameras, are widely commercially available, ranging both in price and in operation from sophisticated single lens reflex (SLR) cameras used by professional photographers to inexpensive “point-and-shoot” cameras that nearly anyone can use with relative ease. Unlike conventional film cameras, however, digital cameras typically collect frame data for displaying a “live view” for the user and/or for use in various processes embodied in firmware for the digital camera.

For purposes of illustration, frame data may be used by the firmware for auto-exposure calculations. The output may be used by another process to adjust exposure settings. More frame data may then be gathered using the new exposure settings to display a revised “live view” for the user and/or continued auto-exposure calculations. Of course gathering frame data and displaying the “live view” for the user typically occurs very fast, e.g., generating “live view” images on the order of 24 to 30 times per second.

In any event, the frame data may be needed by more than one process at a time and/or the results of various processes may be needed for another process. For example, frame data may be needed by both auto-exposure calculations and auto-focus calculations in addition, output from the auto-exposure calculations may also be needed for display calculations. Accordingly, firmware for the camera is written such that the frame data and results cascade from one process to the next.

Such firmware designs may result in slower operations (e.g., all processes may stall while waiting for one of the processes to complete) in addition, if a process is changed or a new process is added, the firmware may need to be rewritten to accommodate the change and/or new process.

FIG. 1 is a block diagram of an exemplary camera system which may implement event management.

FIG. 2 is a timing diagram illustrating an exemplary process flow for various processes executing on a camera system.

FIG. 3 is a functional block diagram illustrating exemplary event management for camera systems.

FIG. 4 is a flowchart illustrating exemplary operations which may be implemented in camera systems for event management.

Systems and methods described herein disclose event management for camera systems. Exemplary systems and methods arrange processes as individual program code modules which may be executed concurrently and independently of one another. An event manager coordinates the individual program code modules by notifying the program code modules when an event occurs (e.g., frame data is available from the sensor device and/or results of a calculation are available). The program code modules execute if the event occurs.

Individual modules may then publish results of the process to the event manager, which in turn publishes an event notifying other modules that the results are available. Accordingly, the program code modules may be provided independent of one another and may be added/changed without having to rewrite the firmware.

In exemplary embodiments, a closed-loop data path is provided between the event manager, the program code modules, and the sensor device. The closed-loop data path receives output from the executing program code modules and feeds back the output to the sensor device for changing the frame data for the event manager. For example, output from the executing program code modules may indicate the frame data needs to be brighter or darker for the other processes. Accordingly, the aperture settings on the camera system may be changed to generate brighter or darker frame data.

In other exemplary embodiments, the program code modules may verify that the frame data has not changed during execution (e.g., by execution of another program code module). If the frame data changed during execution, the program code module may be re-executed so that the output is based on the most recently available frame data.

FIG. 1 is a block diagram of an exemplary camera system 100 which may implement event management. The exemplary camera system 100 may be implemented as digital still-photo and/or video cameras, now known or that may be later developed.

Exemplary camera system 100 may include a lens 120 positioned in the camera system 100 to focus light 130 reflected from one or more objects 140 in a scene 145 onto an image sensor 150 when a shutter 155 is open (e.g., for image exposure). Exemplary lens 150 may be any suitable lens which focuses light 130 reflected from the scene 145 onto image sensor 150 .

Exemplary image sensor 150 may be implemented as a plurality of photosensitive cells, each of which builds-up or accumulates an electrical charge in response to exposure to light. The accumulated electrical charge for any given pixel is proportional to the intensity and duration of the light exposure. Exemplary image sensor 150 may include, but is not limited to, a charge-coupled device (CCD), or a complementary metal oxide semiconductor (CMOS) sensor.

Camera system 100 may also include pre-image logic 160 . In digital cameras, the pre-image logic 160 receives electrical signals from the image sensor 150 representative of the light 130 captured by the image sensor 150 during exposure to generate a digital pre-image of the scene 145 .

Using a pre-image enables the camera system to perform various processes (e.g., anti-aliasing, auto-focus, image stabilizing, detecting white balance, etc.) independent of any knowledge about the lens quality, zoom, and/or aperture settings. The pre-image(s) may be one or more of the pre-images already being implemented by the camera for focusing, auto-exposure, pre-flash calculations, and/or the pre-image(s) may be obtained specifically for displaying to the user.

Camera system 100 may also include processing logic 170 . Processing logic 170 may receive the pre-image data from pre-image logic 160 and perform various calculations or processes on the pre-image (or “frame data”). Exemplary processing logic 170 may include processes for, e.g., anti-aliasing, auto-focus, image characterizing, generating a visual image for the user, etc.

Processing logic 170 may receive input from other sources in addition to the pre-image logic 160 . For example, camera settings data 180 may also be provided as input to the processing logic 170 . Output from the processing logic 170 may be used by one or more other program code modules in the processing logic 170 and/or by system devices (e.g., actuator drives 190 for image stabilizing, auto-focus, and/or auto-exposure). In an exemplary embodiment, the processing logic 170 may include a plurality of processes implemented as individual program code modules to handle process flow in the camera system 100 , as described in more detail below.

Before continuing, it is noted that shutters, image sensors, and other devices and logic, such as those illustrated in FIG. 1 and others not shown, are well-understood in the camera and photography arts. These components may be readily provided for camera system 100 by those having ordinary skill in the art after becoming familiar with the teachings herein, and therefore further description is not necessary.

It is also noted that the camera system 100 shown and described above with reference to FIG. 1 is merely exemplary of a camera system which may implement event management. The systems and methods described herein are not intended to be limited only to use with the camera system 100 . Other embodiments of camera systems which may implement event management are also contemplated.

FIG. 2 is a timing diagram 200 illustrating an exemplary process flow for various processes executing on a camera system (e.g., the camera system 100 shown in FIG. 1 ). A timeline 205 is illustrated along the bottom portion of the drawing indicating various times t 1 -t 5 during the process flow.

At time t 1 , the image sensor 210 may generate frame data 215 (e.g., a pre-image). The frame data 215 is available to one or more of the processes. Process A and Process C (processes 220 a and 220 c ) may utilize the frame data at time t 2 , e.g., immediately when the frame data is available. Process B (process 220 b ), however, needs both the frame data 215 and the results 230 a from Process A to execute. At time t 3 , the results 230 a from executing Process A are available and may be delivered to Process B (as illustrated by arrow 235 ) so that Process B may be executed. At time t 4 , the results 230 c from executing Process C are available, and at time t 5 the results 230 b from executing Process B are available.

It is noted that because the processes 220 a - c are implemented as individual program code modules, Process A and Process C may be begin simultaneously (e.g., at time t 2 ). Frame data 215 may be provided to both Process A and Process C without having to wait for one of the other processes 240 a - c to complete. Likewise, when the results 230 a from executing Process A are available, Process B can start without having to wait for Process C to complete. In addition, the results 230 a - c may be made available to other processes 240 a - c on completion of the respective processes 220 a - c without having to wait for other process to complete.

It is noted that the process flow discussed above with reference to FIG. 2 is provided only for purposes of illustration and not intended to be limiting. Other processes may be executing simultaneously and/or at other times. In addition, the process flow may be much more complicated than what is illustrated in FIG. 2 , e.g., with results, frame data, and revised or updated frame data from multiple processes being made available at different stages during the execution of one or more processes. Exemplary functional components for managing process flow as illustrated in FIG. 2 are discussed in more detail below with reference to FIG. 3 .

FIG. 3 is a functional block diagram 300 illustrating exemplary event management for camera systems (e.g., the camera system 100 shown in FIG. 1 ). The functional components may be implemented in program code (e.g., the processing logic 170 shown in FIG. 1 ) residing in memory and executable by a processor (e.g., for camera system 100 ) and may include individual program code modules for executing various processes 310 and 312 (referred to generally as process or processes 310 ).

The camera system may include an image sensor 320 (e.g., the image sensor 150 in FIG. 1 ) which may generate frame data 330 (e.g., via the pre-image logic 160 in FIG. 1 ). Frame data 330 may be provided to an optional frame producer 340 . The frame producer 340 may store the frame data 330 in an optional library 350 , and also may notify an event manager 360 that the frame data 330 is available.

The library 350 may be implemented as a data structure (e.g., residing in the camera system's memory) and program code for managing the data structure. Library 350 provides storage for the frame data and/or process results so that the frame data and/or process results may be accessed by the processes 310 independent and/or concurrently with one another (e.g., as illustrated in one example in FIG. 2 ). If the library 350 is implemented, frame data and/or process results may be made available without having to “lock” the data during execution of one or more of the processes. In addition, the data does not need to be provided in separate buffers for use by the program code modules. Instead, any one or more of the program code modules may retrieve separate instances of the data at any time independent of the other program code modules.

It is noted that the frame producer 340 and event manager 360 may be implemented as program code (e.g., firmware) executing on a processor for the camera system. In other embodiments, it is noted that the function of the frame producer 340 and event manager 360 may be combined, e.g., as a single firmware component.

Any number of processes (e.g., indicated by Process n in FIG. 3 ) may be implemented as separate or independent program code modules. Accordingly, each process is modular and can be executed, modified, added, and/or replaced without affecting the other processes. The processes are coordinated via the event manager 360 . This coordination is handled by an event subscription/publication model, wherein the event may be, e.g., frame data 330 being available from the image sensor 320 and/or results from executing one or more process 310 being available.

A process 310 may subscribe to one or more events by issuing a subscription 370 to the event manager 360 . For example, the process 310 may subscribe to an event when the program code module is first added to the camera system. In another example, the process 310 may subscribe to an event each time the camera system boots. The subscriptions are registered with the event manager 360 , and each time a subscribed event occurs, the event manager 360 issues or publishes an event notification to the processes 310 subscribing to that event. The event manager 360 may then publish event notifications 375 to the subscribed processes 310 each time the subscribed event occurs.

For purposes of illustration, process 310 may be an auto-focus process which needs frame data for every pre-image. Accordingly, process 310 may subscribe to frame data for every pre-image. When frame data 330 representing every pre-image is available, the event manager 360 may publish an event notification 375 for every pre-image to process 310 indicating that the frame data 330 is available.

In another illustration, process 310 may be an image stabilizing process which only requires frame data representing every other pre-image. Accordingly, process 310 may subscribe to frame data representing every other pre-image. When frame data 330 for every other pre-image is available, the event manager 360 may publish an event notification 375 for every other pre-image to process 310 indicating that the frame data 330 is available.

One or more of the processes 310 may also return process results 380 to the event manager 360 during or after execution of the process 310 . The event manager 360 may then notify one or more of the other processes 310 that the process results are available (e.g., via an event notification to a process subscribing to those process results). Optionally, process results 380 may also be stored in the library 350 for retrieval by one or more of the processes 310 .

In an exemplary embodiment, the event notification 375 may notify the subscribed processes 310 if frame data 330 and/or process results 380 are available. Optionally, the event notification 375 may also identify the location of the frame data 330 and/or process results 380 , e.g., in library 350 . The process 310 may then access the frame data 330 and/or process results 380 , e.g., by issuing a call 390 to retrieve the frame data 330 and/or process results 380 (illustrated generally as data 395 in FIG. 3 ) from the library 350 . In another exemplary embodiment, the event notification 375 may also include an instance of the frame data 330 and/or process results 380 so that the processes 310 do not need to separately access the library 350 , and can begin executing without having to make a separate call 390 to the library 350 .

Each process 310 may subscribe to one or more events. In addition, an event notification 375 may be the occurrence of more than one event. For example, an event notification 375 may be issued after frame data representing every third pre-image is available, and the results from another process n 312 are also available. In another example, an event notification 375 may be issued including multiple instances of frame data 330 and/or process results 380 . The events and the event notifications will depend at least to some extent on design considerations, such as, but not limited to, the frame data 330 and/or process results 380 which are needed by the process or processes 310 to execute.

It is also noted that the processes do not need to begin execution immediately upon receipt of an event notification 320 . Execution timing will also depend at least to some extent on design considerations, such as, but not limited to, when the process results are needed and/or whether execution of the process would consume too many resources of the camera system at any given time.

In exemplary embodiments, a closed-loop data path is provided between the event manager 360 , the program code modules (e.g., process 310 ), and the image sensor 320 , e.g., as illustrated by the event notification path between event manager 360 and process 310 and the return data path 311 to image sensor 320 . The closed-loop data path receives output from the executing program code modules and feeds back the output to the sensor device for changing the frame data for the event manager. For example, output from the executing auto-focus program code module(s) may indicate the frame data needs additional focusing. Accordingly, the focus point settings on the camera system may be changed to focus the frame data.

In other exemplary embodiments, the program code modules may verify that the frame data has not changed during execution (e.g., by execution of another program code module). If the frame data changed during execution of a process (e.g., was changed by the auto-focus process while a brightness processes were executing), the program code module may be re-executed so that the output (e.g., from the brightness processes) is based on the most recently available frame data.

Before continuing, it is noted that the functional components shown in FIG. 3 and described above are not intended to be limiting. Other functional components (not shown) may also be provided and are not limited to those shown and described herein.

FIG. 4 is a flowchart illustrating exemplary operations which may be implemented in camera systems (e.g., the camera system 100 shown in FIG. 1 ) for event management. Operations 400 may be embodied as logic instructions on one or more computer-readable medium. When executed on a processor, the logic instructions cause a general purpose computing device to be programmed as a special-purpose machine that implements the described operations. In an exemplary implementation, the components and connections depicted in the figures may be used for event management in camera systems.

In operation 410 , frame data and/or process results which are available are identified. For example, the operation may identify frame data available from the pre-image logic and/or the operation may identify process results available from the execution of other processes. In operation 420 , an event notification is published which identifies the available frame data and/or available process results. Optionally, the event may also include an instance of the frame data and/or process results.

In operation 430 , the program code modules may subscribe to one or more of the events. The program code modules may subscribe to events identifying frame data and/or the program code modules may subscribe to events identifying the process results. It is noted that although a program code module may subscribe to all of the events identifying frame data and/or process results, the program code modules do not need to subscribe to all events.

In operation 440 , a determination is made whether the subscribed event has been published. If not, operations may return to operation 410 to continue identifying available frame data and/or process results. If the subscribed event is published, then a process (or processes) corresponding to the published event may be executed in operation 450 .

The operations shown and described with reference to FIG. 4 are provided to illustrate exemplary implementations of event management in camera systems. The operations are not limited to the ordering shown. In addition, still other operations may also be implemented as will be readily apparent to those having ordinary skill in the art after becoming familiar with the teachings herein.

It is noted that the exemplary embodiments shown and described are provided for purposes of illustration and are not intended to be limiting. Still other embodiments are also contemplated for event management in camera systems.