Estimating a pose of a camera for volume estimation

The present invention is directed to systems and methods for estimating a position (or pose) of a camera relative to a surface upon which an object rests in an image captured by that camera such that a volume can be estimated for that object.

Efforts have been expended in determining a volume from an image. For instance, “ Automatic Portion Estimation And Visual Refinement In Mobile Dietary Assessment ”, Woo et al, Proceedings of the SPIE, Vol. 7533, (January 2010), discloses a technique for processing an image acquired by a RGB camera to estimate a volume of a food item captured in that image using camera calibration, image processing, volumetric modeling, and interactive user refinements to obtain that estimation. In another method disclosed in: “ Measurement of Food Volume Based on Single 2- D Image without Conventional Camera Calibration ”, by Y. Yue, W. Jia and M. Sun, 34 th Annual Intl Conf. of IEEE EMBS, San Diego, Calif. USA, (August 2012), an image capture by a RGB camera is processed to estimate food volume. These approaches require the image to be acquired under tightly constrained conditions regarding the camera's position (or pose) relative to the food item or the use of a reference/calibration target with known physical dimensions, so that volume can be accurately estimated. Before this technology can be widely employed for general use, methods are needed for accurately estimating the pose of the camera used to acquire images of an object for which a volume is intended to be estimated.

Accordingly, what is needed in this art is a system and method for estimating a position (or pose) of a camera relative to a surface upon which an object rests in an image captured by that camera such that a volume can be estimated for that object.

The following U.S. Patents, U.S. Patent Applications, and Publications are incorporated herein in their entirety by reference.

“Contemporaneously Reconstructing Images Captured Of A Scene Illuminated With Unstructured And Structured Illumination Sources”, U.S. patent application Ser. No. 13/533,678, by Xu et al.

“3D Imaging Using Structured Light For Accurate Vehicle Occupancy Determination”, U.S. patent application Ser. No. 13/476,334, by Mestha et al.

What is disclosed is a system and method for estimating a position (or pose) of a camera relative to a surface upon which an object rests in an image captured by that camera such that a volume can be estimated for that object. In one embodiment, a matrix K is determined from parameters intrinsic to a camera used to capture image. An amount of a camera translation T is determined with respect to a set of real-world coordinates in (X,Y,Z). An amount of a camera rotation matrix R is determined from camera angles measured with respect to the real-world coordinates. A distance Z c of the camera at location (i,j) can then be estimated. A volume of the object in the image can be estimated from the camera pose parameters.

Features and advantages of the present method will become apparent from the following detailed description and accompanying drawings.

Advantages of the subject matter disclosed herein will become apparent from the following description in conjunction with the accompanying drawings, in which:

FIG. 1 which illustrates one example embodiment of the present method for estimating a pose of a camera relative to a surface upon which a 3D object rests for volume estimation purposes;

FIG. 2 shows a camera used to capture an image of an object resting on a horizontal surface such that a volume can be estimated for that object;

FIG. 3 shows the various camera angles (φ,θ,ω) relative to the X, Y and Z axis of a set of real-world reference coordinates; and

FIG. 4 illustrates a block diagram of one example system for implementing various aspects of the present method as described with respect to the flow diagram of FIG. 1 .

What is disclosed is a system and method for estimating a position (or pose) of a camera relative to a surface upon which an object rests in an image captured by that camera such that a volume can be estimated for that object.

Reference is being made to the flow diagram of FIG. 1 which illustrates one example embodiment of the present method. Flow processing starts at step 100 and immediately proceeds to step 102 .

Step 102 , determine a matrix K from parameters intrinsic to the camera used to capture at least one image of an object for volume estimation purposes. FIG. 2 shows a camera 202 used to capture an image of an object 203 resting on a flat surface 204 such that a volume can be estimated for that object. The methods hereof estimate a distance d c ( 205 ) of the camera from surface location (i,j) as measured by depth sensor 206 . Object 203 may be, for example, a food item. In one embodiment, matrix K is given by:

K = [ f 1 α ⁢ ⁢ f 1 c 1 0 f 2 c 2 0 0 1 ] ,

where f 1 and f 2 are focal lengths of a lens of the camera in each of two directions respectively, α is a non-zero skew of the camera's sensor, and c 1 and c 2 represent a center position of the camera in pixels. For a given camera, these parameters can be determined offline acquiring calibration targets and processing them via known camera calibration algorithms.

Step 104 , determine a camera translation vector T=[t X t Y t Z ] relative to a set of real-world reference coordinates in (X,Y,Z). Methods for translating 3D locations to a set of real world reference coordinates are well established.

Step 106 , determine a camera rotation matrix R from camera angles (φ,θ,ω) relative to the X, Y and Z axis, respectively, of the real-world reference coordinates. FIG. 3 shows the various camera angles (φ,θ,ω) relative to the X, Y and Z axis of the real-world reference coordinates. In one embodiment, matrix R is given by:

R = [ 1 0 0 0 cos ⁢ ⁢ φ - sin ⁢ ⁢ φ 0 sin ⁢ ⁢ φ cos ⁢ ⁢ φ ] ⁡ [ cos ⁢ ⁢ θ 0 - sin ⁢ ⁢ θ 0 1 0 sin ⁢ ⁢ θ 0 cos ⁢ ⁢ θ ] ⁡ [ cos ⁢ ⁢ ω - sin ⁢ ⁢ ω 0 sin ⁢ ⁢ ω cos ⁢ ⁢ ω 0 0 0 1 ] .

The amount of a camera translation and camera rotation is based upon depth values associated with pixels at locations on the surface of the object as measured by a depth sensor 206 or, as approximated, by a planar surface fitted to the measured depth values of the surface. In one embodiment, instructions are provided to a camera operator so that the camera rotation approximates a rotation angle any of: (90°, 90°, 0°), (90°, 0°, 90°), and (0°, 90°, 90°). Preferably, the images and depth maps acquired at such angles should be approximately centered relative to the position of the target object. More specifically, with respect to FIG. 2 , the user would be instructed to aim the camera such that Z c and d c are approximately equal for the (90°, 90°, 0°) case.

Step 108 , estimate a distance Z c from the camera to the table in the direction of the Z coordinate axis ( 208 ) of the camera, comprises solving the following system of equations:

Z c ⁡ [ i j 1 ] = KR ⁡ [ X Y Z ] + KT T .

The estimated distance can be communicated to a memory or to a remote device over a network for storage or for further processing. In this embodiment, processing stops.

Reference is now being made to FIG. 2 . Given the estimated intrinsic and pose camera parameters (K,R,T), one can now compute the real-world coordinates XYZ, X S (i,j),Y S (i,j),Z S (i,j), of the flat surface 204 at each pixel location (i,j) using the above equation and the depth output d s (i,j) from the depth sensor 206 . Pixel location 210 represents a location on the surface of surface 204 that is not occluded by object 203 . Pixel location 212 represents a location of a pixel on surface 204 that is occluded by object 203 . Note that the real-world coordinates of the points on the flat surface that are occluded by the object that rests on them can still be determined, for example by model-fitting a plane to the depth coordinates of the pixels that are not occluded by the object. Similarly, the real-world coordinates XYZ, X o (i,j),Y o (i,j),Z o (i,j), of the top surface of the object that rests on flat surface 204 can be determined in the same manner.

A volume V can then be estimated for the object in an image as follows:

V = ∑ ( i , j ) ∈ Ω ⁢ Δ ⁢ ⁢ Z ⁡ ( i , j ) · Δ ⁢ ⁢ X ⁡ ( i , j ) · Δ ⁢ ⁢ Y ⁡ ( i , j )

where Ω is a set of pixel locations of the object in the image; ΔZ(i,j)=Z s (i,j)−Z o (i,j) is an angle-corrected delta between the depth of the surface of the object d o (i,j) at location (i,j) as measured by a sensor and the depth of the surface d s (i,j) at the (i,j) position as determined by the aforementioned plane fitting; and ΔX(i,j)=X o (i,j)−X o (i−1,j), ΔY(i,j)=Y o (i,j)−Y o (i,j−1) are the physical sizes corresponding to the area imaged by pixel (i,j) in real-world coordinates, which are obtained by calculating the deltas along the X and Y axes between pixel (i,j) and its neighboring pixels. The angle correction delta ΔZ(i,j) is a function of cos φ, cos θ, cos ω and d s (i,j)−d o (i,j).

It should be appreciated that the above volume formula assumes that the surface on which the object lies is flat. The derivations hereof can be extended to a non-flat surface if an approximation of the shape of the surface is available (e.g. a quadratic or spherical bowl). Another assumption implicit to the above computations is that the object to be measured is in full contact with the surface. This constraint can be relaxed to accommodate broader object classes if an approximation of the object shape is available (e.g. symmetry relative to the Z axis).

It should be appreciated that the flow diagrams hereof are illustrative. One or more of the operative steps may be performed in a differing order. Other operations may be added, modified, enhanced, or consolidated. Variations thereof are intended to fall within the scope of the appended claims. Various aspects of the method of FIG. 1 may be implemented in software which may include object or object-oriented structures and techniques. Such software may comprise portable code that can be executed on a variety of platforms and further may be implemented partially or fully in hardware operating in conjunction with software.

Block Diagram of Networked System

Reference is now being made to FIG. 4 which illustrates a block diagram of one example system 400 capable of implementing various aspects of the present method as shown and described with respect to the flow diagram of FIG. 1 .

The embodiment of FIG. 4 is shown comprising a workstation 404 in communication with a camera 202 . The networked workstation 404 comprises a computer case housing a motherboard with a processor and memory, a communications link such as a network card, video card, an internal hard drive capable of reading/writing to machine readable media such as a floppy disk, optical disk, CD-ROM, DVD, magnetic tape, and other software and hardware needed to perform the functionality of a computing system. Workstation 401 has a monitor 403 for effectuating a user input or selection. The computer is also in communication with network 401 via a network communications interface (not shown). A user may use the keyboard 405 and monitor 403 of the workstation to identify or otherwise select areas of an image for processing or provide other user input required for the implementation hereof. Such information may also be retrieved by the workstation over a network 401 . It should be appreciated that workstation 404 has specialized software configured to display alphanumeric values, menus, scroll bars, dials, slideable bars, pull-down options, selectable buttons, and the like, for entering, selecting, modifying, and accepting any information needed for processing the image. Software to configure a user interface or any portion thereof to display/enter/accept data is generally customizable. In the embodiment of the networked system of FIG. 4 , computer workstation 404 and camera 202 are in communication with Processor 406 which executes machine readable program instructions for performing the methods hereof. Processor 406 , and the various modules, memory, and sub-processors thereof, may reside in the camera 202 or the workstation 404 .

Processor 406 is shown comprising a Memory 407 which provides storage for the processor and further stores data and mathematical formulas and representations to determine camera pose and for estimating a volume of an object in an image captured by the camera in accordance with the teachings hereof. All of the sub-processing modules of Processor 406 are in communication with Memory 407 via pathways not shown. Buffer 408 provides a means for storing information such as, for instance, regions of interest within an image being processed for volume estimation, mathematical values, the real-world reference coordinates in (X,Y,Z), distance measurements, rotation angles, the intrinsic parameters of the camera 202 such as focal lengths of the camera's lens, the value of a non-zero skew of the camera's sensor, and representations of a center position of the camera in pixels, and the like, as are needed processing. Matrix K Module 409 retrieves the camera's intrinsic parameters from Buffer 408 and facilitates a determination of matrix K which, in turn, is stored to Memory 407 . Translation T Module 410 facilitates a determination of translation vector T which, in turn, is stored to Memory 407 . Rotation Matrix R Module 411 facilitates a determination of rotation matrix R from camera angles (φ,θ,ω) relative to the X, Y and Z axis, respectively, of the real-world reference coordinates, as shown by way of illustration in FIG. 3 . The rotation matrix is stored to Memory 407 . Distance Estimator 412 estimates a distance Z, of the camera in a manner as disclosed herein. The estimated distance is communicated to Memory 407 or to a remote device over network 401 for storage or for further processing. Volume Estimation Module 413 receives the distance values for each location (i,j) and estimates a volume V for the object as provided herein.

Any of the computational values, results, interim values, distances, locations, and the like, can be viewed on the monitor 403 where a user can view the displayed values and make a selection from menu options displayed thereon. A user or technician of the system of FIG. 4 may use the graphical user interface of the workstation to identify regions of interest, set parameters, use a rubber-band box to select image portions and/or regions of images for processing. These selections may be stored and retrieved as needed. Default settings and initial parameters can be retrieved from Memory 407 , as needed. Although shown as a desktop computer, it should be appreciated that workstation 404 can be a laptop, a mainframe, a client/server, or a special purpose computer such as an ASIC, circuit board, dedicated processor, or the like. The embodiment of FIG. 4 is illustrative and may include other functionality known in the arts.

Any of the modules and processing units of FIG. 4 can be placed in communication with one or more devices over network 401 . Various modules may designate one or more components which may, in turn, comprise software and/or hardware designed to perform the intended function. A plurality of modules may collectively perform a single function. Each module may have a specialized processor capable of executing machine readable program instructions. A module may comprise a single piece of hardware such as an ASIC, electronic circuit, or special purpose processor. A plurality of modules may be executed by either a single special purpose computer system or a plurality of special purpose computer systems in parallel. Connections between modules include both physical and logical connections. Modules may further include one or more software/hardware modules which may further comprise an operating system, drivers, device controllers, and other apparatuses some or all of which may be connected via a network. It is also contemplated that one or more aspects of the present method may be implemented on a dedicated system and may be practiced in distributed computing environments where tasks are performed by remote devices that are linked through a network. It should be appreciated that some or all of the functionality performed by any of the modules or processing units of system 406 can be performed, in whole or in part, by a remote device placed in communication with system 400 over network 401 . The embodiment shown is illustrative and should not be viewed as limiting the scope of the appended claims. Although shown as a computer workstation, it should be appreciated that workstation 404 can be a laptop, tablet, mainframe, client/server, smartphone, or a special purpose computer such as an ASIC.

It will be appreciated that the above-disclosed and other features and functions, or alternatives thereof, may be desirably combined into many other different systems or applications. The teachings hereof can be implemented in hardware or software using any known or later developed systems, devices, and/or software by those skilled in the applicable art without undue experimentation from the functional description provided herein with a general knowledge of the relevant arts. One or more of the capabilities hereof can be emulated in a virtual environment or leverage off-the-shelf computer graphics software such as that in Windows or Java.

One or more aspects of the methods described herein are intended to be incorporated in an article of manufacture, including one or more computer program products, having computer usable or machine readable media. The article of manufacture may be included on at least one storage device readable by a machine architecture embodying executable program instructions capable of performing the methodology described herein. The article of manufacture may be included as part of an operating system, a plug-in, or may be shipped, sold, leased, or otherwise provided separately either alone or as part of an add-on, update, upgrade, or product suite. It will be appreciated that various of the above-disclosed and other features and functions, or alternatives thereof, may be combined into other systems or applications.

Various presently unforeseen or unanticipated alternatives, modifications, variations, or improvements therein may become apparent and/or subsequently made by those skilled in the art which are also intended to be encompassed by the following claims. Accordingly, the embodiments set forth above are considered to be illustrative and not limiting. Changes to the above-described embodiments may be made without departing from the spirit and scope of the invention. The teachings of any printed publications including patents and patent applications, are each separately hereby incorporated by reference in their entirety.