USPatentGranted
B1

Radar imaging on a mobile computing device

Granted 3 Nov 2020 · 2 office actions

Current assignee: Google · originally Alphabet Inc.

Law firm: Law firm · Log in to unlock

Attorney: Attorney · Log in to unlock

Inventors: Ivan Poupyrev, Jaime Lien · Examiner: Erin F Heard · AU 3648 · TC 3600

Application
15/142,471
filed 29 Apr 2016
Publication
Not published
not published
Patent· this page
US 10,823,841
granted 3 Nov 2020

Life of the patent

12 dated events
⤢ drag to zoom20162018202020222024202620282030203220342036ProsecutionOwnershipTerm & fees
ProsecutionOwnershipTerm & feeshover for detail · click to open

Abstract

Systems and methods of capturing images are disclosed. For instance, a plurality of position signals associated with a mobile computing device can be received, the plurality of position signals can be obtained at least in part using one or more sensors implemented within the mobile computing device. A relative motion between the mobile computing device and a scattering point associated with a target can be determined. A plurality of return signals reflected from the scattering point can be received. Each return signal can correspond to a pulse transmitted by the mobile computing device. A target response associated with the scattering point can be determined based at least in part on the relative motion between the mobile computing device and the scattering point.

Description

11 parts
›PRIORITY CLAIM

The present application is based on and claims priority to U.S. Provisional Application 62/237,975 having a filing date of Oct. 6, 2015, which is incorporated by reference herein.

›FIELD

The present disclosure relates generally to capturing images, and more particularly to generating images using synthetic aperture radar imaging techniques on a mobile computing device.

›BACKGROUND

Synthetic aperture radar technology is typically implemented in platforms such as aircrafts, satellites, and/or fixed track moving radar platforms. In particular, such synthetic aperture radar implementations are typically designed for systems wherein a motion of the platform is very precisely constrained to predetermined trajectories and/or precisely measured with GPS. Further, such synthetic aperture radar implementations typically require a large amount of space due to the size of the radar hardware (e.g. circuitry, antennas, etc.). Some synthetic aperture radar implementations can be configured to capture images of a target by simulating a synthetic aperture based on a relative motion between the target and the radar. Such imaging implementations typically generate image through post-processing techniques based at least in part on the predetermined platform trajectory.

Such systems do not meet the constraints of a consumer mobile computing device having limited size, cost, and processing resources. In addition, such a mobile computing device may not have a predetermined trajectory that can be used to create the synthetic aperture.

›SUMMARY

Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

One example aspect of the present disclosure is directed to a computer-implemented method of capturing images using a mobile computing device. The method includes receiving, by a mobile computing device, a plurality of position signals associated with the mobile computing device. The plurality of position signals are obtained at least in part using one or more sensors implemented within the mobile computing device. The method further includes determining, by the mobile computing device, a relative motion between the mobile computing device and a scattering point associated with the target scene based at least in part on the plurality of position signals. The method further includes receiving, by the mobile computing device, a plurality of return signals reflected from the scattering point. Each return signal corresponds to a pulse transmitted by the mobile computing device while the mobile computing device is in view of the scattering point. The method further includes determining, by the mobile computing device, a target response associated the scattering point based at least in part on the relative motion between the mobile computing device and the scattering point.

Other example aspects of the present disclosure are directed to systems, apparatus, tangible, non-transitory computer-readable media, user interfaces, memory devices, and electronic devices for capturing synthetic aperture radar images using a mobile computing device.

These and other features, aspects and advantages of various embodiments will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the related principles.

›BRIEF DESCRIPTION OF THE DRAWINGS

Detailed discussion of embodiments directed to one of ordinary skill in the art are set forth in the specification, which makes reference to the appended figures, in which:

FIG. 1 depicts an example system for capturing images according to example embodiments of the present disclosure;

FIG. 2 depicts an example system for capturing images according to example embodiments of the present disclosure;

FIG. 3 depicts an example antenna configuration according to example embodiments of the present disclosure;

FIG. 4 depicts a flow diagram of an example method of capturing images according to example embodiments of the present disclosure; and

FIG. 5 depicts an example system according to example embodiments of the present disclosure.

›DETAILED DESCRIPTION · 1 of 6

Reference now will be made in detail to embodiments, one or more examples of which are illustrated in the drawings. Each example is provided by way of explanation of the embodiments, not limitation of the present disclosure. In fact, it will be apparent to those skilled in the art that various modifications and variations can be made to the embodiments without departing from the scope or spirit of the present disclosure. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that aspects of the present disclosure cover such modifications and variations.

Example aspects of the present disclosure are directed to determining synthetic aperture radar images by a mobile computing device, such as a smartphone, tablet computing device, wearable computing device, laptop computing device, or any other suitable computing device capable of being carried by a user while in operation. Synthetic aperture radar images are typically captured from radar platforms having a substantially fixed and/or predetermined path or trajectory, such as an airplane, satellite, or a fixed track moving radar system. Such images can be captured by simulating a synthesized antenna aperture by capturing radar data associated with a target at a number of positions and times as the platform travels along its trajectory. The simulated aperture can have a larger size than the physical aperture of the antenna. The captured data can then be combined to form an image. The fixed or predetermined motion of the platform can be used to account for a migration of the target within the captured data at each of the different data capture positions.

Synthetic aperture imaging techniques are not typically implemented within consumer mobile devices because the motion of the device in capturing an image is not fixed or predetermined. In this manner, the migration of the target within the captured data at each position does not typically follow a known or predetermined pattern, and is not easily determined. In addition, computing, hardware, and/or size constraints associated with mobile computing devices can preclude implementation of such imaging techniques.

According to example aspects of the present disclosure, a motion of a mobile device can be determined in real time as the mobile device transmits and receives radar data with respect to a target scene, and the determined motion can be used to update a target response associated with received radar data obtained by the mobile device. In particular, data associated with a plurality of locations of a mobile computing device can be received. The data can be determined at least in part by one or more sensors or other components implemented within the mobile computing device. The plurality of locations of the mobile device can correspond to a movement of the mobile device along a trajectory proximate a target scene by a user during an image capture period. One or more relative positions between the mobile device and the target scene can be determined based at least in part on the data associated with the plurality of locations. As the mobile device is being moved along the trajectory, the mobile device can transmit a sequence of pulses and receive one or more reflected signals indicative of the target scene. The mobile device can then determine one or more radar images by compensating for the trajectory of the mobile device in real-time. In various implementations, the image can be a two-dimensional (2D) image, a three-dimensional image (3D), and/or a see-through-the-wall (STTW) image determined using one or more synthetic aperture radar processing techniques.

More particularly, the mobile device can include a radar module having one or more antenna elements configured to transmit a sequence of pulses and/or to receive return signals from the target scene. The mobile device can further be configured to determine positional information of the mobile device using one or more accelerometers, gyroscopes, depth cameras, optical cameras, ranging base stations, and/or various other suitable components. Upon initiation of an image capture period, for instance, in response to an input from a user, the mobile device can begin transmitting a periodic sequence of modulated pulses as the user moves the mobile device on a trajectory proximate the target scene. As another example, during the image capture period, the target scene may be moved on a trajectory proximate the mobile device. Such relative motion between the mobile device and the target scene can be used to simulate a synthetic antenna aperture that is larger than the physical antenna aperture of the mobile device. A plurality of return signals can be received corresponding to time-delayed versions of the transmitted signals. The plurality of return signals can correspond to a superposition of reflections from all scattering points within the antenna field of view. For instance, the return signals can correspond to a sum of the contribution of all scattering points in the target scene. In some implementations, the return signals can include amplitude data and/or phase data associated with the return signals. Return signals can be received corresponding to each transmitted pulse. In this manner, data indicative of a particular scattering point of the target scene can be received multiple times as the mobile device moves proximate the target scene.

One or more target responses associated with the target scene can be determined in real-time based at least in part on the relative motion between the mobile device and the target scene. A target response can be indicative of reflected energy received by the mobile device, and can vary based at least in part on the relative range and velocity between the target scene and the mobile device. In some implementations, the range or distance from the mobile device to the target scene (e.g., to one or more scattering points within the target scene) can vary with each pulse transmitted and received by the mobile device. In particular, the reflections from the scattering points within the antenna field of view may be modulated by the relative range and velocity between the various scattering points and the mobile device as the mobile device moves relative to the target scene. In this manner, as the mobile device moves relative to the target scene, the target response can be updated to compensate for the modulated return signal.

›DETAILED DESCRIPTION · 2 of 6

In particular, the target response can be updated based at least in part on a relative position and/or velocity between the mobile device and the target scene. As indicated, as the mobile device moves proximate the target scene during an imaging period, a trajectory and/or velocity of the mobile device can be monitored. For instance, in some implementations, the movement of the mobile device by the user can be an ad hoc movement that does not follow a predefined motion or path. In this manner, the relative motion between the target scene and the mobile device during an imaging period is not known prior to the imaging period. In some implementations, the relative trajectory and/or velocity of the mobile device and/or target can be determined using one or more onboard position sensors, such as one or more accelerometers, gyroscopes, depth cameras, optical cameras, etc. in conjunction with the return signals received by the mobile device.

The relative trajectory and/or velocity between the mobile device and the target can be used to determine a migration of an individual scattering point. In particular, as the scattering point moves through the aperture of the receiving antenna of the mobile device over a plurality of pulses during an imaging period, the range between the scattering point and the aperture varies. The determined relative trajectory and/or velocity can be used to determine the degree of variation of the range. The mobile device can compensate for such range variations to reduce or eliminate the range variation based at least in part on the determined degree of variation. The target response associated with the return signals can be updated to reflect the compensated return signals. In particular, the target response can be determined by combining the compensated return signals for each scattering point in the target scene. In some implementations, one or more synthetic aperture radar processing techniques can be used to compensate the return signals and/or to determine the updated target response. For instance, one or more pulse compression techniques, range-Doppler techniques, range and/or Doppler migration correction techniques, Doppler mapping techniques can be used.

In some implementations, such processing techniques can further be used to generate an image of the radio frequency (RF) reflectivity of the target scene. Such generated image can have a higher resolution than an image generated using real aperture imaging techniques. In various implementations, the generated image can be a 2D image providing range and azimuth information associated with the target scene, or a 3D image providing range, azimuth, and elevation information associated with the target scene. In some implementations, the generated image may be a STTW image of one or more objects located behind a wall or other barrier. The type of image that is generated can be based on the motion or trajectory of the mobile device during the imaging period. In particular, the motion of the mobile device can be used to simulate an antenna aperture suitable for capturing different image types. For instance, a trajectory of the mobile device having only transverse motion relative to the target scene can be suitable for generating a 2D image. As another example, a trajectory of the mobile device having transverse motion and longitudinal motion relative to the target scene can be suitable for generating a 3D image. In this manner, the trajectory of the mobile device during an imaging period can be chosen by a user to generate a desired image type.

As an example, a user of a mobile computing device can initiate an imaging process, for instance, through interaction with a user interface associated with the user device. The mobile device can then prompt the user to move the mobile device with respect to a target scene of which the user desires to capture an image. As the user moves the mobile device with respect to the target scene, the mobile device can begin transmitting a sequence of pulses and receiving return data associated with a target scene. The mobile device can further determine a relative motion between the mobile device and target scene. The mobile device can then generate one or more images of the target scene by combining the return data received as the mobile device moved with respect to the target scene, and compensating for the determined relative motion. The mobile device can provide the image for display on the user interface of the mobile device.

With reference now to the figures, example embodiments of the present disclosure will be discussed in more detail. For instance, FIG. 1 depicts an example system 100 for capturing a synthetic aperture radar image according to example embodiments of the present disclosure. System 100 includes an image capture device 102 configured to capture one or more images of a target 104 . Image capture device can be any suitable device configured to capture one or more radar images. For instance, image capture device 102 can be a standalone image capture device, or can be implemented or integrated within mobile computing device. For instance, the mobile computing device can be any suitable mobile computing device, such as a smartphone, tablet, laptop, wearable computing device, or other suitable mobile computing device capable of being carried by a user while in operation. Image capture device 102 includes one or more antenna elements 106 , one or more position sensors 108 , and a SAR controller 110 . Image capture device 102 can be configured to generate energy signals, such as a modulated periodic signal (e.g. pulse train), to be transmitted in a general direction of target 104 . In some implementations, the generated energy signals can be frequency-modulated continuous wave (FMCW) energy signals having various frequencies between about 50 GHz and about 70 GHz. Antenna element(s) 106 can be configured to broadcast or transmit the generated energy signals into space in the direction of target 104 , and to receive one or more reflected signals indicative of target 104 . For instance, the energy signals can be periodically transmitted at a pulse repetition frequency (PRF) rate. During each transmission, one or more receiver antenna elements can simultaneously be powered. One or more intercepted RF signals can, for instance, be mixed to an intermediate frequency and converted to discrete samples that are provided to SAR controller 110 . In some implementations, image capture device 102 can be configured amplify and demodulate the received signals, and to provide the amplified and demodulated signals to SAR controller 110 .

›DETAILED DESCRIPTION · 3 of 6

For instance, FIG. 2 depicts an example image data acquisition process 140 according to example embodiments of the present disclosure. In particular, image capture device 102 can be configured to transmit energy signals (e.g. original waves) 142 towards target 104 , and to intercept or receive return signals (e.g. reflected waves) 144 from target 104 . In this manner, image capture device 102 can both emit electromagnetic radio frequency (RF) waves, and receive reflected waves. In particular, energy signals can be propagated in a straight line at the speed of light in one or more directions associated with an antenna beam pattern of antenna element(s) 106 . Objects within the propagation path (e.g. target 104 ) can either absorb the electromagnetic energy or scatter the electromagnetic energy, which results in a change of wave direction.

Electromagnetic radiation reflected coherently back in the direction of image capture device 102 can be intercepted by antenna element(s) 106 (e.g. one or more receiving antenna elements). Such received return signal is a superposition of reflections from a plurality of scattering points within the field of view of image capture device 102 . As indicated data indicative of the received return signals can be provided to SAR controller 110 for processing to generate one or more images.

As indicated above, image capture device 102 can be configured to simulate a synthesized antenna aperture using synthetic aperture radar techniques. For instance, during an image capture process or sequence, a user of image capture device 102 can facilitate a relative motion 112 between image capture device 102 and target 104 . For instance, a user may facilitate relative motion 112 by moving image capture device 102 in various directions and distances with respect to target 104 . For instance, relative motion 112 can be a non-predefined motion that has not set path or trajectory. In this manner, relative motion 112 can include a user defined motion that is not known prior to initiation of relative motion 112 . In some implementations, the user can rotate image capture device 102 about one or more axes during the image capture sequence, such that image capture device 102 and/or an antenna beam associated with image capture device 102 is continuously oriented in a general direction of target 102 during the image capture sequence. As another example, the user can facilitate relative motion 112 by maintaining image capture device 102 in a substantially constant position while target 104 moves in one or more non-predefined directions relative to image capture device 102 .

Referring back to FIG. 1 , antenna element(s) 106 can include various suitable antenna element types. In addition, antenna element(s) 106 can include various suitable numbers of antenna elements configured in various suitable manners. For instance, in some implementations, antenna element(s) 106 can include one or more microstrip antennas configured as a steered or unsteered array. For instance, antenna element(s) 106 can include multiple transmitter-receiver pairs configured to produce a broad antenna beam that can be digitally steered with respect to target 104 . As another example, a single transmitter-receiver pair or an unsteered combination of antenna elements can be enabled to produce one-dimensional ranging and tracking along a line of sight associated with image capture device 102 . In this manner, one or more transmitter-receiver pairs can be configured to provide one or more channels of data associated with the received signals.

As an example, FIG. 3 depicts an example antenna configuration 120 according to example embodiments of the present disclosure. Antenna configuration 120 can correspond to antenna element(s) 106 of FIG. 1 . As shown, antenna configuration 120 can include transmitting antennas 122 and 124 , and receiving antennas 126 , 128 , 130 , and 132 . Antenna configuration 120 further includes a feed network 134 configured to transmit energy signals to transmitting antennas 122 , 124 , and from receiving antennas 126 - 130 . Antenna configuration 120 can be configured as a steered array (e.g. phased array) configured to electronically or digitally steer an associated radar beam with respect to target 104 . Each transmitting antenna 122 , 124 can be paired with each receiving antenna 126 - 132 to provide eight channels of data, which can be coherently combined, for instance, in a digital domain using one or more suitable beam-forming algorithms to enable 3D spatial discrimination.

Example antenna configuration 120 is provided for illustrative purposes only. As indicated, it will be appreciated that various other suitable antenna configurations can be used without deviating from the scope of the present disclosure. For instance, various suitable antenna configurations can be used having various suitable antenna element types, numbers, and/or arrangements, and having various suitable feed networks.

Referring back to FIG. 1 , image capture device 102 can further include one or more position sensor(s) 108 . In particular, position sensor(s) 108 can be integrated or implemented within image capture device 108 . In some implementations, one or more position sensors 108 can be external to image capture device 108 . As indicated above, position sensor(s) 108 can include one or more accelerometers, gyroscopes, depth cameras, optical cameras, ranging base station. Position sensor(s) can be configured to monitor real-time motion and/or position of image capture device 102 . For instance, image capture device 102 , can be configured to determine a plurality of positions of image capture device 102 based at least in part on a plurality of positioning signals obtained by position sensor(s) 108 . In this manner, a position and orientation of image capture device 102 can be tracked as image capture device 102 transmits and receives energy signals during an image capture sequence. In some implementations, a velocity of image capture device 102 can further be determined based at least in part on the positioning signals obtained by position sensor(s) 108 . In this manner, a position, velocity, orientation, or other physical characteristics of image capture device 102 can be determined during an image capture sequence.

›DETAILED DESCRIPTION · 4 of 6

As an example, if the user facilitates relative motion 112 by moving image capture device 102 with respect to the target 104 during an image capture sequence, position sensor(s) 108 can monitor the motion and/or velocity of image capture device 102 . For instance, position sensor(s) 108 can obtain a plurality of positioning signals indicative of one or more positions, orientations, velocities, etc. of image capture device 102 as the user moves image capture device 102 to facilitate relative motion 112 . The positioning signals, along with the return signals received by antenna element(s) 106 can be used to determine relative motion 112 .

In particular, a timing associated with the return signals can be used to determine a position (e.g. spatial coordinates) of target 104 and/or a range between target 104 and image capture device 102 . Such determined position and/or range can be used in conjunction with the obtained positioning signals to determine relative motion 112 . In some implementations, one or more relative positions and/or one or more relative velocities can be determined.

As indicated above, SAR controller 110 can be configured to receive the (digitally sampled) return signals intercepted by antenna element(s) 106 and to process the signals to generate one or more images depicting target 104 and/or a scene surrounding target 104 . For instance, SAR controller 110 can implement one or more SAR or inverse SAR (ISAR) processing techniques to resolve spatially separated points associated with target 104 and/or the surrounding scene. In example implementations, such processing techniques can include one or more pulse compressions techniques, range-Doppler processing techniques, Doppler mapping techniques, range migration correction techniques, Doppler migration correction techniques and/or other suitable processing techniques.

As indicated above, a plurality of target responses can be determined and/or updated based at least in part on relative motion 112 . In particular, a target response can be indicative of the energy reflected by a scattering point associated with target 104 . The reflected energy can be modulated based at least in part on the relative range and the relative velocity between image capture device 102 and target 104 . As image capture device 102 moves relative to target 104 , the relative range and velocities may vary. In this manner, the modulation of various return signals obtained at different times and/or positions may vary. For instance, a first return signal associated with a scattering point obtained at a first position and/or velocity may have different characteristics than a second return signal associated with the scattering point obtained at a second position and/or velocity.

In this manner, the varying return signals can be resolved and combined to generate an image. For instance, a first target response associated with the reflected energy received at a first position can be updated to reflect the relative motion between image capture device 102 and target 104 . In particular, such updated target response can compensate for the varying modulations in return signals associated with a scattering point that were received at different relative positions. As indicated above, such modulation can cause a response associated with a scattering point to migrate in a non-predetermined manner. For instance, the range of the scattering point with respect to image capture device 102 can vary based at least in part on relative motion 112 . Such range migration can be corrected or compensated for based at least in part on the determined relative positions and/or velocities. The updated target response can reflect such compensated range migration. In this manner, the target response can be updated one or more times to reflect various relative positions and/or velocities between image capture device 102 and target 104 .

As indicated, such described SAR techniques can be used to process the received return signals to generate an image depicting target 104 . Such image can be a radar image having a higher resolution than an image generated using real aperture radar imaging techniques associated with antenna element(s) 106 . In various implementations, the image can be a 2D image, a 3D image, and/or a STTW image. For instance, target 104 may include an occluded object 105 located behind target 104 relative to image capture device 102 . Such STTW image may depict occluded object 105 .

As indicated above, in implementations wherein a 2D image is generated using SAR processing techniques, the image can provide range and azimuth information associated with target 104 . Such 2D image can be captured by facilitating a transverse relative motion 112 between image capture device 102 and target 104 . For instance, the user can facilitate a motion of image capture device 102 along a single plane parallel to a face of target 104 . In implementations, wherein a 3D image is generated using SAR processing techniques, the image can provide range, azimuth and elevation information associated with target 104 . Such 3D image can be captured by facilitating a transverse and longitudinal relative motion 112 between image capture device 102 and target 104 . For instance, a user can facilitate a motion of image capture device 102 along a parallel plane relative to the face of target 104 and along a perpendicular plane relative to the face of target 104 .

In some implementations, image capture device 102 can include a user interface configured to receive an input from the user requesting initiation of an image capture sequence. In some implementations, the user interface can prompt the user to select an image type to be generated (e.g. 2D, 3D, STTW, etc.). The user interface can further prompt the user to move image capture device in an appropriate manner based at least in part on the selected image type. Upon initiation of the image capture sequence, the user can begin moving image capture device 102 in accordance with the selected image type, and image capture device 102 can begin transmitting energy signals. In implementations, wherein antenna element(s) 106 are configured as an electronically steered array, image capture device 102 may identify target 104 and steer the antenna beam generated by antenna element(s) 106 towards target 104 , such that the main lobe of the antenna beam in pointed towards target 104 throughout the image capture sequence. In this manner, a signal-to-noise ratio of the return signals can be improved.

›DETAILED DESCRIPTION · 5 of 6

FIG. 4 depicts a flow diagram of an example method ( 200 ) of capturing one or more images according to example embodiments of the present disclosure. Method ( 200 ) can be implemented by one or more computing devices, such as one or more of the computing devices depicted in FIG. 5 . In particular implementations, the method ( 300 ) can be implemented by the SAR controller 210 depicted in FIG. 1 . In addition, FIG. 4 depicts steps performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the steps of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, or modified in various ways without deviating from the scope of the present disclosure.

At ( 202 ), method ( 200 ) can include transmitting a plurality of pulses during an image capture period associated with a mobile computing device. For instance, the pulses can be periodically transmitted FMCW RF signals. As indicated, the image capture period can be associated with a synthetic aperture radar imaging technique wherein a synthetic aperture is simulated using a relative motion between the mobile computing device and a target.

At ( 204 ), method ( 200 ) can include receiving a plurality of return signals associated with a scattering point associated with the target. For instance, the scattering point can be a point located on the target that reflects received energy in a direction of the mobile computing device. In some implementations, each return signal associated with the scattering point can be received from a different relative position between the target and the mobile computing device.

At ( 206 ), method ( 200 ) can include receiving a plurality of positioning signals associated with the mobile computing device during the image capture period. As indicated, the positioning signals can be obtained by one or more position sensors associated with the mobile computing device. For instance, the position sensors can be embedded within the mobile computing device and/or external to the mobile computing device. In particular, the positioning signals can be indicative of one or more positions, orientations, velocities, and/or other physical characteristics of the mobile computing device as the mobile computing device moves with respect to a target.

At ( 208 ), method ( 200 ) can include determining a relative motion between the mobile computing device and the target. As indicated above, the relative motion can be determined based at least in part on the positioning signals. The relative motion can further be determined based at least in part on the received return signals. For instance, the relative motion can be determined at least in part from a timing between transmission of pulses and reception of the corresponding return signals by the mobile computing device. Such timing can be indicative of a range and/or distance between the target and the mobile computing device. Doppler frequencies associated with the return signals can further be used to determine a velocity of the target. Such range and velocity determined from the return signals can be used in conjunction with the positioning signals obtained by the position sensors to determine the relative motion.

At ( 210 ), method ( 200 ) can include determining a target response associated with the scattering point based at least in part on the relative motion. As indicated, the target response can be indicative of received energy reflected by the scattering point.

At ( 212 ), method ( 200 ) can include updating the target response based at least in part on the determined relative motion between the target and the mobile computing device. As indicated, the updated target response can be determined to compensate for variations in return signals obtained from different positions relative to the scattering point. In some implementations, updating the target response can include determining a second target response to reflect the discrepancies between the relative range and velocity of the mobile computing device and the target associated with different return signals.

At ( 214 ), method ( 200 ) can include generating an image depicting the target based at least in part on the updated target response. The image can be a 2D image, a 3D image and/or a STTW image depicting an occluded object associated with the target.

FIG. 5 depicts an example computing system 300 that can be used to implement the methods and systems according to example aspects of the present disclosure. The system 300 can be implemented using a client-server architecture that includes a mobile computing device 310 that communicates with one or more servers 330 over a network 340 . The system 300 can be implemented using other suitable architectures, such as a single computing device.

The system 300 includes a mobile computing device 310 . The mobile computing device 310 can be implemented using any suitable computing device(s). The mobile computing device 310 can correspond to image capture device 102 of FIG. 1 or other device. In some implementations, the mobile computing device 310 can be a smartphone, tablet, wearable computing device, laptop, or any other suitable computing device capable of being carried by a user while in operation. The mobile computing device 310 can have one or more processors 312 and one or more memory devices 314 . The mobile computing device 310 can also include a network interface used to communicate with one or more servers 330 over the network 340 . The network interface can include any suitable components for interfacing with one more networks, including for example, transmitters, receivers, ports, controllers, antennas, or other suitable components.

The one or more processors 312 can include any suitable processing device, such as a microprocessor, microcontroller, integrated circuit, logic device, or other suitable processing device. The one or more memory devices 314 can include one or more computer-readable media, including, but not limited to, non-transitory computer-readable media, RAM, ROM, hard drives, flash drives, or other memory devices. The one or more memory devices 314 can store information accessible by the one or more processors 312 , including computer-readable instructions 316 that can be executed by the one or more processors 312 . The instructions 316 can be any set of instructions that when executed by the one or more processors 312 , cause the one or more processors 312 to perform operations. For instance, the instructions 316 can be executed by the one or more processors 312 to implement a user interface 320 for capturing images according to example embodiments of the present disclosure and a SAR controller 110 described with reference to FIG. 2 .

›DETAILED DESCRIPTION · 6 of 6

As shown in FIG. 5 , the one or more memory devices 314 can also store data 318 that can be retrieved, manipulated, created, or stored by the one or more processors 312 . The data 318 can include, for instance, return signals generated according to example aspects of the present disclosure, and other data. The data 318 can be stored in one or more databases. The one or more databases can be connected to the mobile computing device 310 by a high bandwidth LAN or WAN, or can also be connected to mobile computing device 310 through network 340 . The one or more databases can be split up so that they are located in multiple locales.

The mobile computing device 310 can include various input/output devices for providing and receiving information from a user, such as a touch screen, touch pad, data entry keys, speakers, and/or a microphone suitable for voice recognition. For instance, the mobile computing device 310 can have a display device 335 for presenting a user interface for displaying images according to example aspects of the present disclosure. Mobile computing device 310 can further include position sensors 108 described with respect to FIG. 1 .

Mobile computing device 310 can further include a 322 . Radar module can include one or more antenna elements 106 as described with reference to FIG. 1 . Radar module 322 can further include a signal generator configured to generate energy signals to be transmitted, and to provide the generated energy signals to antenna element(s) 106 .

In some implementations, the mobile computing device 310 can exchange data with one or more remote computing devices, such as server 330 over the network 340 . For instance, server 330 can be a web server. Server 330 can be implemented using any suitable type of computing device. Similar to the mobile computing device 310 , a server 330 can include one or more processor(s) 332 and a memory 334 . The one or more processor(s) 332 can include one or more central processing units (CPUs), graphics processing units (GPUs) dedicated to efficiently rendering images or performing other specialized calculations, and/or other processing devices. The memory 334 can include one or more computer-readable media and can store information accessible by the one or more processors 332 , including instructions 336 that can be executed by the one or more processors 332 and data 338 .

In some implementations, one or more example aspects of the present disclosure can be performed by server 330 . For instance, one or more operations associated with SAR controller 110 can be performed by server 330 and communicated to the mobile computing device 310 via the network 340 .

The server 330 can also include a network interface used to communicate with one or more remote computing devices (e.g. mobile computing device 310 ) over the network 340 . The network interface can include any suitable components for interfacing with one more networks, including for example, transmitters, receivers, ports, controllers, antennas, or other suitable components.

The network 340 can be any type of communications network, such as a local area network (e.g. intranet), wide area network (e.g. Internet), cellular network, or some combination thereof. The network 340 can also include a direct connection between a server 330 and the mobile computing device 310 . In general, communication between the mobile computing device 310 and a server 330 can be carried via network interface using any type of wired and/or wireless connection, using a variety of communication protocols (e.g. TCP/IP, HTTP, SMTP, FTP), encodings or formats (e.g. HTML, XML), and/or protection schemes (e.g. VPN, secure HTTP, SSL).

The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. One of ordinary skill in the art will recognize that the inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, server processes discussed herein may be implemented using a single server or multiple servers working in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

While the present subject matter has been described in detail with respect to specific example embodiments thereof, it will be appreciated that those skilled in the art, upon attaining an understanding of the foregoing may readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the scope of the present disclosure is by way of example rather than by way of limitation, and the subject disclosure does not preclude inclusion of such modifications, variations and/or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art.

Claims

20 · 2 independent · depth 4
1234567891011121314151617181920
20 granted claims

Classifications

3 codes
IPC · International Patent Classification
Section G — Physics
  • G01S13/931
  • G01S13/88
  • G01S13/90

Claim changes

Soon
Coming soonHow the claims changed between publication and grant

See which claims were amended, added or cancelled during examination, with every added and removed word marked.

AmendedAddedCancelledUnchanged

The published claims of this patent are not paired with the granted ones in what we hold.

File wrapper

⤢ drag to zoomJul 2016Jan 2017Jul 2017Jan 2018Jul 2018Jan 2019Jul 2019Jan 2020Jul 2020Jan 2021USPTOApplicantResponse after non-finalApplicant-initiated interviewNotice of allowance
USPTOApplicanthover for detail · click to open
Pendency
4.5 y
1,649 days filing → grant
Office actions
1
non-final + final
Responses
1
1 RCE
Interviews
1
examiner interview summaries
Examiner
Erin F Heard
art unit 3648 · TC 3600
Citations: 998 back · 31 forward

See the full prosecution history — every USPTO and applicant action on this file, in order.

Log in to unlock

Chain of title

⤢ drag to zoom20162018202020222024202620282030203220342036Owner 2
Titlehover for detail · click to open

See the full assignment history — every owner this patent has passed through, with recordation dates and reel/frame numbers.

Log in to unlock

Term & fees

See the term timeline — pendency span, in-force span, the maintenance fees paid and both computed expiry dates.

Log in to unlock

Priority chain

1 priority documents
Priority
6 Oct 2015
earliest claimed
›Priority documents — 1
TypeDocumentDate
provisionalUS 622379756 Oct 2015

Worldwide family

63 members · 6 offices
US43EP3JP6KR6CN4WO1
this patentIP5 & PCTother officessolid = grantedhover for detail · click to open
Members
63
DOCDB simple family 58446760
Offices
6
US · EP · JP · KR · CN · WO
Granted
35 of 63
grant date present
Non-English titles
16
shown as filed, never translated
›IP5 & PCT — 63 members
OfficePublicationKindPublishedFiledStatusTitle
USUS-2017097413-A1A16 Apr 20175 Oct 2016publishedRadar-Enabled Sensor Fusion
USUS-2019011989-A1A110 Jan 20195 Oct 2016publishedGesture Component with Gesture Library
USUS-10222469-B1B15 Mar 20196 Oct 2016grantedRadar-based contextual sensing
USUS-10300370-B1B128 May 20195 Oct 2016grantedAdvanced gaming and virtual reality control using radar
USUS-10310621-B1B14 Jun 20196 Oct 2016grantedRadar gesture sensing using existing data protocols
USUS-2019232156-A1A11 Aug 201910 Apr 2019publishedAdvanced Gaming and Virtual Reality Control Using Radar
USUS-10379621-B2B213 Aug 20195 Oct 2016grantedGesture component with gesture library
USUS-2019257939-A1A122 Aug 20192 May 2019publishedGesture Component with Gesture Library
USUS-10401490-B2B23 Sep 20195 Oct 2016grantedRadar-enabled sensor fusion
USUS-2019321719-A1A124 Oct 20193 Jul 2019publishedRadar-Enabled Sensor Fusion
USUS-10459080-B1B129 Oct 20196 Oct 2016grantedRadar-based object detection for vehicles
USUS-10503883-B1B110 Dec 20196 Oct 2016grantedRadar-based authentication
USUS-10540001-B1B121 Jan 20206 Oct 2016grantedFine-motion virtual-reality or augmented-reality control using radar
USUS-2020089314-A1A119 Mar 202020 Nov 2019publishedFine-Motion Virtual-Reality or Augmented-Reality Control Using Radar
USUS-10705185-B1B17 Jul 20206 Oct 2016grantedApplication-based signal processing parameters in radar-based detection
USUS-2020278422-A1A13 Sep 202015 May 2020publishedApplication-Based Signal Processing Parameters in Radar-Based Detection
USUS-10768712-B2B28 Sep 20202 May 2019grantedGesture component with gesture library
USUS-10817065-B1B127 Oct 20207 Apr 2016grantedGesture recognition using multiple antenna
USthis patentUS-10823841-B1B13 Nov 202029 Apr 2016grantedRadar imaging on a mobile computing device
USUS-2020393912-A1A117 Dec 202027 Aug 2020publishedGesture Recognition Using Multiple Antenna
USUS-10908696-B2B22 Feb 202110 Apr 2019grantedAdvanced gaming and virtual reality control using radar
USUS-2021096653-A1A11 Apr 202111 Dec 2020publishedAdvanced Gaming and Virtual Reality Control Using Radar
USUS-11080556-B1B13 Aug 20216 Oct 2016grantedUser-customizable machine-learning in radar-based gesture detection
USUS-11132065-B2B228 Sep 20213 Jul 2019grantedRadar-enabled sensor fusion
USUS-2021326642-A1A121 Oct 202129 Jun 2021publishedUser-Customizable Machine-Learning in Radar-Based Gesture Detection
USUS-11175743-B2B216 Nov 202127 Aug 2020grantedGesture recognition using multiple antenna
USUS-2021365124-A1A125 Nov 20214 Aug 2021publishedRadar-Enabled Sensor Fusion
USUS-2022019291-A1A120 Jan 202228 Sep 2021publishedGesture Recognition Using Multiple Antenna
USUS-2022043519-A1A110 Feb 202220 Oct 2021publishedFine-Motion Virtual-Reality or Augmented-Reality Control Using Radar
USUS-11256335-B2B222 Feb 202220 Nov 2019grantedFine-motion virtual-reality or augmented-reality control using radar
USUS-2022066567-A1A13 Mar 202213 Oct 2021publishedGesture Recognition Using Multiple Antenna
USUS-2022066568-A1A13 Mar 202210 Nov 2021publishedGesture Recognition Using Multiple Antenna
USUS-11385721-B2B212 Jul 202215 May 2020grantedApplication-based signal processing parameters in radar-based detection
USUS-11481040-B2B225 Oct 202229 Jun 2021grantedUser-customizable machine-learning in radar-based gesture detection
USUS-11592909-B2B228 Feb 202320 Oct 2021grantedFine-motion virtual-reality or augmented-reality control using radar
USUS-11656336-B2B223 May 202311 Dec 2020grantedAdvanced gaming and virtual reality control using radar
USUS-11693092-B2B24 Jul 202310 Nov 2021grantedGesture recognition using multiple antenna
USUS-11698438-B2B211 Jul 202328 Sep 2021grantedGesture recognition using multiple antenna
USUS-11698439-B2B211 Jul 202313 Oct 2021grantedGesture recognition using multiple antenna
USUS-2023273298-A1A131 Aug 20234 May 2023publishedAdvanced Gaming and Virtual Reality Control Using Radar
USUS-12085670-B2B210 Sep 20244 May 2023grantedAdvanced gaming and virtual reality control using radar
USUS-12117560-B2B215 Oct 20244 Aug 2021grantedRadar-enabled sensor fusion
USUS-2024369685-A1A17 Nov 202417 Jul 2024publishedAdvanced Gaming and Virtual Reality Control Using Radar
EPEP-3359976-A1A115 Aug 20186 Oct 2016publishedFusion de capteurs à radar activéfr
EPEP-3359976-B1B111 Feb 20266 Oct 2016grantedFusion eines radaraktivierten sensorsde
EPEP-4697288-A2A218 Feb 20266 Oct 2016publishedRadaraktivierte sensorfusionde
JPJP-2018527558-AA20 Sep 20186 Oct 2016publishedレーダ対応センサフュージョンja
JPJP-6694053-B2B213 May 20206 Oct 2016grantedレーダ対応センサフュージョンja
JPJP-2020101560-AA2 Jul 202020 Feb 2020publishedレーダ対応センサフュージョンja
JPJP-6889293-B2B218 Jun 202120 Feb 2020grantedレーダ対応センサフュージョンja
JPJP-2021131396-AA9 Sep 202120 May 2021publishedレーダ対応センサフュージョンja
JPJP-7257447-B2B213 Apr 202320 May 2021grantedレーダ対応センサフュージョンja
KRKR-20180030123-AA21 Mar 20186 Oct 2016published레이더-사용가능 센서 퓨전ko
KRKR-102245190-B1B126 Apr 20216 Oct 2016granted레이더-사용가능 센서 퓨전ko
KRKR-20210047964-AA30 Apr 20216 Oct 2016published레이더-사용가능 센서 퓨전ko
KRKR-102352236-B1B114 Jan 20226 Oct 2016granted레이더-사용가능 센서 퓨전ko
KRKR-20220011218-AA27 Jan 20226 Oct 2016published레이더-사용가능 센서 퓨전ko
KRKR-102377488-B1B121 Mar 20226 Oct 2016grantedRadar-enabled sensor fusion
CNCN-107710012-AA16 Feb 20186 Oct 2016published支持雷达的传感器融合zh
CNCN-107710012-BB27 Aug 20216 Oct 2016grantedMethod, apparatus and system on chip for radar-enabled sensor fusion
CNCN-113655471-AA16 Nov 20216 Oct 2016publishedMethod, apparatus and system on chip for radar-enabled sensor fusion
CNCN-113655471-BB29 Mar 20246 Oct 2016grantedMethod, apparatus and system-on-chip for supporting sensor fusion of radar
WOWO-2017062566-A1A113 Apr 20176 Oct 2016publishedFusion de capteurs à radar activéfr

Validity challenges

See the validity challenges on record — reexaminations, IPRs and PGRs, with their institution decisions and outcomes.

Log in to unlock

Citations

See every patent this one cites and every patent that cites it back — publication, assignee, and how each one was found.

Log in to unlock