Traffic signal detection device and traffic signal detection method

The present invention relates to a traffic signal detection device and a traffic signal detection method for detecting a traffic signal.

A device is proposed which can continuously monitor an object ahead of a vehicle by selecting a detection result based on any of a wide-angle image captured by a wide-angle camera having a wide angle of view and a narrow-angle image captured by a narrow-angle camera having a narrow angle of view (see Japanese Patent Laid-Open Publication No. 2011-121398). Such a device selects a narrow-angle image when a distance to the object exceeds a distance in which an object can be detected with a desired accuracy by a wide-angle image, and a wide-angle image when an object is out of the field of view of the narrow-angle camera.

However, for the technology described in Japanese Patent Laid-Open Publication No. 2011-121398, there is no mention of a possibility of delay of determination of vehicle control due to recognition delay time or non-recognition, etc., occurred in switching detection results. The delay of determination to stop a vehicle may require a larger deceleration.

The present invention has been made in view of the above problem, and an object thereof is to provide a traffic signal detection device and a traffic signal detection method which can prevent effects on vehicle control due to delay time or non-recognition in switching of the detection results.

The traffic signal detection device considers a region in which a vehicle should start deceleration in order to stop the vehicle, and thereby switches detection results of a traffic signal by a narrow-angle camera and a wide-angle camera so as to reduce effects on the vehicle control in switching of the detection results.

FIG. 1 is a block diagram schematically illustrating a basic configuration of a traffic signal detection device according to an embodiment of the present invention:

FIG. 2 is a block diagram illustrating a configuration of a traffic signal detection device according to an embodiment of the present invention and a dataflow therein;

FIG. 3 is an explanatory diagram of a region where a narrow-angle camera and a wide-angle camera can detect a traffic signal, a deceleration start region, and a switchable region;

FIG. 4A to FIG. 4C are explanatory diagrams of angle-of-view region of the narrow-angle camera and the wide-angle camera;

FIG. 5 is an explanatory diagram of a region where the narrow-angle camera and the wide-angle camera can detect a traffic signal, a deceleration start region, and a switchable region (W 1 side);

FIG. 6 is an explanatory diagram of a region where the narrow-angle camera and the wide-angle camera can detect a traffic signal, a deceleration start region, and a switchable region (N 2 side); and

FIG. 7 is a flowchart illustrating an exemplary traffic signal detection method by the traffic signal detection device according to an embodiment of the present invention.

With reference to the drawings, embodiments of the present invention will be described. In the description of the drawings, the same or similar parts are denoted by the same or similar reference signs, and overlapping descriptions are omitted.

(Traffic Signal Detection Device)

As shown in FIG. 1 , the traffic signal detection device according to the present embodiment includes an imaging part 1 with a narrow-angle camera 11 and a wide-angle camera 12 , a traffic signal detector 4 having a camera selector 2 and an image processor 3 , and a vehicle speed sensor 10 . The vehicle speed sensor 10 detects speed of a vehicle 5 , and outputs the detected speed to the camera selector 2 as speed information Dv. The traffic signal detector 4 detects, from an image captured by the imaging part 1 , a traffic signal which is installed around a road on which the vehicle 5 is travelling and exists ahead of the vehicle 5 , and outputs the detected traffic signal, to the outside thereof as traffic signal information Dt which is a detection result. The vehicle 5 is, for example, an automatic driving vehicle which automatically travels along a preset travelling passage on the basis of the traffic signal information Dt and the like of the traffic signal detector 4 .

The narrow-angle camera 11 and the wide-angle camera 12 are respectively mounted in the vehicle 5 to capture images ahead of the vehicle 5 . Each of the narrow-angle camera 11 and the wide-angle camera 12 includes a solid-state imaging element, such as a CCD or CMOS, and outputs digital images for which image processing can be executed. The wide-angle camera 12 has an angle of view (viewing angle) wider than the narrow-angle camera 11 . Thus, the wide-angle camera 12 has lower resolution capability than the narrow-angle camera 11 when imaging an object at the same distance with the same resolution. The narrow-angle camera 11 outputs a captured narrow-angle image as narrow-angle image information Dn to the traffic signal detector 4 . The wide-angle camera 12 outputs a captured wide-angle image as wide-angle image information Dw to the traffic signal detector 4 .

The camera selector 2 and the image processor 3 can be implemented using, for example, a microcontroller including a central processing unit (CPU), a memory, and an input/output unit. In this case, the CPU executes computer programs preinstalled in the microcontroller to configure the camera selector 2 and the image processor 3 . The microcontroller may be used also as an electronic control unit (ECU) used for other control on the vehicle 5 , such as automatic driving control, for example.

The image processor 3 detects a traffic signal by image processing from at least any of narrow-angle images captured by the narrow-angle camera 11 and wide-angle images captured by the wide-angle camera 12 . The image processor 3 outputs, on the basis of determination of the camera selector 2 , any of a traffic signal detected from the narrow-angle image and a traffic signal detected from the wide-angle image, as the traffic signal information Dt. The image processor 3 can reduce the processing load by detecting a traffic signal from any of the narrow-angle image and the wide-angle image. In this case, the image processor 3 can cause a delay time for detection when switching the narrow-angle image and the wide-angle image used for detecting a traffic signal.

The image processor 3 performs image processing for traffic signal detection on each of the narrow-angle image captured by the narrow-angle camera 11 and the wide-angle image captured by the wide-angle camera 12 . For example, the image processor 3 detects, using synchronized image generation, a signal light in the traffic signal that flashes at a specific cycle based on the alternating cycle of commercial power supplies. Alternatively, the image processor 3 can detect a traffic signal using hue and shape similarity determination processing. As described above, the image processor 3 can detect a traffic signal and the color of the traffic signal using various image processing methods or a combination thereof.

Note that the image processor 3 sets not the entire region of each image indicated by the narrow-angle image information Dn and the wide-angle image information Dw, but a part of a detection region on the image and performs image processing only on the detection region, and thereby can reduce processing load and processing time for traffic signal detection. The detection region is set on the image on the basis of the positions of traffic signals estimated from map information including road shape and positions of traffic signals, positional information of the vehicle 5 and the like.

As shown in FIG. 2 , the camera selector 2 includes a detectable region calculator 21 , a deceleration start region calculator 22 , a switchable region calculator 23 , a delay distance calculator 24 , a switch region determiner 25 , and a camera determiner 26 . On the basis of the speed information Dv and the like, the camera selector 2 outputs to the image processor 3 selection camera information Ds for designating to the image processor 3 which any of a traffic signal detected from the narrow-angle image and a traffic signal detected from the wide-angle image is output as the traffic signal information Dt.

The detectable region calculator 21 calculates a detectable region where traffic signals could be detected on the basis of camera information Dc indicating the angles of view, vertical and horizontal angles, and resolution set to the narrow-angle camera 11 and the wide-angle camera 12 , and outputs the detectable region as detectable region information D 01 . The detectable region calculator 21 calculates a region where the detection rate of traffic signals exceeds a predetermined threshold and a region where the traffic signal does not deviate from the angle of view, for each of the narrow-angle camera 11 and the wide-angle camera 12 . The detectable region calculator 21 calculates, as a detectable region, a region where both the narrow-angle camera 11 and the wide-angle camera 12 could detect a traffic signal, from the region calculated for each of the narrow-angle camera 11 and the wide-angle camera 12 .

Note that, the region where the detection rate of traffic signals exceeds the predetermined threshold is calculated by predicting the number of pixels on the images acquired by the narrow-angle camera 11 and the wide-angle camera 12 on the basis of a distance between a detection object and the vehicle 5 and then using whether or not the predicted number of pixels exceeds that of pixels to be detected, which is a threshold, in historical detection data. The region where the traffic signal does not deviate from the angle of view is calculated on the basis of deviation of the traffic signal from each optical axis of the narrow-angle camera 11 and the wide-angle camera 12 along the travelling direction of the vehicle 5 .

As shown in FIG. 3 , assuming that the distance from the vehicle 5 to the traffic signal is X, the region where the narrow-angle camera 11 can detect a traffic signal is marked by the right-downward diagonal hatching, and in a region where traffic signals T 1 and T 2 are located, is a region of N 21 <X<N 1 at a distance in the longitudinal direction of the vehicle. Moreover, a region where the wide-angle camera 12 can detect a traffic signal is marked by the left-downward diagonal hatching, and in the region where traffic signals T 2 and T 3 are located, is a region of W 2 <X<W 1 at a distance in the longitudinal direction of the vehicle. Note that the vehicle width direction of the vehicle indicates a region to not deviate from the angle of view in the horizontal direction of the narrow-angle camera 11 and the wide-angle camera 12 and becomes narrower as a distance in the longitudinal direction of the vehicle is shorter.

N 1 and W 1 are distances in which rates of the detection of traffic signals by the narrow-angle camera 11 and the wide-angle camera 12 exceed a predetermined threshold, and are predetermined values set on the basis of a distance in which a traffic signal can be detected from the respective images, respectively. The distance region where a rate of detection of traffic signals by the narrow-angle camera 11 exceeds a predetermined threshold is X<N 1 , and a distance region in which a rate of detection of traffic signals by the wide-angle camera 12 exceeds the predetermined threshold is X<W 1 . N 2 and W 2 are predetermined values set on the basis of a position of the vehicle in the longitudinal direction of positions in which a traffic signal deviates from the angle of view of the narrow-angle camera 11 and the wide-angle camera 12 . A region where a traffic signal deviates from the angle of view of the narrow-angle camera 11 is X<N 2 in the longitudinal direction of the vehicle, while a region where a traffic signal deviates from the angle of view of the wide-angle camera 12 is X<W 2 in the longitudinal direction of the vehicle. Since N 2 <W 1 , a region where both the narrow-angle camera 11 and the wide-angle camera 12 can detect a traffic signal is N 2 <X<W 1 in the longitudinal direction of the vehicle in a region where the traffic signal T 2 is located. The detectable region calculator 21 outputs N 2 <X<W 1 as the detectable region information D 01 .

FIGS. 4( a ) to 4( c ) are diagrams showing images, with a traffic signal captured in the forward direction from the vehicle 5 , acquired from the narrow-angle camera 11 and the wide-angle camera 12 . FIGS. 4( a ) to 4( c ) show states where the traffic signals T 1 to T 3 of FIG. 3 are located, respectively. The range of the angle of view of the narrow-angle camera 11 is situated inside the range of the angle of view of the wide-angle camera 12 . FIG. 4( a ) shows the traffic signal T 1 located in a region where the traffic signal T 1 can be detected only by the narrow-angle camera 11 . A distance region where a rate of detection of the traffic signal by the wide-angle camera 12 is equal to or smaller than a predetermined threshold is X≥W 1 . FIG. 4( b ) shows the traffic signal T 2 located in a region where both the narrow-angle camera 11 and the wide-angle camera 12 can detect the traffic signal. FIG. 4( c ) shows the traffic signal T 2 located in a region where the traffic signal T 2 can be detected only by the wide-angle camera 12 . A region where the traffic signal deviates from the angle of view of the narrow-angle camera 11 is X<N 2 .

On the basis of target deceleration Dr which is a target deceleration at the time of decelerating the vehicle 5 , and the speed information Dv, the deceleration start region calculator 22 calculates a deceleration start region in which the vehicle 5 should start deceleration in order to stop the vehicle 5 at a stop position for each traffic signal, and outputs the deceleration start region as deceleration start region information D 02 . The target deceleration Dr may be a range of the target deceleration.

Assuming that a range of the deceleration indicated by the target deceleration Dr is a0 to a1 (a0<a1), speed of the vehicle 5 indicated by the speed information Dv is v, and a distance from the traffic signal to a stop line of the traffic signal is α, a distance in the longitudinal direction from the vehicle in the deceleration start region can be expressed by Formula (1).

v 2/(2× a 0)−α˜ v 2/(2× a 1)−α  (1)

In the example shown in FIG. 3 , C 1 corresponds to v2/(2×a1)−α, and C 2 corresponds to v2/(2×a0)−α. In other words, the deceleration start region information D 02 means that, if deceleration is started while the distance X to the traffic signal is C 2 ≤X≤C 1 , the vehicle 5 can stop at the stop position with the deceleration a0 to a1. The deceleration may be adjusted in magnitude and range (a0 to a1) in consideration of the ride comfort of a driver and the like.

On the basis of the detectable region information D 01 and the deceleration start region information D 02 , the switchable region calculator 23 calculates a switchable region in which the image processor 3 can switch from the narrow-angle image to the wide-angle image so as not to overlap the timing at which the vehicle control starts deceleration, and outputs the switchable region as switchable region information D 03 . In the example shown in FIG. 3 , a region indicated by the switchable region information D 03 is a region obtained by excluding the deceleration start region information D 02 from the detectable region information D 01 .

Assuming that the image processor 3 switches from the narrow-angle image to the wide-angle image when X is in the deceleration start region, the vehicle 5 may delay decision for stopping itself and require deceleration higher than the target deceleration Dr. The vehicle 5 determines whether or not to decelerate in order to stop at a position where X is in the switchable region and starts deceleration control, and thus can stop at a stop position for each traffic signal with deceleration in the target deceleration Dr.

On the basis of a predetermined delay time t when the image processor 3 switches a narrow-angle image used for detection of a traffic signal to a wide-angle image and speed v of the vehicle 5 indicated by the speed information Dv, the delay distance calculator 24 calculates a delay distance for the vehicle 5 to move for the delay time t, and outputs the delay distance as delay distance information D 04 . The t may be set in advance. In the example shown in FIG. 3 , assuming that the vehicle 5 moves at speed v until X becomes D 2 from D 1 for the delay time t, the delay distance |D 1 -D 2 | is vt.

The switch region determiner 25 determines, on the basis of the switchable region information D 03 and the delay distance information D 04 , a switch region indicating a region in which the image processor 3 switches from the narrow-angle image to the wide-angle image, and outputs the switch region as switch region information D 05 .

The switch region determiner 25 determines a region in which the delay distance falls into the switchable region as a switch region. Moreover, the switch region is determined so that a traffic signal detected from the narrow-angle image with high resolution capability can be a detection result as long as possible, even if a delay distance is generated when the traffic signal is located in the switchable region. In other words, the switch region determiner 25 determines a region in which the delay distance |D 1 -D 2 | is close to N 2 and falls into the switchable region (N 2 <X<W 1 excluding C 2 ≤X≤C 1 ) as a switch region.

In the example shown in FIG. 5 , since the delay distance does not fall into the N 2 side from the deceleration start region, the region (E 2 ≤X≤E 1 ) indicated by the switch region information D 05 is determined to be a region as close as possible to C 1 of the region on the W 1 side. Note that E 1 and E 2 correspond to D 1 and D 2 . In other words, when the traffic signal passes through E 1 , the image processor 3 switches from the narrow-angle image to the wide-angle image to detect the traffic signal. It is possible to switch from the narrow-angle image to the wide-angle image before the traffic signal enters the deceleration start region, and to switch the detection result of the traffic signal by preventing effects on the vehicle control. Moreover, in the example shown in FIG. 6 , since the delay distance fits between C 2 and N 2 , the region (E 2 ≤X≤E 1 ) indicated by the switch region information D 05 is determined to be a region as close as possible to N 2 . The image processor 3 can detect a traffic signal with a narrow-angle image of the narrow-angle camera 11 until the traffic signal passes through the deceleration start region. Longer use of the narrow-angle image with high resolution while switching the detection result of a traffic signal with preventing effects on the vehicle control allows higher precision detection of the traffic signal over a longer time.

On the basis of the switchable region information D 03 , traffic signal's relative position information Dp, selection camera historical information Dh and the speed information Dv, the camera determiner 26 outputs the selection camera information Ds indicating which of the narrow-angle camera 11 and the wide-angle camera 12 is used for detecting the traffic signal. The selection camera historical information Dh indicates a camera that has captured an image used for detecting a previous traffic signal and the detection result of the previous traffic signal. The selection camera information Ds designates, to the image processor 3 , which of a traffic signal detected from the narrow-angle image and a traffic signal detected from the wide-angle image is to be output as the traffic signal information Dt.

The relative position information Dp indicates a relative position of each traffic signal with respect to the vehicle 5 . The relative position information Dp is estimated from map information including road shapes and positions of traffic signals, and positional information of the vehicle 5 . The positional information of the vehicle 5 is acquired from a positioning device such as a Global Positioning System (GPS) receiver, an acceleration sensor mounted in the vehicle 5 , the steer-angle sensor and the vehicle speed sensor 10 . Note that, a traffic signal indicated by the relative position information Dp is a traffic signal which presents a signal to the vehicle 5 , in other words, a traffic signal which the vehicle should follow.

(Operation of Camera Determiner)

Hereinbelow, with reference to the flowchart of FIG. 7 , an exemplary traffic signal detection method by the traffic signal detection device according to the present embodiment will be described. The following process is performed, by the camera determiner 26 , for each traffic signal that presents a signal to the vehicle 5 .

In Step S 1 , the camera determiner 26 determines whether or not the position of a traffic signal is in a region where the narrow-angle camera 11 can detect the traffic signal. In other words, it is determined whether or not the distance X from the vehicle 5 to the traffic signal is equal to or greater than N 2 calculated by the detectable region calculator 21 , and the traffic signal is in a region where the traffic signal does not deviate from the angle of view in the horizontal direction of the narrow-angle camera 11 . When the position of the traffic signal is in a region where the narrow-angle camera 11 can detect the traffic signal, the camera determiner 26 sets the process proceeding to Step S 2 , and selects the wide-angle camera 12 in Step S 7 when the position of the traffic signal is not in a region where the narrow-angle camera 11 can detect the traffic signal, since the narrow-angle image cannot detect the traffic signal.

In Step S 2 , the camera determiner 26 determines whether or not the position of the traffic signal is in a region where the wide-angle camera 12 can detect the traffic signal. Note that, since the angle of view in the horizontal direction of the wide-angle camera 12 is wider than the angle of view in the horizontal direction of the narrow-angle camera 11 , in Step S 2 , the traffic signal is in a region where the traffic signal does not deviate from the angle of view of the horizontal direction of the wide-angle camera 12 . In other words, it is determined whether or not the distance X from the vehicle 5 to the traffic signal is equal to or smaller than W 1 calculated by the detectable region calculator 21 . When the vehicle 5 is running straight toward the traffic signal, the camera determiner 26 will determine whether or not a traffic signal detected from the narrow-angle image of the narrow-angle camera 11 by the image processor 3 enters a region where the wide-angle camera 12 can detect the traffic signal. The camera determiner 26 sets the process proceeding to Step S 3 , when the position of the traffic signal is in a region where the wide-angle camera 12 can detect the traffic signal, and selects the narrow-angle camera 11 in Step S 8 when the position of the traffic signal is not in a region where the wide-angle camera 12 can detect the traffic signal, since the traffic signal cannot be detected from the wide-angle image.

In Step S 3 , the camera determiner 26 determines whether or not the traffic signal detected by the image processor 3 is a green light. The camera determiner 26 sets the process proceeding to Step S 4 when the traffic signal is not a green light. The camera determiner 26 sets the process proceeding to Step S 5 , when the traffic signal is a green light.

In Step S 4 , the camera determiner 26 determines whether or not the position of the traffic signal passes through the deceleration start region. In other words, it is determined whether or not the distance X from the vehicle 5 to the traffic signal enters the region of X≤C 1 calculated by the deceleration start region calculator 22 . When the position of the traffic signal does not pass through the deceleration start region, the process goes to Step S 5 . When the traffic signal is not a green light, or the traffic signal was not detected, and when the position of the traffic signal passes through the deceleration start region, the camera determiner 26 selects the wide-angle camera 12 in Step S 9 since the vehicle 5 has already started deceleration, and even if a delay occurs in switching images to detect the traffic signal, when the traffic signal changed to the green light, or the green light is detected, the timing of stopping the deceleration control of the vehicle is only delayed and there is little influence. As a result, the camera determiner 26 can prevent a delay due to switching of detection results while the traffic signal is a green light.

In Step S 5 , the camera determiner 26 determines whether or not the detection result of the previous traffic signal from the selection camera historical information Dh is from the narrow-angle camera 11 . The camera determiner 26 sets the process proceeding to Step S 7 , when the narrow-angle camera 11 is selected in the previous process. The camera determiner 26 sets the process proceeding to Step S 6 , when the wide-angle camera 12 is selected in the previous process.

In Step S 6 , the camera determiner 26 determines, for the previous process, whether or not the vehicle 5 passes through the intersection (traffic signal), on the basis of the position of the intersection (traffic signal) estimated from positional information of the vehicle 5 , etc. When the camera determiner 26 determines that the vehicle 5 passes through the intersection (traffic signal), the process goes to Step S 7 . When the camera determiner 26 determines that the vehicle 5 does not pass through the intersection (traffic signal), the process goes to Step S 9 , in order to minimize the switching of detection results, the wide-angle camera 12 is continuously selected until it is determined that the vehicle 5 passes through the intersection (traffic signal).

In Step S 7 , the camera determiner 26 determines whether or not the position of the detected traffic signal is in a switch region determined by the switch region determiner 25 on the basis of the switch region information D 05 . In other words, it is determined whether or not the distance X from the vehicle 5 to the traffic signal is E 2 ≤X≤E 1 . The camera determiner 26 selects the wide-angle camera 12 in Step S 9 when the position of the traffic signal is in the switch region. The camera determiner 26 continuously selects the narrow-angle camera 11 in Step S 8 when the position of the traffic signal is not in the switch region.

The camera determiner 26 generates the selection camera information Ds to designate, to the image processor 3 , to output the traffic signal detected from the image of the camera selected in Step S 8 or Step S 9 as the traffic signal information Dt, and output it to the image processor 3 .

With the traffic signal detection device according to the present embodiment, while the traffic signal is located in the switch region determined to avoid the deceleration start region, the detection result is switched from the traffic signal detected from the narrow-angle image to the traffic signal detected from the wide-angle image. As a result, the traffic signal detection device according to the present embodiment can suppress the effects on the vehicle control, switch the detection result of the traffic signal, and perform smooth deceleration control of the vehicle 5 .

Moreover, with the traffic signal detection device according to the present embodiment, the switch region is set on the basis of a distance in which the traffic signal deviates from the angle of view of the narrow-angle camera 11 , so that the detection result by the narrow-angle camera 11 with high resolution capability can be selected for a long time. Accordingly, the traffic signal detection device according to the present embodiment can improve the detection accuracy of traffic signals.

Moreover, with the traffic signal detection device according to an embodiment of the present invention, a detection result is switched to that with the wide-angle camera 12 on the basis of a distance in which a traffic signal deviates from the angle of view of the narrow-angle camera 11 when the detected traffic signal is a green light. As a result, the traffic signal detection device according to the present embodiment can select the detection result by the narrow-angle camera 11 with high resolution capability for a long time and improve the detection accuracy of a traffic signal.

Moreover, with the traffic signal detection device according to the present embodiment, the switch region is set on the basis of a distance in which a traffic signal can be detected from a wide-angle image, so that the detection result with the narrow-angle camera 11 can be continuously selected in a position in which the detection accuracy with the wide-angle camera 12 is low. Accordingly, the traffic signal detection device according to the present embodiment can select the detection result by the narrow-angle camera 11 with high resolution capability for a long time and improve the detection accuracy of a traffic signal.

Moreover, with the traffic signal detection device according to the present embodiment, when the detected traffic signal is not a green light, or the traffic signal cannot be detected, and when the detected traffic signal passes through the deceleration start region, since the vehicle 5 has already started deceleration, a traffic signal detected from the wide-angle image is selected as a detection result. As a result, the traffic signal detection device according to the present embodiment can prevent delay due to the switching of the detection result while the traffic signal is a green light.

Moreover, with the traffic signal detection device according to the present embodiment, when a detection result by the wide-angle camera 12 is selected, the wide-angle camera 12 is continuously selected until the vehicle 5 passes through the intersection. As a result, the traffic signal detection device according to the present embodiment can reduce unnecessary switching of detection results and minimize switching of detection results.

Although the present invention has been described by the embodiments as described above, one should not understand that the statements and the drawings as part of the disclosure limit the present invention. From this disclosure, various modified embodiments, examples and operation techniques will be apparent to those who skilled in the art.

For example, in the embodiments described above, the processes of Step S 3 and Step S 4 in the flowchart of FIG. 6 may be performed in reverse order. Alternatively, the present invention, of course, includes various embodiments not described in this description, such as each configuration mutually applied. Therefore, the technical scope of the present invention is defined only by the appropriate features according to the scope of claims in view of the explanations made above.

According to the present invention, a traffic signal detection device and a traffic signal detection method can be provided which can reduce effects on the vehicle control due to switching by performing switching of detection results in consideration of a deceleration start region in which the vehicle should start deceleration.

4 traffic signal detector

5 vehicle

10 vehicle speed sensor

11 narrow-angle camera

12 wide-angle camera