Calculation system and calculation method

The present invention relates to a calculation system and a calculation method.

Incorporation of inference functions into an edge is progressing for autonomous operations of automobiles and industrial equipment. Since an inference error of edge equipment causes a serious accident, error detection is essential. A configuration is known in which an inference units such as neural network are made multiple redundant, inference results of the inference units for the same input are compared, and an inference unit that has an inference result different from the other inference units is specified, so as to perform detection of inference errors due to faults or failures. However, in a case of multiple redundancy, all inference units output the same inference result even if the inference is incorrect due to poor learning. Therefore, inference errors due to the poor learning cannot be detected due to the multiple redundancy of the inference units. PTL 1 disclose a signal processing device which includes a labeling unit for labeling according to a type of an imaged subject, a distance measuring unit for acquiring distance measuring information at distance measuring points disposed discretely, and an interpolation generation unit for interpolating and generating distance information between adjacent distance measuring points on the subject labeled on the same type based on a labeling result by the labeling unit.

Patent Literature

PTL 1: JP-A-2019-028861

Technical Problem

The invention described in PTL 1 cannot detect a discrimination error in a type of an object.

Solution to Problem

A calculation system according to a first aspect of the invention is a calculation system including a first device and a second device. The first device includes: a first object map generation unit configured to calculate, using first image information that is image information acquired by the first device, a first object map indicating a type of an object and a position of the object; and a first communication unit configured to transmit the first object map to the second device. The second device includes: a second object map generation unit configured to calculate, using second image information that is image information acquired by the second device, a second object map indicating a type of an object and a position of the object; and a comparison unit configured to compare the first object map and the second object map.

A calculation system according to a second aspect of the invention is a calculation system including a server and a plurality of terminals. Each of the plurality of terminals includes: an object map generation unit configured to calculate, using acquired image information, an object map indicating a type of an object and a position of the object; and a terminal communication unit configured to transmit the object map to the server. The server includes: an object map database storing the object map received from the plurality of terminals; and an error determination unit configured to determine correctness of the type of the object included in the received object map by a majority decision based on information stored in the object map database, and notify the terminal from which the object map is transmitted of the determination.

A calculation method according to a third aspect of the invention is a calculation method performed by a calculation system including a first device and a second device. The calculation method includes: by the first device, calculating, using first image information that is image information acquired by the first device, a first object map indicating a type of an object and a position of the object; and transmitting the first object map to the second device, and by the second device, calculating, using second image information that is image information acquired by the second device, a second object map indicating a type of an object and a position of the object; and comparing the first object map and the second object map.

Advantageous Effect

According to the invention, it is possible to detect a discrimination error in a type of an object.

FIG. 1 is a functional configuration diagram of a calculation system according to a first embodiment.

FIG. 2 is a diagram showing an example of information created by a second device.

FIG. 3 is a diagram showing an example of an object map generated by a second object map generation unit.

FIG. 4 is a diagram showing a position calculation method of a movable object by an object map generation unit.

FIG. 5 is a flowchart showing an operation of the calculation system according to the first embodiment.

FIG. 6 is a functional configuration diagram of a calculation system according to a second embodiment.

FIG. 7 is a diagram showing an example of an object map including a movable object position error.

FIG. 8 is a flowchart showing an operation of the calculation system according to the second embodiment.

FIG. 9 is a diagram for showing processing of step S 206 .

FIG. 10 is a functional configuration diagram of a calculation system according to a third embodiment.

FIG. 11 is a flowchart showing an operation of the calculation system according to the third embodiment.

FIG. 12 is a functional configuration diagram of a calculation system according to a fourth embodiment.

FIG. 13 is a functional configuration diagram of a calculation system according to a fifth embodiment.

FIG. 14 is a functional configuration diagram of a calculation system according to a sixth embodiment.

FIG. 15 is a flowchart showing an operation of the calculation system according to the sixth embodiment.

First Embodiment

Hereinafter, a calculation system according to a first embodiment will be described with reference to FIGS. 1 to 5 .

FIG. 1 is a functional configuration diagram of a calculation system S 1 in the first embodiment. The calculation system S 1 includes a first device 101 A and a second device 101 B. Configurations of the first device 101 A and the second device 101 B are common except for one functional block. The first device 101 A and the second device 101 B are, for example, vehicle-mounted cameras that have built-in cameras and are mounted in a vehicle. However, the first device 101 A and the second device 101 B are any of a device attached to a vehicle-mounted camera, a fixed camera, and a device attached to a fixed camera. The first device 101 A and the second device 101 B may be the same or different. For example, the first device 101 A may be a fixed camera and the second device 101 B may be a device attached to a vehicle-mounted camera. However, the configuration in which the first device 101 A and the second device 101 B receive a video signal from the same imaging element is excluded from a configuration of the present embodiment.

The first device 101 A includes a first image acquisition unit 1 A, a first time acquisition unit 2 A, a first orientation acquisition unit 3 A, a first position acquisition unit 4 A, a first inference unit 5 A, a first object map generation unit 6 A, a first object map storage unit 7 A, a first control determination unit 8 A, and a first communication unit 9 A. The second device 101 B includes a second image acquisition unit 1 B, a second time acquisition unit 2 B, a second orientation acquisition unit 3 B, a second position acquisition unit 4 B, a second inference unit 5 B, a second object map generation unit 6 B, a second object map storage unit 7 B, a second control determination unit 8 B, and a second communication unit 9 B, which have similar functions as those of the first device 101 A. The second device 101 B further includes a comparison unit 10 B.

Calculation performed by the individual functional blocks included in the first device 101 A and the second device 101 B may be implemented by, for example, a CPU (not shown) that is a central processing unit, a ROM (not shown) that is a read-only storage device, and a RAM (not shown) that is a readable and writable storage device. Specifically, the calculation may be implemented by the CPU expanding a program stored in the ROM into the RAM and executing the program. Further, the calculation may be implemented by a field programmable gate array (FPGA) that is a rewritable logic circuit or an application specific integrated circuit (ASIC) instead of a combination of CPU, ROM, and RAM. Furthermore, the calculation may be implemented by a combination of different configurations, for example, a combination of CPU, ROM, RAM, and FPGA, instead of a combination of CPU, ROM, and RAM.

The first image acquisition unit 1 A acquires a first captured image captured by a camera (not shown). However, the camera may be built in the first device 101 A or may be a hardware device different from the first device 101 A. The first image acquisition unit 1 A outputs the acquired first captured image to the first inference unit 5 A.

The first time acquisition unit 2 A acquires a time using a global positioning satellite system or a network protocol. However, a receiving circuit that receives a signal from a satellite configuring the global positioning satellite system (hereinafter referred to as “GNSS receiving circuit”) or a communication circuit that performs communication using the network protocol may be built in the first device 101 A, or may be a different hardware device connected to the first device 101 A.

The first orientation acquisition unit 3 A acquires orientation, for example, an angle from 0 to 359 degrees where north is 0 degrees and east is 90 degrees, of the camera on which the first image acquisition unit 1 A acquires the first captured image. However, as described above, in the present embodiment, since the first device 101 A has the built-in camera, the first orientation acquisition unit 3 A may also acquire orientation of the first device 101 A. The first orientation acquisition unit 3 A may include a gyro, or may be a communication interface with a gyro (not shown) connected to the first device 101 A.

The first position acquisition unit 4 A acquires a position, for example, a combination of latitude and longitude, of the camera at which the first image acquisition unit 1 A acquires the first captured image. However, as described above, in the present embodiment, since the first device 101 A has the built-in camera, the first position acquisition unit 4 A may also acquire a position of the first device 101 A. The first position acquisition unit 4 A may include the receiving circuit that receives the signal from the satellite configuring the global positioning satellite system, or may be a communication interface with a GNSS receiving circuit (not shown) connected to the first device 101 A.

The first inference unit 5 A processes the first captured image acquired by the first image acquisition unit 1 A, and infers a type of a subject in the first captured image by using a classifier configured with a neural network acquired by learning in advance. Further, the first inference unit 5 A specifies the type of the subject captured in the first captured image and coordinates of the subject on the first captured image. In the following, the subject may be referred to as an “object” or a “movable object”, and is simply referred to as a “movable object” for the sake of convenience since it is not intended to exclude a stationary object, but a movable object that is not fixed to the ground is detected more frequently.

The first object map generation unit 6 A generates an object map periodically or at a timing defined by a user based on an inference result by the first inference unit 5 A, a time acquired by the first time acquisition unit 2 A, position information of the camera acquired by the first position acquisition unit 4 A, and orientation information of the camera acquired by the first orientation acquisition unit 3 A. The object map will be described below. The first object map storage unit 7 A stores a plurality of object maps generated by the first object map generation unit 6 A. Hereinafter, an object map generated by the first object map generation unit 6 A may be referred to as a “first object map”, and an object map generated by the second object map generation unit 6 B may be referred to as a “second object map”.

The first control determination unit 8 A performs calculation using the inference result output by the first inference unit 5 A, and outputs control information 151 A to the outside of the first device 101 A. A content of the calculation executed by the first control determination unit 8 A is not particularly limited. For example, when the first device 101 A is mounted on the vehicle, the control information 151 A indicating a steering angle based on the inference result may be output. Further, the first control determination unit 8 A may output, as the control information 151 A, the inference result as it is without performing special calculation.

The first communication unit 9 A performs communication with the second device 101 B and calculation described below. The first communication unit 9 A includes a communication module configured to perform wireless communication. The first communication unit 9 A acquires position information 152 B from the second device 101 B, and when it is determined that a condition is satisfied by the calculation described below, the first communication unit 9 A transmits an object map 153 A stored in the first object map storage unit 7 A to the second device 101 B. The above is the description of the configuration provided in the first device 101 A.

Configurations and operations of the second image acquisition unit 1 B, the second time acquisition unit 2 B, the second orientation acquisition unit 3 B, the second position acquisition unit 4 B, the second inference unit 5 B, the second object map generation unit 6 B, and the second object map storage unit 7 B, which are provided in the second device 101 B, are substantially the same as the configurations and the operations of the corresponding functional blocks in the first device 101 A, and only differences will be described below. Operations of the second control determination unit 8 B, the second communication unit 9 B, and the comparison unit 10 B will be described in detail.

The second image acquisition unit 1 B acquires a second captured image captured by a camera (not shown). The second captured image is an image different from the first captured image. The second image acquisition unit 1 B outputs the acquired second captured image to the second inference unit 5 B. The second orientation acquisition unit 3 B acquires orientation of the camera on which the second image acquisition unit 1 B acquires the second captured image. The second position acquisition unit 4 B acquires a position of the camera at which the second image acquisition unit 1 B acquires the second captured image. The second inference unit 5 B infers a type of a subject in the second captured image acquired by the second image acquisition unit 1 B, and specifies coordinates of the subject on the second captured image. An algorithm and implementation of the second inference unit 5 B may be the same as or different from that of the first inference unit 5 A. The second communication unit 9 B transmits the position information 152 B acquired by the second position acquisition unit 4 B to the first device 101 A.

The second object map generation unit 6 B generates an object map periodically or at a timing defined by the user based on an inference result by the second inference unit 5 B, a time acquired by the second time acquisition unit 2 B, position information of the camera acquired by the second position acquisition unit 4 B, and orientation information of the camera acquired by the second orientation acquisition unit 3 B. The second object map storage unit 7 B stores a plurality of object maps generated by the second object map generation unit 6 B.

The comparison unit 10 B compares the object map received from the first device 101 A with the object map created by the second device 101 B. Specifically, the comparison unit 10 B detects a contradiction between the object map received from the first device 101 A and the object map created by the second device 101 B on the assumption that the object map received from the first device 101 A has no error. Since images received from the first device 101 A and the second device 101 B are different, it does not matter that the object maps created by the first device 101 A and the second device 101 B do not completely match. However, for example, when in the object map created by the first device 101 A, a vehicle is detected at a certain position, and in the object map created by the second device 101 B, a pedestrian is detected at the same position, it is determined that there is a contradiction. The comparison unit 10 B outputs a comparison result to the second control determination unit 8 B.

The second control determination unit 8 B determines whether the object map created by the second device 101 B includes an error using the output of the comparison unit 10 B. When the second control determination unit 8 B determines that the object map created by the second device 101 B includes an error, the second control determination unit 8 B outputs control information 151 B instructing a control that attaches importance to safety, for example, a degenerate operation. When the second control determination unit 8 B determines that the object map created by the second device 101 B does not include an error, the second control determination unit 8 B outputs the control information 151 B instructing normal traveling.

FIG. 2 is a diagram showing an example of information created by the second device 101 B. In the example shown in FIG. 2 , three times t 1 , t 2 , and t 3 acquired by the second time acquisition unit 2 B, positions of the second device 101 B acquired by the second position acquisition unit 4 B and orientation of the second device 101 B acquired by the second orientation acquisition unit 3 B, and object maps generated by the second object map generation unit 6 B at each time are shown.

FIG. 3 is a diagram showing an example of an object map generated by the second object map generation unit 6 B. However, the rightmost column in FIG. 3 shows the comparison result of the object maps. As shown in FIG. 3 , positions and the number of movable objects described in the object map change with the time. The comparison result of the object maps shown in the rightmost column in FIG. 3 is a result acquired by the second device 101 B receiving the object map from the first device 101 A and by the comparison unit 10 B comparing the object maps. In FIG. 3 , the matched movable object is described as “matched”, the mismatched movable object is described as “mismatched”, and the movable object not compared is described as “not compared”. In FIG. 3 , the position of the movable object is described as a representative point of the movable object, for example, a position of a center of gravity, but a range of a space occupied by the movable object may be the position of the movable object. In this case, position information of each movable object has a width for each of the three dimensions of XYZ.

FIG. 4 is a diagram showing a position calculation method of a movable object by the first object map generation unit 6 A. A position calculation method of the movable object by the second object map generation unit 6 B is the same, and here, the position calculation method of the movable object by the first object map generation unit 6 A will be representatively described. FIG. 4( a ) shows a plan view and FIG. 4( b ) shows a side view. Assuming that the position of the first device 101 A is (x c , y c , z c ), a three-dimensional position (x m , y m , z m ) of the movable object is represented by the following Formula 1.

However, in the above Formula 1, L represents a distance from the camera to the movable object, and θ h and θ v respectively represent an angle of view in a plane and an angle of view in a side surface of the camera. Further, Δθ h represents an angle in a plane from the camera to the movable object when a length direction of the camera is taken as a reference axis, and Δθ h represents an angle in a side surface from the camera to the movable object.

FIG. 5 is a flowchart showing an operation of the calculation system S 1 according to the first embodiment. However, since an execution subject of each step described below is different for each step, the execution subject will also be described for each step. The calculation system S 1 executes processing described below in a predetermined time period. First, in step S 101 , the first object map generation unit 6 A and the second object map generation unit 6 B generate individual object maps. In subsequent step S 102 , the second communication unit 9 B transmits the position information acquired by the second position acquisition unit 4 B to the first device 101 A.

In subsequent step S 103 , the first communication unit 9 A of the first device performs the following determination. That is, the first communication unit 9 A determines whether a difference of at least one movable object position between in the position information received in step S 102 and in the object map generated by the first object map generation unit 6 A in step S 101 is within a predetermined threshold. When the first communication unit 9 A determines that the difference is within the predetermined threshold, the processing proceeds to step S 104 , and when the first communication unit 9 A determines that there is no movable object within the predetermined threshold, the processing shown in FIG. 5 is completed.

In step S 104 , the first communication unit 9 A transmits the object map generated by the first object map generation unit 6 A to the second device 101 B. In subsequent step S 105 , the comparison unit 10 B compares the two object maps, that is, the object map generated by the first object map generation unit 6 A and the object map generated by the second object map generation unit 6 B. In subsequent step S 106 , the comparison unit 10 B determines, when there is a movable object whose time and position are within predetermined threshold differences, whether the type of the movable object differs by at least one pair.

In step S 106 , when the comparison unit 10 B makes an affirmative determination, for example, when the first object map generation unit 6 A and the second object map generation unit 6 B output different types at the same position and the same time, the processing proceeds to step S 107 . In step S 106 , when the comparison unit 10 B makes a negative determination, the processing shown in FIG. 5 is completed. In step S 107 , the comparison unit 10 B transmits the comparison result to the second control determination unit 8 B, the second control determination unit 8 B detects an inference error of the movable object in the second device 101 B, and the processing shown in FIG. 5 is completed. Accordingly, the second control determination unit 8 B outputs control information instructing a control corresponding to occurrence of the inference error, for example, a degenerate operation, to the outside of the second device 101 B.

According to the first embodiment described above, the following operational effects can be acquired.

(1) The calculation system S 1 includes the first device 101 A and the second device 101 B. The first device 101 A includes: the first object map generation unit 6 A configured to calculate, using first image information that is image information acquired by the first device 101 A, a first object map indicating a type of an object and a position of the object; and the first communication unit 9 A configured to transmit the first object map to the second device 101 B. The second device 101 B includes: the second object map generation unit 6 B configured to calculate, using second image information that is image information acquired by the second device 101 B, a second object map indicating a type of an object and a position of the object; and the comparison unit 10 B configured to compare the first object map and the second object map. Therefore, it is possible to detect a discrimination error in the type of the object. Accordingly, the maintainability and reliability of the second device 101 B are improved. It is conceivable that many stationary objects such as buildings fixed to the ground are stored in a database in advance. However, since an update frequency and an accuracy of the database may not always be sufficient, there is certain significance in detecting a position of a movable object as well as a stationary object and estimating the type of the object.

(2) The first communication unit 9 A of the first device 101 A is configured to transmit the first object map when a difference between a position indicated by position information received from the second device 101 B and a position of the object included in the first object map is within a predetermined distance. Therefore, when the distance between the positions is large, the object map is not transmitted, and useless communication can be prevented.

(3) The first object map generation unit 6 A and the second object map generation unit 6 B set a movable object that is not fixed to a ground as a detection target.

(4) The first object map generation unit 6 A and the second object map generation unit 6 B are configured to calculate the type of the object by inference based on machine learning. The comparison unit 10 B is configured to detect an inference error by comparing the first object map and the second object map. Therefore, the type of the object can be calculated more accurately than in the case of using pattern matching.

(Modification 1)

At least one of the first inference unit 5 A and the second inference unit 5 B may not perform the inference using machine learning. In this case, a type of a subject may be determined using another known method, for example, pattern matching.

(Modification 2)

The second communication unit 9 B may not transmit position information to the first device 101 A, and the first device 101 A may unconditionally transmit the object map 153 A. That is, in the present modification, the first device 101 A, the second device 101 B, and the like may perform broadcast communication in which the created object map is transmitted to surroundings without limiting a destination.

(Modification 3)

In the first embodiment described above, the objects detected by the first inference unit 5 A and the second inference unit 5 B are a stationary object and a movable object. However, the first inference unit 5 A and the second inference unit 5 B may limit the detection target to a movable object.

(Modification 4)

In the first embodiment described above, functional configurations of the first device 101 A and the second device 101 B are not the same. However, both may have the same functional configuration. That is, the first device 101 A may further include a functional block having the same function as the comparison unit 10 B. In this case, the first device 101 A and the second device 101 B mutually transmit the position information and the object maps.

Second Embodiment

A calculation system according to a second embodiment will be described with reference to FIGS. 6 to 9 . In the following description, the same components as those of the first embodiment are denoted by the same reference numerals, and differences will be mainly described. Points that are not specifically described are the same as those of the first embodiment. The present embodiment is different from the first embodiment mainly in a point of considering an identification error.

FIG. 6 is a functional configuration diagram of a calculation system S 2 according to the second embodiment. In FIG. 6 , the same configurations as those in the first embodiment are denoted by the same reference numerals. The calculation system S 2 further includes a sensor error first registration unit 11 A and a sensor error second registration unit 11 B in addition to the configurations of the calculation system S 1 according to the first embodiment. Sensor error information such as a distance identification error and a GPS error is registered in the sensor error first registration unit 11 A and the sensor error second registration unit 11 B. The first object map generation unit 6 A and the second object map generation unit 6 B generate object maps including a movable object position error by using the respective information.

FIG. 7 is a diagram showing an example of the object map including the movable object position error. A movable object position including an error represents a range, not a single value. Δx in ±Δx is a constant. When the first line of FIG. 7 is specifically described, it is shown that an X coordinate of a movable object determined to be a pedestrian at a time t 1 is at any position from “x 1 −Δx 1 ” to “x 1 +Δx 1 ” when an error is added.

FIG. 8 is a flowchart showing an operation of the calculation system S 2 according to the second embodiment. The similar operations as those in the first embodiment are denoted by the same step numbers and the description thereof is omitted. First, in step S 201 , the first object map generation unit 6 A and the second object map generation unit 6 B generates individual object maps including a position error. Processing of subsequent step S 102 and step S 103 is similar to that in the first embodiment. However, when an affirmative determination is made in step S 103 , the processing proceeds to step S 204 .

In step S 204 , the first communication unit 9 A transmits the object map including the position error generated by the first object map generation unit 6 A to the second device 101 B. In subsequent step S 205 , the comparison unit 10 B compares the object maps including the two position errors. In subsequent step S 206 , the comparison unit 10 B determines whether there is a movable object whose movable object position ranges partially match in the three dimensions at a time within a predetermined threshold difference and whose movable object types are different in all three dimensions. The processing of this step will be described below with reference to the drawing.

FIG. 9 is a diagram showing the processing of step S 206 , and shows ranges of a movable object position including an error in object maps created respectively by a first device 102 A and the second device 102 B and movable object types. However, in FIG. 9 , the position of a movable object is shown separately for each of the X coordinate, the Y coordinate, and the Z coordinate. In the X coordinate, the range of the movable object position including the error that the first device 102 A infers the movable object as a person is x 1 <x<x 3 and x 4 <x<x 6 . Further, the range of the movable object position including the error that the second device 102 B infers the movable object as a vehicle is x 2 <x<x 5 . Therefore, the ranges of the movable object position match in the ranges of x 2 <x<x 3 and x 4 <x<x 5 . Further, in both ranges that match, the movable object types are different, which are person and vehicle.

Regarding the Y coordinate, the ranges of the movable object position match in both ranges of y 2 <y<y 3 and y 4 <y<y 5 , and the movable object types are different in both ranges. Furthermore, regarding the Z coordinate, the ranges of the movable object position match in both ranges of z 2 <z<z 3 and z 4 <z<z 5 , and the movable object types are different in both ranges. As in the example shown in FIG. 9 , the range of the movable object position including the error in the object map of the second device 102 B matches with a part of the range of the movable object position including the error in the object map of the first device 102 A in all three dimensions of XYZ, and when the movable object types are different in all three dimensions, an affirmative determination is made in step S 206 .

When an affirmative determination is made in step S 206 , the processing proceeds to step S 107 , and when a negative determination is made in step S 206 , the processing shown in FIG. 8 is completed. Since the processing of step S 107 is similar to that in the first embodiment, the description is omitted.

According to the second embodiment described above, the following operational effect can be acquired.

(5) The first object map generation unit 6 A is configured to calculate, based on a position error related to the first image information, the first object map including the position error in the position of the object. The second object map generation unit 6 B is configured to calculate, based on a position error related to the second image information, the second object map including the position error in the position of the object. The comparison unit 10 B is configured to compare the first object map including the position error and the second object map including the position error. Therefore, by generating the object maps including a position error of an object and comparing the object maps including movable object position errors among a plurality of devices, it is possible to detect an inference error of the movable object even if there is a position error of the object.

Third Embodiment

A calculation system according to a third embodiment will be described with reference to FIGS. 10 and 11 . In the following description, the same components as those of the first embodiment are denoted by the same reference numerals, and differences will be mainly described. Points that are not specifically described are the same as those of the first embodiment. The present embodiment is different from the first embodiment mainly in a point that only object maps included in detection ranges of two cameras are transmitted.

FIG. 10 is a functional configuration diagram of a calculation system S 3 according to the third embodiment. In FIG. 10 , the same configurations as those in the first embodiment are denoted by the same reference numerals. A first device 103 A includes a first registration unit 12 A, a first detection region information generation unit 13 A, and a common detection region specifying unit 14 A in addition to the configuration of the first device 101 A in the first embodiment. A second device 103 B includes a second registration unit 12 B and a second detection region information generation unit 13 B in addition to the configuration of the second device 101 B in the first embodiment.

Information including horizontal and vertical angles of view and the maximum detection length of a camera provided in the first device 103 A is input to the first registration unit 12 A. The information may be input to the first registration unit 12 A from the outside of the first device 103 A, or the first registration unit 12 A may read the information from a storage region provided in the first device 103 A. Information including horizontal and vertical angles of view and the maximum detection length of a camera provided in the second device 103 B is input to the second registration unit 12 B. The information may be input to the second registration unit 12 B from the outside of the second device 103 B, or the second registration unit 12 B may read the information from a storage region provided in the second device 103 B.

The first detection region information generation unit 13 A generates detection region information of the camera provided in the first device 103 A by using the following formula and information acquired by the first position acquisition unit 4 A, the first orientation acquisition unit 3 A, and the first registration unit 12 A.

x c ≤x≤x c +L max

− x tan θ h +y c ≤y≤x tan θ h +y c

− x tan θ v +z c ≤z≤x tan θ v +z c   [Formula 2]

Here, ranges of x, y, and z respectively represent a range of a detection region in a three-dimensional coordinate system. L max is the maximum detection length of the camera. The first detection region information generation unit 13 A outputs the generated detection region information to the common detection region specifying unit 14 A. The detection region information may be a specific numerical value acquired by evaluating the above formula, that is, a range of latitude or longitude, or may be a value of each parameter used in the above formula.

The second detection region information generation unit 13 B generates detection region information 154 B of the camera provided in the second device 103 B by using the same formula as the first detection region information generation unit 13 A and information acquired by the second position acquisition unit 4 B, the second orientation acquisition unit 3 B, and the second registration unit 12 B. The second detection region information generation unit 13 B transmits the generated detection region information 154 B to the first device 103 A via the second communication unit 9 B. The first communication unit 9 A of the first device 103 A outputs the detection region information 154 B received from the second device 103 B to the common detection region specifying unit 14 A. In the present embodiment, the second communication unit 9 B does not transmit position information of the camera to the first device 103 A.

The common detection region specifying unit 14 A specifies, based on the detection region information generated by the first device 103 A and the detection region information 154 B received from the second device 103 B, a common detection region, which is a region where the first device 103 A and the second device 103 can commonly detect an object. For example, the common detection region specifying unit 14 A specifies a region where two regions indicated by the two pieces of detection region information overlap as the common detection region, sets the maximum rectangle included in the common detection region, and outputs latitude and longitude of four vertices of the rectangle. The common detection region specifying unit 14 A outputs the specified common detection region to the first communication unit 9 A. When the two regions indicated by the two pieces of detection region information do not overlap, the common detection region specifying unit 14 A may output, for example, the common detection region as an empty set, or may not perform any output.

The first communication unit 9 A transmits, to the second device 101 B, only information 155 A of a movable object included in the common detection region set by the common detection region specifying unit 14 A in the object map 153 A stored in the first object map storage unit 7 A.

FIG. 11 is a flowchart showing an operation of the calculation system S 3 according to the third embodiment. The similar operations as those in the first embodiment are denoted by the same step numbers and the description thereof is omitted. First, in step S 301 , the first device 103 A and the second device 103 B generates individual object maps and the detection region information of the cameras. That is, in this step, the first object map generation unit 6 A, the second object map generation unit 6 B, the first detection region information generation unit 13 A, and the second detection region information generation unit 13 B operate.

In subsequent step S 302 , the second communication unit 9 B of the second device 103 B transmits the detection region information generated by the second detection region information generation unit 13 B to the first device 103 A. Upon receiving the detection region information, the first communication unit 9 A of the first device 103 A outputs the detection region information to the common detection region specifying unit 14 A. In subsequent step S 303 , the common detection region specifying unit 14 A specifies the overlap between the two pieces of detection region information as the common detection region, and outputs the common detection region to the first communication unit 9 A. The first communication unit 9 A determines whether at least a part of a detection region of the first device 103 A and a detection region of the second device 103 B match, that is, whether there is the common detection region. When the first communication unit 9 A determines that there is the common detection region, the processing proceeds to step S 304 , and when the first communication unit 9 A determines that there is no common detection region, for example, is an empty set, the processing shown in FIG. 11 is completed.

In step S 304 , the first communication unit 9 A transmits, to the second device 103 B, the object map 155 A included in the common detection region in the object map 153 A generated by the first object map generation unit 6 A, and the processing proceeds to step S 105 . The processing after step S 105 is similar to that in the first embodiment, and the description is omitted.

According to the third embodiment described above, the following operational effect can be acquired.

(6) The second device 103 B includes the second communication unit 9 B configured to transmit, to the first device 103 A, information indicating a second detection region that is a geographical range of a subject included in the second image information. The first communication unit 9 A of the first device 103 A is configured to transmit, to the second device 103 B, only information of an object included in the second detection region in the first object map. As described above, in the present embodiment, information of a detection region is shared between the first device 103 A and the second device 103 B, and the first device 103 A transmits the object map in the common detection region to the second device 103 B. Therefore, a communication load can be reduced as compared with a case where all data of the object map is transmitted. Therefore, inference error detection of an object can be performed more quickly and efficiently than in the first embodiment and the second embodiment.

Fourth Embodiment

A calculation system according to a fourth embodiment will be described with reference to FIG. 12 . In the following description, the same components as those of the third embodiment are denoted by the same reference numerals, and differences will be mainly described. Points that are not specifically described are the same as those of the third embodiment. The present embodiment is different from the first embodiment mainly in a point of considering a time shift in evaluation of the detection region.

First, problem awareness which is a background of the present embodiment will be described. Since calculation and communication take time, communication latency is inevitable. Therefore, in the third embodiment, when the common detection region specifying unit 14 A receives the two pieces of detection region information at the same time, the information of the second device 103 B is older than the information of the first device 103 A. In this case, it is difficult to specify the common detection region at the same time. Further, in object maps at a plurality of times stored in the first object map storage unit 7 A of the first device 103 A, when there is no object map at a time of the detection region information received from the second device 103 B, the object maps cannot be compared. The present embodiment solves the problem.

FIG. 12 is a functional configuration diagram of a calculation system S 4 according to the fourth embodiment. In FIG. 12 , the same configurations as those in the third embodiment are denoted by the same reference numerals. A first device 104 A includes a detection region information correction unit 15 A and an object map information complementation unit 16 A in addition to the configuration of the first device 103 A in the third embodiment. A configuration of a second device 104 B is the same as the configuration of the second device 103 B in the third embodiment.

The detection region information correction unit 15 A corrects current detection region information of the first device 104 A to detection region information of the first device 104 A at a past time, that is, at a time slightly before. Specifically, the detection region information correction unit 15 A corrects the current detection region information of the first device 104 A to the detection region information of the first device 104 A at the time of the detection region information of the second device 104 B.

For example, the detection region information correction unit 15 A is caused to store detection region information at a plurality of times t 1 and t 2 received from the first detection region information generation unit 13 A. Then, when the detection region information correction unit 15 A receives detection region information at a time t between the time t 1 and the time t 2 from the second device 104 B, the detection region information correction unit 15 A performs the following processing. That is, the detection region information correction unit 15 A generates the detection region information at the time t assuming that during the period from time t 1 to time t 2 , (x c , y c , z c ), θ h , θ v in Formula 2 change linearly with a change in the time. L max in Formula 2 is constant regardless of time.

The object map information complementation unit 16 A generates an object map at the time t between the time t 1 and the time t 2 using the object maps at the plurality of times t 1 and t 2 . For example, the object map information complementation unit 16 A generates the object map at the time t between the time t 1 and the time t 2 assuming that a movable object position changes linearly with a change of time from the time t 1 to the time t 2 if movable object types of a specific movable object match at the time t 1 and the time t 2 .

When the second device 104 B transmits time information together with the detection region information, the detection region information correction unit 15 A may use the received time information as the time t. When the second device 104 B does not transmit the time information together with the detection region information, a time, which is acquired by subtracting a known communication latency, for example, 10 ms from the time when the second device 104 B receives the detection region information, is the time t.

According to the fourth embodiment described above, the following operational effect can be acquired.

(7) The first device 104 A includes: the detection region information correction unit 15 A configured to acquire first detection regions that are geographical ranges of a subject included in the first image information at a plurality of times, and calculate the first detection region at any time; and the object map information complementation unit 16 A configured to calculate the first object map at any time by using the first object maps at a plurality of times calculated by the first object map generation unit 6 A. The first communication unit 9 A of the first device 104 A is configured to, when receiving the second detection region from the second device 104 B, transmit to the second device, the first object map at a time related to the second detection region calculated by the object map information complementation unit 16 A, that is, information of an object included in a common region between the second detection region and the first detection region at the time related to the second detection region which is calculated by the detection region information correction unit 15 A. In the fourth embodiment described above, by adding a function of correcting detection region information and a function of complementing object map information, it is possible to perform inference error detection of a movable object by comparing object maps even if there is communication latency between devices or when there is no object map at a specific time.

Fifth Embodiment

A calculation system according to a fifth embodiment will be described with reference to FIG. 13 . In the following description, the same components as those of the first embodiment are denoted by the same reference numerals, and differences will be mainly described. Points that are not specifically described are the same as those of the first embodiment.

FIG. 13 is a functional configuration diagram of a calculation system S 5 according to the fifth embodiment. In FIG. 13 , the same configurations as those in the first embodiment are denoted by the same reference numerals. A second device 105 B further includes a communication destination device abnormality determination unit 17 B in addition to the configuration of the second device 101 B in the first embodiment. In the present embodiment, the comparison unit 10 B also outputs a comparison result of object maps to the communication destination device abnormality determination unit 17 B. The communication destination device abnormality determination unit 17 B outputs, to the second communication unit 9 B, an abnormality determination signal for changing a communication destination when an inference error of a movable object is detected continuously for a predetermined number of times or more. The second communication unit 9 B changes the communication destination when receiving the abnormality determination signal. Two or more communication destinations are registered in advance in the second communication unit 9 B, and when the abnormality determination signal is received, the communication destination is switched in order.

According to the fifth embodiment described above, the following operational effect can be acquired.

(8) The second device 105 B includes the communication destination device abnormality determination unit 17 B configured to determine an abnormality of a first device 105 A according to a plurality of comparison results of movable object maps, and change a communication destination of the second device 105 B. In another embodiment, when the first inference unit 5 A of the first device is faulty and a type of an object is always wrong, the comparison unit determines that an inference error of a movable object occurs in the second device even though the type of the object calculated by the second device is correct. However, according to the present embodiment, when a failure occurs in the first device 105 A, the communication destination of the second device 105 B can be changed to a device other than the first device 105 A. Therefore, it is possible to prevent the case where an inference error in the second device 105 B is determined, which causes an unnecessary abnormal stop.

Sixth Embodiment

A calculation system according to a sixth embodiment will be described with reference to FIGS. 14 and 15 . In the following description, the same components as those of the first embodiment are denoted by the same reference numerals, and differences will be mainly described. Points that are not specifically described are the same as those of the first embodiment. The present embodiment is different from the first embodiment mainly in a point that object maps are collected from a large number of devices and a majority decision is made.

FIG. 14 is a functional configuration diagram of a calculation system S 6 according to the sixth embodiment. In FIG. 14 , the same configurations as those in the first embodiment are denoted by the same reference numerals. The calculation system S 6 includes two or more first devices 106 A having the same configuration as the first device 101 A in the first embodiment, and a server 116 . A functional configuration of the first device 106 A is similar to that of the first embodiment. However, in the present embodiment, the first communication unit 9 A communicates with a server 20 and transmits not only an object map but also position information. Hereinafter, the first device 106 A may be referred to as a “terminal”.

The server 116 includes a server communication unit 9 C that communicates with a plurality of first devices 106 A, an object map DB 18 C that stores the object map 153 A transmitted from the plurality of first devices 106 A as a database (DB), and an error determination unit 19 C that determines an inference error of a movable object. The server communication unit 9 C receives the object map 153 A and position information from the first device 106 A and records in the object map DB 18 C. When the first device 106 A transmits time information together with the object map 153 A, the time information is also recorded in the object map DB 18 C, and when the first device 106 A does not transmit the time information, the time of receiving is recorded together with the object map 153 A.

When the server 116 receives the object map 153 A from any of the first devices 106 A, the error determination unit 19 C performs the following processing. That is, the error determination unit 19 C reads, from the object map DB 18 C, an object map whose differences of a time and a movable object position with respect to the received object map 153 A are within thresholds. Then, the error determination unit 19 C determines whether a movable object type of an object map in the device whose time and movable object position are within the threshold differences matches with a result of a majority decision related to a movable object type of the object map stored in the object map DB 18 C, for example. Further, the error determination unit 19 C notifies the first device 106 A from which the object map 153 A is transmitted of a notification 159 indicating whether the movable object type matches the result of the majority decision.

The first control determination unit 8 A of the first device 106 A changes a control content using the notification 159 received from the server 116 . For example, when the notification 159 indicating that the movable object type does not match the result of the majority decision is received from the server 116 , the first control determination unit 8 A outputs control information instructing a control that attaches importance to safety, for example, a degenerate operation since the inference includes an error. Further, when the notification 159 indicating that the movable object type matches the result of the majority decision is received from the server 116 and no special notification is received from the server 116 , the first control determination unit 8 A outputs control information instructing normal traveling.

FIG. 15 is a flowchart showing an operation of the calculation system S 6 according to the sixth embodiment. The similar operations as those in the first embodiment are denoted by the same step numbers and the description thereof is omitted. First, in step S 101 , the first device 106 A generates an object map. In subsequent step S 402 , the first device 106 A transmits, to the server 116 , the object map generated in step S 101 and position information acquired by the first position acquisition unit 4 A.

In subsequent step S 403 , the error determination unit 19 C of the server 116 determines whether there is one or more pieces of information of a movable object whose time and position with respect to the object map received in the object map DB 18 C are within predetermined thresholds. When the error determination unit 19 C determines that there is at least one piece of information of the movable object that satisfies the condition, the processing proceeds to step S 404 , and when the error determining unit 19 C determines that there is no such information, the processing shown in FIG. 15 is completed since comparison cannot be performed.

In step S 404 , the error determination unit 19 C counts types of the movable object that is determined to satisfy the condition in step S 403 . For example, the error determination unit 19 C counts the received object map and the movable object types of the movable object whose time and position are within the predetermined thresholds, and calculates that the total is “3” for vehicle and “1” for person. In subsequent step S 405 , the error determination unit 19 C uses the result of the count in step S 404 to determine whether a majority decision, that is, the movable object type having the largest total number does not match the movable object type in the object map received in step S 402 . When the error determination unit 19 C determines that the movable object types do not match, the error determination unit 19 C transmits the notification 159 indicating that the movable object types do not match to the first device 106 A from which the object map is transmitted in step S 102 , and the processing proceeds to step S 107 .

When the error determination unit 19 C determines that the movable object types match in step S 405 , the error determination unit 19 C transmits the notification 159 indicating that the movable object types match to the first device 106 A from which the object map is transmitted in step S 102 , and the processing shown in FIG. 15 is completed. However, when the error determination unit 19 C determines that the movable object types match in step S 405 , the processing shown in FIG. 15 may be completed without transmitting any notification to the first device 106 A from which the object map is transmitted in step S 102 .

According to the sixth embodiment described above, the following operational effect can be acquired.

(9) The calculation system S 6 includes the server 116 and the plurality of terminals 106 A. Each of the plurality of terminals 106 A includes: the first object map generation unit 6 A configured to calculate, using acquired image information, the object map indicating a type of an object and a position of the object; and the communication unit 9 A configured to transmit the object map to the server 116 . The server 116 includes: the object map database 18 C storing the object maps received from the plurality of terminals 106 A; and the error determination unit 19 C configured to determine correctness of the type of the object included in the received object map by a majority decision based on information stored in the object map database 18 C, and notify the terminal from which the object map is transmitted of the determination. Therefore, by comparing an inference result of a movable object in one terminal with a majority decision inference result of the movable object in a plurality of other terminals, the inference error of the movable object in the one terminal can be efficiently detected.

For example, in the first embodiment, the inference error of the movable object is detected by the communication between the first device 101 A and the second device 101 B. However, in this method, not only when the inference error of the movable object occurs in the second device 101 B, but also when the inference error of the movable object occurs in the first device 101 A, the inference error of the movable object in the second device 101 B is detected. Therefore, it is impossible to specify the device in which the inference error of the movable object occurs. However, according to the present embodiment, this problem can be solved.

Functional block configurations in the embodiments and modifications described above are merely examples. Some functional configurations shown as separate functional blocks may be integrated, or a configuration represented by one functional block diagram may be divided into two or more functions. A part of functions of each functional block may be provided in another functional block.

The embodiments and modifications described above may be combined with each other. Although various embodiments and modifications have been described above, the invention is not limited to the embodiments and modifications. Other embodiments that are regarded within the scope of the technical idea of the invention are also included within the scope of the invention.

1 A: first image acquisition unit

1 B: second image acquisition unit

5 A: first inference unit

5 B: second inference unit

6 A: first object map generation unit

6 B: second object map generation unit

8 A: first control determination unit

8 B: second control determination unit

9 A: first communication unit

9 B: second communication unit

9 C: server communication unit

10 B: comparison unit

13 A: first detection region information generation unit

13 B: second detection region information generation unit

14 A: common detection region specifying unit

15 A: detection region information correction unit

16 A: object map information complementation unit

17 B: communication destination device abnormality determination unit

18 C: object map database

19 C: determination unit

20 : server

101 A to 105 A: first device

101 B to 105 B: second device

106 A: terminal

116 : server