Face identification device and face identification method

1. Field of the Invention

The present invention relates to a face identification technique which detects human faces from images taken by a camera and compares them with a previously registered face, thereby detecting a specific person.

2. Description of the Prior Art

In order to search for a specific person like a criminal on a wanted list, a system is available that videotapes an unspecified number of the general public and records their images by installing surveillance cameras in airports, stations, shopping centers, busy streets, and the like. In this system, videotaped images are recorded as they are or face images are exclusively recorded by being extracted from the videotaped images. In these methods, there is a problem since the face images of the people other than the specific person are also recorded, failing to protect their rights of portrait or piracy.

Therefore, there is a system now available according to which the face image of a person videotaped by a surveillance camera is compared with the face image of the specific person registered: the face image of a videotaped person is recorded if these face images match with each other, whereas when these face images do not match with each other, the face image of the videotaped person is abandoned. In this system, out of the face images videotaped, those which do not match with the face image of the registered specific person are not kept, thereby securing protection of their privacy.

However, in the above-described system, what is recorded is only the face image matching with the face image of the specific person, and other useful information including the dress and baggage of the specific person and the situation around him or her are lost. As a result, when there is a match and a responsible official rushes to the display monitor by a report or when recorded images are confirmed later, the absence of information other than his or her face makes it difficult to search for the specific person.

On the other hand, in order to protect individuals' privacy, there is an approach of applying a mosaic process on face images as described in Japanese Laid-Open Patent Application No. 2001-86407. However, in this patent application, skin color is detected from videotaped human images and a mosaic process is automatically applied to face images, which forces all face images videotaped to be applied with a mosaic process, thereby making it impossible to apply a mosaic process to the face images excluding the face image of the specific person.

Japanese Laid-open Patent Application No. 2000-216 also discloses privacy protection measures by applying a mosaic process to human body images. However, when the technique of this patent application is used to protect the privacy of the people other than the specific person, it is difficult to check the situation around the specific person, because the whole image of the specific person is applied with the mosaic process.

The present invention, which is intended to solve the aforementioned problems, has the object of facilitating confirmation of information other than the face of a specific person and situation around the specific person, while protecting the right of portrait or privacy of the people other than the specific person.

A face identification device of the present invention comprises: detection means for detecting face images from human body images taken by a camera; storage means in which a face image of a specific person is previously stored; determination means for determining whether a face image detected by the detection means matches with the face image stored in the storage means by comparing both face images; and abstraction means for applying an abstraction process to a predetermined face image out of the face images detected by the detection means in order to make the predetermined face image unrecognizable. The abstraction means applies the abstraction process exclusively to a detected face image when the determination means determines that both face images do not match with each other. On the other hand, the abstraction means does not apply the abstraction process to a detected face image when the determination means determines that both face images match with each other.

A face identification method of the present invention comprises the steps of: detecting face images from human body images taken by a camera; determining whether a detected face image matches with the face image previously stored by comparing both face images; applying an abstraction process exclusively to a detected face image in order to make the detected face image unrecognizable, when it is determined that both face images do not match with each other; and not applying the abstraction process to the detected face image when it is determined that both face images match with each other.

The term “match” referred to in the above description means not only a complete match between a detected face image and the face image previously stored, but also a high degree of resemblance between these face images.

According to the present invention, when a detected face image and the registered face image do not match with each other, that is, when the detected face does not belong to the specific person, an abstraction process is applied on the face image in order to protect the right of portrait or privacy of the people other than the specific person. Furthermore, the abstraction process is applied only to the face images of the people, keeping images other than their faces, such as their dress and baggage. On the other hand, when a detected face image and the registered face image match with each other, that is, when the detected face belongs to the specific person, both the face image and images including the dress and baggage of the specific person are kept without being applied with the abstraction process. Thus keeping the images for the dress and baggage of a specific person and of the other people around as they are helps facilitate the search for the specific person by using these images as an important clue.

A typical example of the abstraction process in the present invention is a mosaic process. The use of the well-known mosaic process can abstract face images easily. Besides the mosaic process, various other methods are available for abstraction, such as blurring or filling in face image portions.

In the present invention, when it is determined that a detected face image matches with the stored face image, a marker is applied to the detected face image portion instead of applying an abstraction process thereto. The application of a marker makes a specific person distinguishable at a glance.

In the present invention, when it is determined that a detected face image matches with the stored face image, the image of the specific person applied with the abstraction process neither to his or her face nor remaining parts and the images of the other people applied with the abstraction process exclusively to their faces are displayed. At the same time, a warning is outputted. This system realizes a quick response because the warning is outputted at the time of the detection of a specific person. Concerning a specific person, the videotaped image is displayed as it is, and as for the people other than the specific person, the abstraction process is applied to their faces to protect their privacy while the images of their dress and baggage are displayed as they are. This makes it possible to obtain useful information about the specific person and the situation around him or her, thereby facilitating the search for the specific person.

FIG. 1 shows a block diagram of the face identification system to which the present invention is applied.

FIG. 2 shows a flowchart depicting the procedure for registering face images.

FIG. 3 shows a flowchart depicting the procedure of the face identification method of the present invention.

FIG. 4 shows a flowchart depicting a specific procedure of identification.

FIG. 5 shows an image example to which mosaic process has been applied.

FIG. 6 shows a flowchart depicting the identification procedure in the case where a face image is registered afterwards.

FIG. 7 shows a flowchart depicting another embodiment of the face identification method.

FIG. 8 shows a flowchart depicting the identification procedure in the case where a face image is registered afterwards in another embodiment.

FIG. 9 shows a block diagram showing another embodiment of the present invention.

FIG. 1 shows a block diagram of the face identification system to which the present invention is applied. In the drawing, the face identification system is composed of surveillance cameras 1 for videotaping the images of people, and the face identification device 2 of the present invention for identifying face images of the people videotaped by the surveillance cameras 1 . A plurality of surveillance cameras 1 are installed in airports, stations, shopping centers, busy streets, and other places where a number of people gather, and each of the cameras 1 is connected with the face identification device 2 installed in a monitor center or the like.

In the face identification device 2 , computer 3 is for face identification; reference numeral 4 is an identification results output device for outputting the results of face identification by the computer 3 ; and storage device 5 stores the face image data of a specific person (e.g., a suspect on the wanted list). Image input devices 6 and 7 are for registering a face image with the storage device 5 . Input device 6 is a scanner, and input device 7 is a digital camera. The computer 3 includes detection means for detecting face images from images of people videotaped by the surveillance cameras 1 ; determination means for determining the presence or absence of a match between a detected face image and the face image stored in the storage device 5 by comparing them; and abstraction means for applying an abstraction process to a predetermined face image out of the detected face images out of the detected face images so as to make the face unrecognizable. The identification results output device 4 is composed of a display monitor for displaying identification results, and other output devices. The storage device 5 can be a hard disk or the like, which composes the storage means of the present invention. Besides a hard disk, an optical disk can be used as the storage means.

Operation of the face identification system having the above-explained structure will be described as follows. In this system, the face image of a specific person is previously registered in the storage device 5 . FIG. 2 shows a flowchart depicting the procedure of face image registration, which is executed by the computer 3 . At first, an image including the face of a specific person to be registered is captured into the computer 3 from the scanner 6 or the digital camera 7 (Step S 1 ). Next, the face to be registered is detected from the captured image (Step S 2 ). This face detection can be done by using a well-known approach, such as skin color detection, abstraction of difference with a background image, or abstraction of face likelihood from pattern matching, After the face detection, featuring points such as eyes, nose, mouth, and ears are detected from the face image (Step S 3 ), and the amount of feature about the shape and position of each detected featuring point is extracted (Step S 4 ). This abstraction of feature amount also can be done by using a well-known approach, such as extraction of the amount of feature from gray-scale images including eyes, nose, mouth, and ears by template matching. The amount of feature extracted is stored and registered in the storage device 5 as face image data (Step S 5 ).

FIG. 3 shows a flowchart depicting the procedure of the face identification method of the present invention, the procedure being executed by the computer 3 . In the computer 3 , images videotaped by the surveillance cameras 1 are captured as still images at the interval of e.g. one second (Step S 11 ), and human faces are detected from the captured images (Step S 12 ). If there is no face detected (Step S 13 : NO), the process goes back to Step S 11 and captures images in the next timing. If a face is detected (Step S 13 : YES), the detected face image is compared with the face image previously registered in the storage device 5 (Step S 14 ) so as to determine whether or not these face images match with each other (Step S 15 ).

FIG. 4 shows a flowchart depicting the specific procedure of the comparison in Step S 14 . In the comparison, featuring points (eyes, nose, mouth, and the like) are detected from the detected face image (Step S 31 ), and the amount of feature about the shape and position of each detected featuring point is extracted (Step S 32 ). The extracted amount of feature is compared with the amount of feature of the face registered in the storage device 5 (Step S 33 ). Based on the comparison results, it is determined whether these face images match with each other or not at Step S 15 shown in FIG. 3 . The term “match” referred to in this case means not only a complete match, but also a high degree of resemblance between these face images. To be more specific, the degree of resemblance is calculated from the amount of feature of each face image, and is compared with a predetermined threshold value. When the degree is equal to or over the threshold value, it is determined as a match.

As the results of the determination at Step S 15 , when a detected face image and the registered face image do not match with each other (Step S 15 :NO), the detected face image does not belong to the specific person registered. Therefore, the mosaic process is applied to the detected face image portion (Step S 16 ). FIG. 5 shows an example of images to which the mosaic process has been applied The mosaic process 12 is applied to each face image of the persons 10 B to 10 F except the specific person so as to abstract their faces unrecognizably. The mosaic process 12 is applied exclusively to the face image portions, keeping dress and baggage portions unprocessed.

On the other hand, when a detected face image matches with the registered face image (Step S 15 :Yes), it means that the detected face image belongs to the registered specific person, so a marker 13 is applied to the detected face image portion instead of applying the mosaic process thereto (Step S 17 ). In FIG. 5 , the person 10 A is the specific person, and the mosaic process 12 is not applied to the face image portion of the specific person 10 A, thereby keeping the face 11 the same as when it is videotaped by the surveillance cameras 1 . The face 11 is enclosed by the marker 13 to make the specific person distinguishable at a glance. The mosaic process 12 is not applied to the dress or baggage of the specific person 10 A. In other words, concerning the specific person 10 A, the image videotaped by the surveillance cameras 1 appears as it is.

After the process in Step S 17 , the identification results output device 4 displays, with a warning, the images shown in FIG. 5 including the image of the specific person 10 A whose face and remaining parts are not applied with the mosaic process 12 , and the images of the persons 10 B to 10 F whose faces are exclusively applied with the mosaic process (Step S 18 ). This warning is displayed as a message including the name of the specific person, the time and date of the image videotaped, and the place videotaped. Furthermore, an alarm is issued as necessary with the display in order to draw the responsible official's attention. This enables the official to prepare the deployment of members for the search for the specific person.

Since the faces of the people other than the specific person 10 A are applied with the mosaic process 12 as shown in FIG. 5 , when the images are displayed on the display monitor, there is no fear of violating their rights of portrait or privacy. Furthermore, the mosaic process 12 is applied exclusively to their faces, keeping the remaining parts unprocessed, which enables easy recognition of the dress and baggage of the people around the specific person 10 A. On the other hand, concerning the specific person 10 A, the videotaped image of the whole body appears as it is, making it possible to confirm his dress and baggage in addition to his face. Thus, the mosaic process 12 is applied exclusively to face portions which raise the issue of privacy, and the remaining parts are kept the same as when they are videotaped, thereby facilitating the search for the specific person 10 A by using the dress and baggage as a useful clue.

After Steps S 16 to S 18 are executed, it is determined whether matching is done for all the detected faces (Step S 19 ). When there is still a face to be checked (S 19 : NO), the process goes back to Step S 14 to check the next face. When all the detected faces have been checked (S 19 : YES), the image to which the procedure of Steps S 16 and S 17 has been applied is recorded to the storage device 5 (Step S 20 ). Later, the process goes back to Step S 11 , and videotaped images are captured from the surveillance cameras 1 again in the next timing (in one second in this case). Hereafter, the procedure in Steps S 12 to S 20 is repeated.

The above example describes the case where images videotaped by the surveillance cameras 1 are captured as still images at the interval of one second; however, it is also possible to adopt a system which detects human faces from motion images videotaped by the surveillance cameras 1 and checks the faces while automatically chasing them. In this system, there is no need for the repeated recording of the image of the same person, which can reduce the amount of image data to be recorded in the storage device 5 , thereby enabling miniaturization of the storage device 5 .

Images stored in the storage device 5 can be outputted by being displayed or printed by the identification results output device 4 later at any point in time. In this case, too, the privacy of the people other than the specific person 10 A is protected by applying the mosaic process to their faces, whereas the images for the dress and baggage including those of the specific person are stored as they are. This information can be used as a clue to facilitate the search for the specific person 10 A.

In the aforementioned embodiment, the face image of a specific person is previously registered, and face images videotaped by surveillance cameras are compared with the registered face image so as to detect the specific person; however, it is also possible to detect a specific person registered afterwards by using images stored in the storage device 5 . This system enables the detection of people whom the police have been requested to find or stray children.

FIG. 6 shows a flowchart depicting the procedure of identification in the case of registering afterwards as described above, the procedure being executed by the computer 3 . At first, images stored in the storage device 5 are captured by the computer 3 (Step S 41 ). The images stored in the storage device 5 are unrecognizable because of the above-described mosaic process, so it is necessary to reconstruct the original images for comparison. Therefore, the mosaic process is cleared to reconstruct the original images (Step S 42 ). After the reconstruction of the original images, faces are detected from the reconstructed images (Step S 43 ). If no face is detected from the reconstructed images (Step S 44 : NO), the process goes back to Step S 41 to capture recorded images. On the other hand, if faces are detected from the reconstructed images (Step S 44 : YES), detected face images are compared with the face image which is registered afterwards (Step S 45 ). The face image registered afterwards is taken by the scanner 6 or the digital camera 7 and recorded to the storage device 5 in the same manner as the previously registered case.

If the comparison results in Step S 45 indicate that a recorded image face does not match with the face registered afterwards (Step S 46 : NO), it is determined whether all faces detected from the recorded images have been checked or not (Step S 47 ). When there is a face left unchecked (Step S 47 : NO), the process goes back to Step S 45 to check the next face. When all faces have been checked (Step S 47 : YES), it is determined whether all the recorded images have been checked or not (Step S 48 ). If all recorded images are not checked (Step S 48 : NO), the process goes back to Step S 41 to capture the next recorded image from the storage device 5 and the aforementioned procedure is repeated. When the comparison for all recorded images is complete (Step S 48 : YES), the absence of the face image matching with the face image registered afterwards is reported by being displayed on the display monitor of the identification results output device 4 (Step S 49 ).

On the other hand, in Step S 46 , if a recorded face image matches with the face registered afterwards (Step S 46 : YES), the reconstructed original image is outputted to the identification results output device 4 and shown on the display monitor (Step S 50 ). And it is reported by being shown on the display monitor that there is a recorded face image which matches with the face registered afterwards (Step S 51 ).

In the comparison in the above-described case where a face image is registered afterwards, it is preferable in terms of security that the process for reconstructing original images by clearing the mosaic process applied to images stored in the storage device 5 can be done exclusively by an authorized person (e.g., system administrator). FIGS. 7 and 8 show flowcharts depicting the embodiment of this case.

FIG. 7 shows a flowchart corresponding to the flowchart shown in FIG. 3 , so the same steps as those in FIG. 3 are referred to with the same symbols. The procedure in Steps S 11 to S 19 is the same as in FIG. 3 , so its explanation is omitted. When all faces have been checked in Step S 19 (Step S 19 : YES), the face image of a recorder who records images is captured (Step S 20 a ); the captured face image of the recorder is embedded as a “digital watermark” in the images processed in Steps S 16 and S 17 ; and the images with the “digital watermark” are stored in the storage device 5 (Step S 20 b ). The term “recorder” used in this case can be a system administrator, and the face image of the recorder which is captured in Step S 20 a is previously taken by the scanner 6 or the digital camera 7 and stored in the storage device 5 .

The following is a description on the procedure for reconstructing original images. FIG. 8 shows a flowchart corresponding to the flowchart of FIG. 6 , so the same steps as those in FIG. 6 are referred to with the same symbols. First, images stored in the storage device 5 are captured in the computer 3 (Step S 41 ). Next, the face image of the operator who performs a reconstructing operation of images is captured (Step S 41 a ). The face image of this operator is taken by the scanner 6 or the digital camera 7 at this point in time. Then, it is determined whether the face image data captured in Step S 41 a matches with the data of “digital watermark” embedded in the images captured in Step S 41 or not (Step S 41 b ). If these pieces of data do not match with each other (Step S 41 b : NO), it is determined that the operator who is going to perform the reconstructing operation is not the above-mentioned recorder (system administrator), and the process is terminated. On the other hand, when these pieces of data match with each other (Step S 41 b : YES), it is determined that the operator and the recorder (system administrator) are the same person, and the mosaic process is cleared to reconstruct the original images (Step S 42 ). Since the procedure from Step S 42 onward is equal to the procedure in FIG. 6 , the explanation will be omitted.

As described hereinbefore, using the face image of the recorder as a “digital watermark” makes it possible to authorize only the recorder to perform reconstruction and to prohibit an unauthorized third party from clearing a mosaic process to reconstruct original images, thereby securing the security.

The present invention can adopt various other embodiments besides the embodiment described above. For example, in the above embodiment, a mosaic process is taken as an example of the abstraction process to be applied to face images; however, it is also possible to apply a process to blur face image portions, instead of applying the mosaic process. Alternatively, face image portions can be filled in black or white, instead of being applied with the mosaic process or the blurring process.

In the above embodiment, images which are videotaped by the surveillance cameras 1 and applied with a mosaic process or a marker are stored in the storage device 5 ; however, the images can be merely displayed on the display monitor of the identification results output device 4 . In this case, there is no need for storage to the storage device 5 .

In the above embodiment, face images which are registered either previously or afterwards are stored in the storage device 5 , and images which are videotaped by the surveillance cameras 1 and applied with a mosaic process or a marker are also stored in the same storage device 5 . Alternatively, as shown in FIG. 9 , a special image storage device 8 can be provided in addition to the storage device 5 so that face images to be registered can be stored in the storage device 5 whereas images videotaped by the surveillance cameras 1 can be stored in the image storage device 8 . The image storage device 8 can be composed of any kind of device dealing with recording media such as a video tape recorder, a DVD (Digital Versatile Disk) recorder, or a hard disk recorder. In that case, faces other than the face of the specific person are applied with an abstraction process, so there is no fear of violating their rights of portrait or privacy during reproduction.

According to the present invention, an abstraction process is applied exclusively to the face images of the people other than a specific person, thereby protecting their rights of portrait or privacy, whereas the images for the dress and baggage of the specific person and the other people are kept as they are. This information can be used as a clue to facilitate the search for the specific person.