Anchor recognition in reality system

The present application is a Divisional Application of the U.S. application Ser. No. 15/863,994, filed Jan. 8, 2018, which is herein incorporated by reference.

The disclosure relates to a reality system. More particularly, the disclosure relates to how to recognize anchor points in a reality system.

Virtual Reality (VR), Augmented Reality (AR), Substitutional Reality (SR), and/or Mixed Reality (MR) devices are developed to provide immersive experiences to users. When a user wearing a head-mounted device, the visions of the user will be covered by the immersive content shown on the head-mounted device. The immersive content shows a scenario of a specific space.

These reality systems usually equip a tracking component to locate users wearing the head-mounted device, so as to acknowledge a location or a movement of the user in the real world. The immersive content displayed on the head-mounted device will vary according to the location or the movement of the user, such that the user may have a better experience in the VR, AR, SR or MR scenario.

The disclosure provides a reality system including a first head-mounted device. The first head-mounted device is located in a physical environment. The first head-mounted device includes a camera unit, a communication unit and a processor. The camera unit is configured for capturing images of the physical environment over time. The communication unit is configured for communicating with second head-mounted devices. The processor is coupled to the camera unit and the communication unit. The processor is configured to: extract first candidate objects and first object features of the first candidate objects from the images captured by the camera unit; generate a first local determination about whether each of the first candidate objects is fixed or not; transmit the first object features and the first local determination to the second head-mounted devices; receive second object features of second candidate objects and second local determinations about whether each of the second candidate objects is fixed or not from the second head-mounted devices; and generate an updated determination about whether each of the first candidate objects is fixed or not according to the first local determination and the second local determinations.

The disclosure also provides a reality system including head-mounted devices and a server device. The head-mounted devices are located in a physical environment. Each of the head-mounted devices is configured to extract candidate objects, extract object features of the candidate objects and generate local determinations about whether each of the candidate objects is fixed or not in view of each of the head-mounted devices. The server device is communicated with the head-mounted devices, wherein the server device is configured to: collect the object features of the candidate objects and the local determinations from the head-mounted devices; generate an updated determination about whether each of the candidate objects is fixed or not according to the local determinations collected from the head-mounted devices; and, transmit the updated determination to each of the head-mounted devices.

The disclosure also provides a control method suitable for head-mounted devices located in a physical environment. The control method includes following operations. Images of the physical environment are captured over time by the head-mounted devices. Candidate objects and object features of the candidate objects are extracted from the images. Local determinations are generated about whether each of the candidate objects is fixed or not. The object features and the local determinations are shared between the head-mounted devices. An updated determination is generated about whether each of the candidate objects is fixed or not according to the local determinations shared between the head-mounted devices.

It is to be understood that both the foregoing general description and the following detailed description are by examples, and are intended to provide further explanation of the invention as claimed.

The disclosure can be more fully understood by reading the following detailed description of the embodiment, with reference made to the accompanying drawings as follows:

FIG. 1 is a schematic diagram illustrating a reality system according to an embodiment of this disclosure.

FIG. 2 is a functional block diagram illustrating the reality system in FIG. 1 according to an embodiment of the disclosure.

FIG. 3 is a flow diagram illustrating a control method suitable to be utilized on the head-mounted device in FIG. 1 and FIG. 2 .

FIG. 4A to FIG. 4C are schematic diagrams illustrating views of different head-mounted devices in the physical environment.

FIG. 5 is a schematic diagram illustrating a head-mounted device newly entered the physical environment in an embodiment of the disclosure.

FIG. 6 is a schematic diagram illustrating a reality system according to another embodiment of the disclosure.

FIG. 7 is a functional block diagram illustrating the reality system in FIG.

FIG. 8 is a flow diagram illustrating a control method suitable for the reality system in FIG. 6 .

Reference will now be made in detail to the present embodiments of the disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts.

Reference is made to FIG. 1 , which is a schematic diagram illustrating a reality system 100 according to an embodiment of this disclosure. The reality system 100 includes at least one head-mounted device located in a physical environment PE. In the embodiment shown in FIG. 1 , the reality system 100 includes three head-mounted devices 110 , 120 and 130 , which are mounted on heads of different users in the physical environment PE. However, the disclosure is not limited to three head-mounted devices as shown in FIG. 1 . The reality system 100 may include two or more head-mounted devices. These users may locate at different positions and face toward various directions in the physical environment PE.

The reality system 100 is one of a Virtual Reality (VR), an Augmented Reality (AR), a Substitutional Reality (SR), or a Mixed Reality (MR) system. In order to provide an immersive experience to the users, the head-mounted devices 110 , 120 and 130 are configured to construct a map of the physical environment PE and sense relative locations of the head-mounted devices 110 , 120 and 130 in the physical environment PE. In some embodiments, Simultaneous Localization and Mapping (SLAM) technology is utilized by the reality system 100 to construct the map of an unknown environment (e.g., the physical environment PE) while simultaneously tracking the head-mounted devices 110 , 120 and 130 within the unknown environment.

As shown in FIG. 1 , there are some candidate objects OBJ 1 -OBJ 9 located at different positions in the physical environment PE. The map of the physical environment PE can be constructed by observing the physical environment PE with some sensors on the head-mounted devices 110 , 120 and 130 . On the other hand, the relative locations of the head-mounted devices 110 , 120 and 130 in the physical environment PE can be sensed by calculating distances between one head-mounted device and reference points in the physical environment PE. While locating the head-mounted devices 110 , 120 and 130 in the physical environment PE, finding the valid reference points is important. In an ideal case, the reference points utilized to locate the head-mounted devices 110 , 120 and 130 in a locating algorithm should be fixed at same positions over time. Since the reference points are fixed at the same positions in the physical environment PE, the locations of the head-mounted devices 110 , 120 and 130 can be generated by measuring relative distances between the head-mounted devices 110 , 120 and 130 and the reference points. If the reference points are moved for some reasons, the locating results generated by the locating algorithm will be shifted and include errors. In this case, visions displayed to the users wearing the head-mounted devices 110 , 120 and 130 will change dramatically and inconsistently. In some embodiments of this disclosure, the reality system 100 is able to extinguish fixed objects from the candidate objects OBJ 1 -OBJ 9 in the physical environment PE.

Reference is further made to FIG. 2 , which is a functional block diagram illustrating the reality system 100 in FIG. 1 according to an embodiment of the disclosure. As shown in FIG. 2 , the head-mounted device 110 includes a camera unit 112 , a processor 114 and a communication unit 116 . The camera unit 112 is configured for capturing images of the physical environment PE over time. In some embodiments, the camera unit 112 can include one camera component located at the front side of the head-mounted device 110 . In some other embodiments, the camera unit 112 can include a dual camera module and/or a depth camera. The communication unit 116 of the head-mounted device 110 is configured for communicating with other head-mounted devices 120 and 130 . In an embodiment, the communication unit 116 can include a WiFi transceiver, a WiFi-Direct transceiver, a Bluetooth transceiver, a BLE transceiver, a Zigbee transceiver and/or any equivalent wireless communication transceiver. In another embodiment, the communication unit 116 can include a wire connector (e.g., a USB connector, an Ethernet connector) for communication. The processor 114 is coupled to the camera unit 112 and the communication unit 116 . The processor 114 can be a central processing unit, a graphic processing unit and/or a control integrated circuit on the head-mounted device 110 .

Similarly, the head-mounted device 120 includes a camera unit 122 , a processor 124 and a communication unit 126 , and the head-mounted device 130 includes a camera unit 132 , a processor 134 and a communication unit 136 . Each of the head-mounted devices 110 , 120 and 130 includes similar components and these components are configured to perform similar functions. For brevity reasons, the following embodiments will demonstrate details of features on the head-mounted device 110 . The head-mounted device 120 or 130 also includes corresponding details of features respectively.

Reference is further made to FIG. 3 and FIG. 4A . FIG. 3 is a flow diagram illustrating a control method 300 suitable to be utilized on the head-mounted device 110 in FIG. 1 and FIG. 2 . FIG. 4A is a schematic diagram illustrating the head-mounted device 110 in the physical environment PE.

As shown in FIG. 2 , FIG. 3 and FIG. 4A , operation S 310 of the control method 300 is performed to capture images IMG 1 of the physical environment PE over time by the camera unit 112 . In some embodiments, the camera unit 112 is configured to capture images IMG 1 of the physical environment PE periodically (e.g., every 10 microseconds or every 100 microseconds). The images IMG 1 are sequential images to reflect a partial view of the physical environment PE in front of the head-mounted device 110 . The partial view corresponds to a field of view (FOV) of the camera unit 112 . In some embodiments, the FOV of the camera unit 112 is unable to cover the whole scene of the physical environment PE. In some embodiments, images IMG 1 captured by the camera unit 112 are utilized by the processor 114 to construct a partial map of the physical environment PE and locate the head-mounted device 110 .

Operation S 320 is performed by the processor 114 to extract candidate objects from the images IMG 1 captured by the camera unit 112 and also extract object features OF 1 of the candidate objects from the images IMG 1 . As shown in FIG. 4A , the images IMG 1 covers the candidate objects OBJ 1 -OBJ 5 . Therefore, the processor is able to extract the candidate objects OBJ 1 -OBJ 5 and the object features OF 1 of the candidate objects OBJ 1 -OBJ 5 from the images IMG 1 . In some embodiments, the object features OF 1 include at least one of colors, sizes and shapes of the candidate objects OBJ 1 -OBJ 5 . Operation S 330 is performed by the processor 114 to generate a local determination LD 1 about whether each of the candidate objects OBJ 1 -OBJ 5 is fixed or not. An example of the object features OF 1 and the local determination LD 1 generated by the processor 114 is listed in the following Table 1:

The local determination LD 1 can be generated by the processor 114 according to local information (e.g., the images IMG 1 captured by the camera unit 112 and/or the object features OF 1 of the candidate objects OBJ 1 -OBJ 5 ). In an embodiment, the local determination LD 1 is generated by the processor 114 based on an object recognition algorithm. For example, the processor 114 recognizes the item 1 in Table 1 (corresponding to the candidate object OBJ 1 in FIG. 4A ) as a fire hydrant based on the red color and the rectangle shape. The processor 114 may recognize the item 2 (corresponding to the candidate object OBJ 4 in FIG. 4A ) as a table based on the brown color and the shape of a horizontal plane with supporters. Similarly, the processor 114 may further recognize the item 3 (corresponding to the candidate object OBJ 2 ) as a picture frame based on the size and the shape. The processor 114 recognize the item 4 (corresponding to the candidate object OBJ 3 as another picture frame. The processor 114 recognize the item 5 (corresponding to the candidate object OBJ 5 as a vase based on the size and the shape. Based on aforesaid object recognition results, the processor 114 can determine whether the items in Table 1 are fixed or not. In the embodiment, the item 1 , the item 3 and the item 4 (i.e., candidate objects OBJ 1 , OBJ 2 and OBJ 3 shown in FIG.) are determined to be fixed locally by the processor 114 on the head-mounted device 110 . In some cases, the fixed items can be regarded as anchor points in the physical environment PE for locating the head-mounted device 110 . However, the head-mounted device 110 is only capable to observe the partial map of the physical environment PE from one vision point, such that it is hard to ensure the correctness of the local determination LD 1 .

For example, the item 5 corresponding to the candidate object OBJ 5 is a vase, which can be movable over time, such that the candidate object OBJ 5 is not suitable to be regarded as an anchor point for locating function. In this case, the local information LD 1 based on limited information (only observing the candidate object OBJ 5 for 3 seconds from one vision point) may determine the candidate object OBJ 5 to be fixed by mistake.

In addition, once the user move to some other positions or faces toward different directions, the item 1 , the item 3 and the item 4 (e.g., the candidate objects OBJ 1 , OBJ 2 and OBJ 3 ) may not appear in the field of view of the camera unit 112 . The head-mounted device 110 will not able to locate itself until new anchor points are found.

As shown in FIG. 2 and FIG. 3 , operation S 340 is performed by the processor 114 to transmit the object features OF 1 and the local determination LD 1 (i.e., as shown in Table 1) through the communication unit 116 to other head-mounted devices 120 and 130 . Operation S 350 is performed by the processor 114 to receive object features OF 2 and local determination LD 2 from the head-mounted devices 120 and receive object features OF 3 and local determination LD 3 from the head-mounted devices 130 .

Reference is further made to FIG. 4B , which is a schematic diagram illustrating the head-mounted device 120 in the physical environment PE. Similar to operations S 310 -S 330 in aforesaid embodiments, the head-mounted devices 120 is able to capture images IMG 2 as shown in FIG. 4B of the physical environment over time. The images IMG 2 captured by the camera unit 122 of the head-mounted device 120 may be different from the images IMG 1 because the head-mounted device 120 is located at different position from the head-mounted devices 110 .

In this embodiment shown in FIG. 4B , the images IMG 2 covers another partial view of the physical environment PE. The image IMG 2 covers candidate objects OBJ 1 , OBJ 5 , OBJ 6 , OBJ 7 , OBJ 8 and OBJ 9 . Similarly, the processor 124 will extract candidate objects from the images IMG 2 captured by the camera unit 122 , extract object features OF 2 of the candidate objects OBJ 1 and OBJ 5 -OBJ 9 from the images IMG 2 , and generate a local determination LD 2 about whether each of the candidate objects OBJ 1 and OBJ 5 -OBJ 9 is fixed or not. An example of the object features OF 2 and the local determination LD 2 generated by the processor 124 is listed in the following Table 2:

Reference is further made to FIG. 4C , which is a schematic diagram illustrating the head-mounted device 130 in the physical environment PE. In this embodiment shown in FIG. 4C , the images IMG 3 covers another partial view of the physical environment PE. The image IMG 3 covers candidate objects OBJ 1 , OBJ 2 , OBJ 3 , OBJ 5 and OBJ 6 . Similarly, the processor 134 will extract candidate objects from the images IMG 3 captured by the camera unit 132 , extract object features OF 3 of the candidate objects OBJ 1 -OBJ 3 and OBJ 5 -OBJ 6 from the images IMG 3 , and generate a local determination LD 3 about whether each of the candidate objects OBJ 1 -OBJ 3 and OBJ 5 -OBJ 6 is fixed or not. An example of the object features OF 3 and the local determination LD 3 generated by the processor 134 is listed in the following Table 3:

After the head-mounted device 110 receives the object features OF 2 and the local determination LD 2 from the head-mounted devices 120 and the receive object features OF 3 and the local determination LD 3 from the head-mounted devices 130 , operation S 360 is performed by the processor 114 of the head-mounted device 110 to generate an updated determination about whether each of the candidate objects is fixed or not according to the local determination LD 1 and the local determinations LD 2 and LD 3 received from other head-mounted devices 120 and 130 .

Based on aforesaid Table 1, Table 2 and Table 3, the updated determination on the head-mounted devices 110 can be generated by comparing the object features OF 1 in Table 1 and the object features OF 2 and OF 3 in Table 2 and Table 3, so as to map a correspondence relationship between the items in different tables. In an embodiment, the head-mounted devices 110 , 120 , 130 generate Table 1, Table 2 and Table 3 individually, and the items in these tables are not necessary ranked in the same sequence. Therefore, the head-mounted device 110 refers the object features OF 2 and OF 3 in Table 2 and Table 3 to align the items from Table 2 and Table 3 to suitable items in Table 1. For example, the item 6 in Table 2 having a color of red, a size of 40 cm*50 cm and a shape of rectangle matches to the item 1 in Table 1 on the head-mounted device 110 . In this case, the head-mounted device 110 will map the item 6 in Table 2 to the item 1 in Table 1, and combine the local determination LD 2 about the item 6 in Table 2 into a corresponding column in the updated determination on the head-mounted devices 110 . The item 1 in Table 3 also having a color of red, a size of 40 cm*50 cm and a shape of rectangle matches to the item 1 in Table 1 on the head-mounted device 110 . In this case, the head-mounted device 110 will map the item 1 in Table 3 to the item 1 in Table 1, and combine the local determination LD 3 about the item 1 in Table 3 into a corresponding column in the updated determination on the head-mounted devices 110 .

Similarly, each of the items in Table 2 and Table 3 can be mapped toward items in Table 1 by processor 114 . An example of the updated determination on the head-mounted devices 110 is listed in the following Table 4:

In an embodiment, if some items in Table 2 or Table 3 fail to find a match in existed items in Table 1, the head-mounted device 110 will merge the unmatched items from Table 2 and Table 3 into a new data column in Table 4. For example, the item 6 and the item 7 are not original items existed in Table 1.

As shown in Table 4, the item 1 corresponding to the candidate object OBJ 1 is an overlapping object found in Table 1, Table 2 and Table 3, such that the updated determination UD relative to the candidate object OBJ 1 in Table 4 is decided by all of the local determinations LD 1 , LD 2 and LD 3 . The candidate object OBJ 1 is determined to be fixed in two of the local determinations LD 1 and LD 2 , and determined to be not fixed in one local determination LD 3 . In this case, the updated determination UD will decide that the candidate object OBJ 1 is fixed, because there are two head-mounted devices 110 and 120 of the opinion that the candidate object OBJ 1 is fixed against one head-mounted device 130 .

As shown in Table 4, the item 5 corresponding to the candidate object OBJ 5 is an overlapping object found in Table 1, Table 2 and Table 3, such that the updated determination UD relative to the candidate object OBJ 5 in Table 4 is decided by all of the local determinations LD 1 , LD 2 and LD 3 . The candidate object OBJ 5 is determined to be not fixed in two of the local determinations LD 2 and LD 3 , and determined to be fixed in one local determination LD 1 . In this case, the updated determination UD will decide that the candidate object OBJ 1 is not fixed, because there are two head-mounted devices 120 and 130 of the opinion that the candidate object OBJ 1 is not fixed against one head-mounted device 110 .

As shown in Table 4, the item 3 corresponding to the candidate object OBJ 2 is an overlapping object found in Table 1 and Table 3, such that the updated determination UD relative to the candidate object OBJ 2 in Table 4 is decided by the local determinations LD 1 and LD 3 . The candidate object OBJ 2 is determined to be fixed in two of the local determinations LD 1 and LD 3 . In this case, the updated determination UD will decide that the candidate object OBJ 2 is fixed.

As shown in Table 4, the item 2 corresponding to the candidate object OBJ 4 is a non-overlapping object found only in Table 1, such that the updated determination UD relative to the candidate object OBJ 4 in Table 4 is decided by the local determination LD 1 . As shown in Table 4, the item 7 corresponding to the candidate object OBJ 7 is a non-overlapping object found only in Table 2, such that the updated determination UD relative to the candidate object OBJ 7 in Table 4 is decided by the local determination LD 2 .

In aforesaid embodiment, the item 6 and the item 7 in Table 4 corresponding to the candidate object OBJ 6 and OBJ 7 are not in the field of view of the camera unit 112 of the head-mounted device 110 . These item 6 and the item 7 corresponding to the candidate objects OBJ 6 and OBJ 7 are established according to information received from other head-mounted devices. However, the camera unit 112 of the head-mounted device 110 currently has no visions to these objects OBJ 6 and OBJ 7 . As shown in FIG. 4A , the field of view of the camera unit 112 covers the candidate objects OBJ 1 -OBJ 5 . Based on the updated determination UD, the candidate objects OBJ 1 , OBJ 2 and OBJ 3 among the candidate objects OBJ 1 -OBJ 5 are fixed. The candidate objects OBJ 1 -OBJ 3 determined to be fixed in the updated determination UD are regarded as anchor points in the physical environment PE. The processor 114 is further configured to calculate relative distances between the head-mounted device 110 and each of the anchor points (OBJ 1 -OBJ 3 ). Operation S 370 in FIG. 3 are performed by the processor 114 to locate the head-mounted device 110 within the physical environment PE according to the relative distances between the head-mounted device 110 and the fixed objects (OBJ 1 -OBJ 3 ).

Based on aforesaid embodiment, the object features OF 1 -OF 3 observed by different head-mounted devices 110 , 120 and 130 and the local determinations made by different head-mounted devices can be shared among the head-mounted devices 110 , 120 and 130 within the physical environment PE. In this case, each of the head-mounted devices 110 , 120 and 130 can acknowledge the opinion (about whether one object is fixed or not) from other head-mount devices located at different positions. The updated determination can be generated in accordance with all opinions observed from different positions, such that the correctness of the updated determination UD shall be higher than individual one of the local determination LD 1 , LD 2 or LD 3 .

The head-mounted devices 120 and 130 can also perform corresponding operations similar to the embodiment shown in FIG. 3 to benefit from the updated determination UD and locate the head-mounted devices 120 and 130 within the physical environment PE.

Furthermore, the user wearing the head-mounted device 110 may rotating his/her head to different directions or move to different positions, such that the field of view of the head-mounted device 110 may change over time. In this case, if the user wearing the head-mounted device 110 rotating his head to the right side, the head-mounted device 110 will capture the image covering the candidate objects OBJ 6 and OBJ 7 . Based on the updated determination UD shown in Table 4, the head-mounted device 110 may acknowledge that the candidate objects OBJ 6 can be utilized as an anchor point and the candidate object OBJ 7 is not suitable to be an anchor point. The head-mounted device 110 can compare object features of newly captured images with Table 4. Once the object features of newly captured images found the match items (corresponding to the candidate objects OBJ 6 and OBJ 7 ) in Table 4, the head-mounted device 110 is able decide whether the candidate objects OBJ 6 and OBJ 7 are fixed or not immediately. Therefore, the head-mounted device 110 is able to find new anchor point faster after the user rotating his/her head.

On the other hand, the object features OF 1 of the candidate objects regarded as the anchor points in view of the camera unit 112 of the head-mounted device 110 can be transmitted from the head-mounted device 110 to another head-mounted device 120 or 130 . The head-mounted device 120 or 130 is configured to recognize the anchor points in the physical environment PE once the anchor points appeared in view of the camera unit 122 or 132 of the head-mounted device 120 or 130 .

In an embodiment as shown in FIG. 4A , the processor 114 of the head-mounted device 110 is configured to measure a relative distance D 1 between a pair of the candidate objects OBJ 1 and OBJ 2 . It is noticed that the relative distance D 1 measured by the head-mounted device 110 may include a measurement error. For example, if the objects OBJ 1 and OBJ 2 gapped with a real distance of 50 cm in the physical environment PE, the relative distance D 1 measured on the head-mounted device 110 may be 48 cm due to the measurement error. As shown in FIG. 4C , the pair of candidate objects OBJ 1 and OBJ 2 also covered by the images IMG 3 captured by the head-mounted device 130 . A relative distance D 2 between the pair of the candidate objects OBJ 1 and OBJ 2 is measured by the head-mounted device 130 in view of the camera unit 132 . The relative distance D 2 measured by the head-mounted device 130 may also include another measurement error.

In this case, the head-mounted device 110 is able to receive the relative distance D 2 measured in view of the head-mounted device 130 between the pair of the candidate objects OBJ 1 and OBJ 2 . The head-mounted device 110 calculate an average relative distance between the pair of the candidate objects OBJ 1 and OBJ 2 according to the relative distance D 1 and the relative distance D 2 measured by two head-mounted device 130 . For example, if the relative distance D 2 measured on the head-mounted device 130 may be 49 cm due to the measurement error, the average relative distance will be 48.5 cm. For example, if the relative distance D 2 measured on the head-mounted device 130 may be 52 cm due to the measurement error, the average relative distance will be 50 cm. By calculating an average of the relative distances measured among different head-mounted devices, the measurement error may be reduced.

Reference is further made to FIG. 5 , which is a schematic diagram illustrating a head-mounted device 140 newly entered the physical environment PE in an embodiment of the disclosure. The object features OF 1 of the candidate objects OBJ 1 -OBJ 3 regarded as the anchor points are transmitted from the head-mounted device 110 to the head-mounted device 140 newly entered the physical environment PE. In this case, the head-mounted device 140 is configured to recognize the anchor points in the physical environment PE according to the object features OF 1 of the candidate objects OBJ 1 -OBJ 3 regarded as the anchor points. Similarly, the head-mounted device 140 may also receive the object features OF 2 /OF 3 of the candidate objects regarded as the anchor points from the head-mounted devices 120 and 130 . In this case, the head-mounted device 140 newly entered the physical environment PE will recognize valid anchor points easier and faster than observing the physical environment PE by itself.

In aforesaid embodiments of FIG. 1 to FIG. 4 , the object features OF 1 -OF 3 and the local determinations LD 1 -LD 3 are shared among the head-mounted devices 110 - 130 , each of the head-mounted devices is configured to generate the updated determination UD. However, the disclosure is not limited thereto.

Reference is further made to FIG. 6 and FIG. 7 . FIG. 6 is a schematic diagram illustrating a reality system 200 according to another embodiment of the disclosure. FIG. 7 is a functional block diagram illustrating the reality system 200 in FIG. 6 . The reality system 200 in FIG. 6 and FIG. 7 include head-mounted devices 210 , 220 and 230 and a server device 250 . The head-mounted devices 210 , 220 and 230 are located in a physical environment PE. As shown in FIG. 7 , the head-mounted device 210 includes a camera unit 212 , a processor 214 and a communication unit 216 . The head-mounted device 220 includes a camera unit 222 , a processor 224 and a communication unit 226 . The head-mounted device 230 includes a camera unit 232 , a processor 234 and a communication unit 236 . Components of the head-mounted devices 210 - 230 are similar to the head-mounted device 110 - 130 in aforesaid embodiment shown in FIG. 2 . The main difference between the reality system 200 in FIG. 6 and FIG. 7 and the reality system 100 in FIG. 1 and FIG. 2 is that, the reality system 200 further includes the server device 250 . The server device 250 is communicated with the head-mounted devices 210 , 220 and 230 . The server device 250 includes a processor 254 and a communication unit 256 . The communication unit 256 can include a WiFi transceiver, a WiFi-Direct transceiver, a Bluetooth transceiver, a BLE transceiver, a Zigbee transceiver and/or any equivalent wireless communication transceiver. The processor 254 is coupled to the communication unit 256 . The processor 254 can be a central processing unit, a graphic processing unit and/or a control integrated circuit on the server device 250 .

Reference is also made to FIG. 8 , which is a flow diagram illustrating a control method 800 , which is suitable to be utilized on the reality system 200 in FIG. 6 and FIG. 7 .

Each of the head-mounted devices 210 , 220 or 230 is configured to capture images of the physical environment PE by the camera unit 212 , 222 or 232 respectively in operation S 810 shown in FIG. 8 . Each of the head-mounted devices 210 , 220 or 230 is configured to extract candidate objects, extract object features OF 1 , OF 2 or OF 3 of the candidate objects in operation S 820 shown in FIG. 8 . Each of the head-mounted devices 210 , 220 or 230 is configured to generate local determination LD 1 , LD 2 or LD 3 about whether each of the candidate objects is fixed or not in view of each of the head-mounted devices 210 , 220 or 230 as operation S 830 shown in FIG. 8 . Features about capture images of the physical environment PE (S 810 ) and how to extract object features OF 1 , OF 2 or OF 3 (S 820 ) and generate local determinations LD 1 , LD 2 or LD 3 (S 830 ) on the head-mounted devices 210 , 220 or 230 are similar to operations S 310 , S 320 and S 330 disclosed in FIG. 3 and the head-mounted devices 110 , 120 or 130 in previous embodiments shown in FIG. 1 , FIG. 2 and FIG. 4A to FIG. 4C , and not repeated here.

In operation S 840 , each of the head-mounted devices 210 , 220 or 230 is configured to transmit the object features OF 1 , OF 2 or OF 3 and the local determination LD 1 , LD 2 or LD 3 (referring to Table 1, Table 2 and Table 3 in aforesaid embodiments) to the server device 250 .

The server device 250 is communicated with the head-mounted devices 210 - 230 . The server device 250 is configured to collect the object features OF 1 -OF 3 of the candidate objects and the local determinations LD 1 -LD 3 from the head-mounted devices 210 - 230 in operation S 850 . The server device 250 is configured to generate an updated determination UD (referring to Table 4 in aforesaid embodiments) about whether each of the candidate objects is fixed or not according to the local determinations LD 1 -LD 3 collected from the head-mounted devices 210 - 230 . The server device 250 is configured to transmit the updated determination UD (referring to Table 4 in aforesaid embodiments) to each of the head-mounted devices 210 - 230 , such that the head-mounted devices 210 - 230 can locate positions of the head-mounted devices 210 - 230 within the physical environment PE relative to the fixed objects (e.g. anchor points) in the updated determination UD.

In an embodiment, the server device 250 is configured to generate the updated determination by comparing the object features of the candidate objects in views of the head-mounted devices 210 - 230 to map a correspondence relationship between the candidate objects in views of the head-mounted devices 210 - 230 . The server device 250 is configured to generate the updated determination UD corresponding to an overlapped object in view of the head-mounted devices according to the local determinations collected from the head-mounted devices. The server device 250 is configured to generate the updated determination UD corresponding to a non-overlapped object according to one of the local determinations LD 1 -LD 3 .

The object features OF 1 -OF 3 includes at least one of colors, sizes and shapes of the candidate objects.

In an embodiment, the candidate objects determined to be fixed in the updated determination UD are regarded by the server device as anchor points in the physical environment PE. Each of the head-mounted devices 210 - 230 is further configured to calculate relative distances between the head-mounted devices 210 - 230 and each of the anchor points. Based on the relative distances, the processor 214 , 224 or 234 is able to locate the head-mounted device 210 , 220 or 230 within the physical environment PE.

In an embodiment, the object features of the candidate objects regarded as the anchor points are transmitted from the server device 250 to a new head-mounted device (not shown in FIG. 6 and FIG. 7 ) newly entered the physical environment PE. The new head-mounted device is configured to recognize the anchor points in the physical environment PE according to the object features of the candidate objects regarded as the anchor points.

In an embodiment, the object features of the candidate objects regarded as the anchor points are transmitted from the server device 250 to all of the head-mounted devices 210 , 220 and 230 . The head-mounted devices 210 , 220 and 230 are further configured to measure local relative distances between a pair of the candidate objects. The server device 250 is configured to collect all of the local relative distances measured in view of the head-mounted devices 210 - 230 between the pair of the candidate objects. The server device 250 is configured to calculate an average relative distance of all of the local relative distances. Each of the head-mounted devices 210 , 220 and 230 are further configured to obtain local measurement errors by comparing the local relative distances and the average relative distance.

Based on aforesaid embodiment, the object features OF 1 -OF 3 observed by different head-mounted devices 210 , 220 and 230 and the local determinations LD 1 -LD 3 made by different head-mounted devices can be collected by the server device 250 . The server device 250 can generate the updated determination UD in accordance with aforesaid information collected from different head-mounted devices 210 , 220 and 230 . The updated determination UD can be generated in accordance with all opinions observed from different positions, such that the correctness of the updated determination UD shall be higher than individual one of the local determination LD 1 , LD 2 or LD 3 .

Although the present invention has been described in considerable detail with reference to certain embodiments thereof, other embodiments are possible. Therefore, the spirit and scope of the appended claims should not be limited to the description of the embodiments contained herein.

It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the present invention without departing from the scope or spirit of the invention. In view of the foregoing, it is intended that the present invention cover modifications and variations of this invention provided they fall within the scope of the following claims.