Application US20190360810

VIDEO SENSOR FUSION AND MODEL BASED VIRTUAL AND AUGMENTED REALITY SYSTEMS AND METHODS
[+] Show Details

US20190360810

1. An apparatus comprising: (8) (1)

a logic device configured to communicate with a navigational sensor, at least one orientation and/or position sensor (OPS), and an imaging module coupled to a mobile structure, wherein the navigational sensor is configured to provide navigational data associated with the mobile structure, and wherein the logic device is configured to: receive at least one image from the imaging module;

receive orientation, position, and/or navigation data corresponding to the at least one image from the OPS and/or the navigational sensor; and

generate an information display view comprising at least one augmented image corresponding to the received at least one image based, at least in part, on the received orientation, position, and/or navigation data and/or the received at least one image.

2. The apparatus of claim 1, wherein the generating the at least one augmented image comprises: (0) (3)

identifying a pixel position and a pixel area of an object detected in the at least one image from the imaging module;

determining an estimated range to the object, an estimated velocity of the object, and/or an estimated relative position of the object based, at least in part, on the pixel position, pixel area, and a position of the imaging module; and

rendering a graphical indicator and/or information graphic overlaying the at least one image from the imaging module configured to indicate the estimated range to the object, the estimated velocity of the object, and/or the estimated relative position of the object, wherein the graphical indicator and/or information graphic are rendered to have a perspective roughly following a surface of a medium surrounding the mobile structure.

3. The apparatus of claim 1, wherein the generating the at least one augmented image comprises generating at least one horizon-stabilized image by: (1) (4)

determining a slant and elevation of the imaging module corresponding to the at least one received image based, at least in part, on the received orientation, position, and/or navigation data and/or the at least one received image;

rotating pixels of the at least one received image to at least partially compensate for the determined slant;

translating the pixels of the at least one received image to at least partially compensate for the determined elevation; and

cropping the rotated and translated at least one received image symmetrically about a reference slant and/or a reference elevation to compensate for an expected extent of motion of the mobile structure.

4. The apparatus of claim 3, wherein the generating the at least one horizon-stabilized image comprises: (0) (3)

5. The apparatus of claim 1, wherein the logic device is configured to generate a synthetic viewpoint elevated image corresponding to the at least one augmented image by: (0) (5)

identifying object pixels and a water contact reference pixel position corresponding to a detected object in the at least one augmented image;

removing the identified object pixels from the at least one augmented image;

backfilling the unpopulated pixel locations corresponding to the removed object pixels;

generating a vertically stretched image from the at least one augmented image; and

copying the identified object pixels into the vertically stretched image according to the water contact reference pixel position to generate the synthetic viewpoint elevated image.

6. The apparatus of claim 1, wherein the logic device is configured to synthetically remove foreground obstructions in the at least one received image, before generating the at least one augmented image, by: (0) (3)

7. The apparatus of claim 1, wherein the at least one augmented image comprises a plurality of augmented images, and wherein the logic device is configured to: (0) (3)

identify a first pixel position and a first pixel area of an object detected in a first one of the plurality of augmented images;

identify a second pixel position and a second pixel area of the object in a second one of the plurality of augmented images, wherein a first position of the imaging module corresponding to the first augmented image is different from a second position of the imaging module corresponding to the second augmented image; and

determine an estimated range to the object, an estimated velocity of the object, and/or an estimated relative position of the object based, at least in part, on the first and second pixel positions, pixel areas, and positions of the imaging module.

8. The apparatus of claim 1, further comprising the imaging module and the OPS, wherein the imaging module is coupled to the mobile structure and the OPS is coupled to the mobile structure or within the imaging module, wherein the at least one augmented image comprises a plurality of augmented images, and wherein the logic device is configured to: (0) (3)

identify a first plurality of pixel positions and pixel areas corresponding to a plurality of features of an object detected in a first one of the plurality of augmented images;

identify a second plurality of pixel positions and pixel areas corresponding to the plurality of features of the object in a second one of the plurality of augmented images, wherein a first position of the imaging module corresponding to the first augmented image is different from a second position of the imaging module corresponding to the second augmented image; and

generate a three dimensional point cloud model of the object based, at least in part, on the first and second pluralities of pixel positions and pixel areas and the first and second positions of the imaging module.

9. The apparatus of claim 1, further comprising the imaging module and the OPS, wherein the imaging module is coupled to the mobile structure by an active or passive mechanically horizon-stabilized mount, wherein the mobile structure comprises a watercraft, and wherein the logic device is configured to: (0) (2)

10. The apparatus of claim 1, wherein the at least one augmented image comprises a plurality of augmented images, and wherein the logic device is configured to: (0) (5)

detect one or more objects in the plurality of augmented images and/or the received navigational data;

determine one or more object characteristics of each of the detected objects based, at least in part, on the plurality of augmented images and/or the received navigational data;

segregate the detected objects into predetermined types based, at least in part, on the determined object characteristics;

identify one or more of the objects based, at least in part, on a navigational database; and

generate a synthetic Automatic Identification System (AIS) signature for at least one unidentified object, wherein the synthetic AIS signature comprises at least one object characteristic corresponding to the at least one unidentified object.

11. A method comprising: (8) (3)

receiving at least one image from an imaging module coupled to a mobile structure;

receiving orientation, position, and/or navigation data corresponding to the at least one image from an orientation and/or position sensor (OPS) and/or a plurality of navigational sensors, wherein each navigational sensor is configured to provide navigational data associated with the mobile structure; and

generating an information display view comprising at least one augmented image corresponding to the received at least one image based, at least in part, on the received orientation, position, and/or navigation data and/or the received at least one image.

12. The method of claim 11, wherein the generating the at least one augmented image comprises: (0) (3)

identifying a pixel position and a pixel area of an object detected in the at least one image from the imaging module;

13. The method of claim 11, wherein the generating the at least one augmented image comprises generating at least one horizon-stabilized image by: (1) (4)

rotating pixels of the at least one received image to at least partially compensate for the determined slant;

translating the pixels of the at least one received image to at least partially compensate for the determined elevation; and

14. The method of claim 13, wherein the generating the at least one horizon-stabilized image comprises: (0) (3)

15. The method of claim 11, further comprising generating a synthetic viewpoint elevated image corresponding to the at least one augmented image by: (0) (5)

identifying object pixels and a water contact reference pixel position corresponding to a detected object in the at least one augmented image;

removing the identified object pixels from the at least one augmented image;

backfilling the unpopulated pixel locations corresponding to the removed object pixels;

generating a vertically stretched image from the at least one augmented image; and

copying the identified object pixels into the vertically stretched image according to the water contact reference pixel position to generate the synthetic viewpoint elevated image.

16. The method of claim 11, further comprising synthetically removing foreground obstructions in the at least one received image, before generating the at least one augmented image, by: (0) (3)

17. The method of claim 11, wherein the imaging module is coupled to the mobile structure and the OPS is coupled to the mobile structure or within the imaging module, wherein the at least one augmented image comprises a plurality of augmented images, the method further comprising: (0) (3)

identifying a first pixel position and a first pixel area of an object detected in a first one of the plurality of augmented images;

identifying a second pixel position and a second pixel area of the object in a second one of the plurality of augmented images, wherein a first position of the imaging module corresponding to the first augmented image is different from a second position of the imaging module corresponding to the second augmented image; and

determining an estimated range to the object, an estimated velocity of the object, and/or an estimated relative position of the object based, at least in part, on the first and second pixel positions, pixel areas, and positions of the imaging module.

18. The method of claim 11, wherein the at least one augmented image comprises a plurality of augmented images, the method further comprising: (0) (3)

identifying a first plurality of pixel positions and pixel areas corresponding to a plurality of features of an object detected in a first one of the plurality of augmented images;

identifying a second plurality of pixel positions and pixel areas corresponding to the plurality of features of the object in a second one of the plurality of augmented images, wherein a first position of the imaging module corresponding to the first augmented image is different from a second position of the imaging module corresponding to the second augmented image; and

generating a three dimensional point cloud model of the object based, at least in part, on the first and second pluralities of pixel positions and pixel areas and the first and second positions of the imaging module.

19. The method of claim 11, wherein the imaging module is coupled to the mobile structure by an active or passive mechanically horizon-stabilized mount, wherein the mobile structure comprises a watercraft, the method further comprising: (0) (2)

20. The method of claim 11, wherein the at least one augmented image comprises a plurality of augmented images, the method further comprising: (0) (5)

detecting one or more objects in the plurality of augmented images and/or the received navigational data;

determining one or more object characteristics of each of the detected objects based, at least in part, on the plurality of augmented images and/or the received navigational data;

segregating the detected objects into predetermined types based, at least in part, on the determined object characteristics;

identifying one or more of the objects based, at least in part, on a navigational database; and

generating a synthetic Automatic Identification System (AIS) signature for at least one unidentified object, wherein the synthetic AIS signature comprises at least one object characteristic corresponding to the at least one unidentified object.

View Abstract and Specification Size

PDF with Images and Document Face >

Full Text Publication >

Application US20190360810

VIDEO SENSOR FUSION AND MODEL BASED VIRTUAL AND AUGMENTED REALITY SYSTEMS AND METHODS

Patent Matrix® Search