An Optical Design for Interaction With Mid-Air Images Using the Shape of Real Objects

Mid-air images, which are augmented reality (AR) technologies, enable computer graphics (CG) images to be superimposed on a physical space. The mid-air image can be placed side-by-side with real objects, allowing various interactions, such as directly manipulating them to contact the mid-air image on the same plane. In this case, the measurement of the shape of real objects is necessary to realize geometric consistency between the mid-air image and real objects. However, in mid-air image optics, real objects cannot be placed behind the mid-air image (i.e., at a position where they interrupt the light rays that form the mid-air image). This limits the placement of the sensor and may prevent accurate measurement of the shape of the real objects. Consequently, we proposed an optical system for interaction with mid-air images that virtually measures the shape of real objects from behind the mid-air image. In our system, a virtual infrared (IR) sensor is formed behind the mid-air image using a hot mirror that reflects only IR light. The optical system considers the visible area of the mid-air image and the measurable area of the sensor. We evaluated the sharpness, luminance, and chromaticity to assess whether the hot mirror had changed the appearance of the mid-air image. The results confirmed that there was little impact on user perception. Furthermore, we developed four supporting applications for our system to show its efficacy.