Virtual touch screen

A virtual touch screen (VTS) is a user interface system that augments virtual objects into reality either through a projector or optical display using sensors to track a person's interaction with the object. For instance, using a display and a rear projector system a person could create images that look three-dimensional and appear to float in midair. Some systems utilize an optical head-mounted display to augment the virtual objects onto the transparent display utilizing sensors to determine visual and physical interactions with the virtual objects projected.

History

 * 2010, Displair introduced its first commercial "screenless" 3D virtual multi-touch display, consisting of cold stable air flow containing particles of water produced by a cavitation method.
 * Late 2010, Microsoft released the Kinect, a first of a kind commercially available and affordable virtual touch screen sensor that could track a person's movement and augment their interactions with objects rendered in a television set.
 * 2015, Microsoft unveiled the Hololens, an optical head-mounted display utilizing the Windows Holographic platform. Virtual objects are displayed through the optical head-mounted to render the Windows 10 system onto the visible environment of the user.

Development
During a 2009 study at the Korea Electronics Technology Institute, a virtual touch screen system was built using the touch point tracking technique based on a time of flight camera with a Kalman filter. To address the widespread problem in conventional virtual touch screen systems (i.e. responding sensitively to even the slightest hand movement and failing to recognize the touch point accurately), the study had applied the Kalman filter tracking technique to predict and track the touch point consecutively. Being forecast and recognized, the predicted touch point was bound to exist all the time, but an abrupt movement of touch point was likely to be recognized only insensitively. A commercial version based on this technology was never implemented.

Usage
Currently, virtual touch screens (VTS) are used primarily for entertainment and research purposes. A popular virtual touch screen system is the Kinect system made commercially available by Microsoft. The Kinect system is primarily used for consumer entertainment such as playing games and using the menus found on the Xbox system. However, in recent years the Kinect has been used for research projects in the field of robotics and 3-D interactions outside of the field of games such as browsing the web, and medical practices.

There is a potential for the technology where the 3D visualization and physical interaction with the objects is necessary, such as teaching students introductory calculus.

Virtual touch screen technology can also be applied to turn non-touch displays into interactive, touch-capable surfaces using a Kinect sensor in combination with a projector. This type of application allows virtual touch technology to enhance personal display usage as well as better communication and presentation of data to be used in education or business settings. In medical practices, virtual touch interaction has been applied to allow users to view and manipulate digital data such as 3D images used in medical scans, allowing medical staff to interact with the data without any physical contact, thus avoiding the need for re-sterilization. Additionally, Microsoft's Kinect sensor has also employed the use of virtual touch screen technology to help stroke patients recover and improve motor function in their limbs through a game-like system involving interpretation of gestures combined with adjustable difficulty levels based on the user's performance.

Human factors and ergonomics
One of the key issues with virtual environments that utilize touch are the haptics. For example, utilizing virtual touch screen for medical training could cut costs down for training medical staff for touch delicate operations such as surgery. However, the haptics of such training does not provide sufficient physical touch feedback necessary for a virtually trained practitioner to perform a real surgery. Incorporating sufficient haptic feedback to virtual touch screen interactions is an active area of research.