Seminar

Abstract

As VR (Virtual Reality) and AR (Augmented Reality) have developed for over several decades, much research has been investigated on the technology. However, till now, according to literature reviews, the number of VR and AR HMD (Head Mount Device) only accounts for less 1% of the number of cellphones. Most VR or AR systems are applied to industrial or research fields, while there is lack in the consumer VR applications. Unnatural interaction, high latency and cybersickness undermine VR or AR user experience. Our research aims are to solve the conflicts caused by applying VR or AR to 3D free-hand sketching, transparent specular object reconstruction.

3D free-hand sketching in VE (virtual environment) enables users to create 3D virtual objects intuitively in an immersive environment. However, 3D free-hand drawing often leads to inaccurate sketches. We present HandPainter, a VR sketching system which focuses more on the direct use of two hands for 3D sketching without additional user interface, which allows the use of one hand as a flexible canvas for drawing and the other hand as a pen. The design is based on a pair of VR gloves that provide precise hand tracking and gesture capture. We devise a set of intuitive gestures, which assist 3D sketching with supporting operations such as canvas panning and drawing positioning. We demonstrate the effectiveness of HandPainter by showing free-hand sketching showcases and a comparison to Tilt Brush via a user study.

Aimed at 3D drawing in traditional systems, previous works tend to exploit depth information to overlay virtual objects to appropriate planes. However, under this design situation, the semantic relationship between virtual objects and physical environment is not taken into consideration. We take advantage of a joint model of CNN and LSTM to interpret the semantic relationship between virtual objects and real worlds in order to offer the smart recommendation of placement position. As for transparent specular object reconstruction, we would extend current reconstruction algorithms to reconstruct 3D shapes about transparent, specular objects after collecting sparse point cloud with VR or AR interface.

Overall, we exploit and extend previous modelling, reconstruction algorithms integrated with machine learning technique aimed at our tasks, so as to reduce unintuitive interaction, unnatural placement and inaccurate free-hand sketching results brought by VR, AR applications in sketching, reconstruction, placement fields. Looking forward, the conflict caused by applying VR, AR techniques to traditional fields would be alleviated much with the assistance the further development. Further applications of integrating VR, AR techniques with different types of fields would be blooming more and more.