Computer-vision based telemanipulation of an interactive kinetic surface

Abstract

One can have a conversation with someone remotely through video cameras and microphones in real-time or edit the same document in real-time remotely, yet the discussion becomes an arduous task when the requirement is to represent, discuss and even manipulate physical objects, spaces and surfaces. An interactive kinetic surface would bridge this gap between telecollaboration across different locations. The aim of this project is to design, implement and manipulate an interactive kinetic surface. This was achieved through familiarisation with previously implemented kinetic surfaces, examination of the required parameters and current limitations. A six by eight interactive kinetic surface was implemented with a 5cm depth resolution for each actuator. Furthermore, the chief manipulation method was achieved through the use of hands and as a result, extensive research was conducted on the acquisition of depth data. This was effected using two methods: through the use of a depth sensing device and by means of a neural network that extracts depth information from monocular RGB images alone. Both avenues presented the resulting depth data differently, the depth sensing device, through a depth image, whereby intensity values represent a change in depth and the neural network provided 21 datapoints for each of the three dimensions. Algorithms to extract the required data were written and this data was sent to the interactive kinetic surface, for rendering. From the results obtained, it was confirmed that the aims and objectives of this project were met and exceeded requirements. The interactive kinetic surface was successfully constructed and, although some inherent limitations are still present, the results are adequate. Hand poses are successfully rendered on the surface, with less than 7% error in all cases. The neural network results show comparable quantitative errors to the depth sensing device method, with the additional advantage of independency from specialised hardware.