Cloud-based rendering of natural video on mobile devices

Abstract

The complexity of applications that render natural video in real-time requires platforms that offers the adequate computational power. One such application is a Free-Viewpoint application, where a set of three-channel camera views and depth maps are used to generate dynamic textured surfaces for user consumption. The bandwidth of the communications channel and the resources on mobile device hardware contrast with the real-time characteristics attributed to Free-Viewpoint applications. These limitations make it unfeasible to perform this kind of video processing on mobile hardware. As a solution to alleviate these limitations, this project proposes a Cloud-based solution as an alternate source of computational power. The Cloud provides a network connected pool of processing resources on demand. The fact that these resources are user location independent adheres to the characteristics that mobile devices desire. To port the Free-Viewpoint application to the Cloud, a distributed environment is implemented that practices optimisation techniques which are otherwise unfeasible to perform on a mobile device. This distributed environment comprises of a Load Balancer tasked with coordinating the components of a Free-Viewpoint application and a set of Worker Machines capable of generating dynamic textures of any dimensions. A tiling mechanism is presented as a means of dividing and distributing the workload required to render a single frame of the generated video stream. Multithreading makes it possible to alleviate all the bottlenecks present in the serial implementation of the Free-Viewpoint system as well as achieve an initial speedup up in the performance of Worker Machines. By distributing the computational workload amongst 18 Worker Machines, the implemented system managed to achieve a 14.5 times speedup over the serial execution for a 1024x768 resolution video. Moreover, by applying further multithreading on the Worker Machines, the system using the same parameters, managed to achieve a 21.2 times speedup. Image quality tests show that both the serial implementation and the Cloud-based system maintain an average of 37.6 dB Peak-Signal-to-Noise Ratio when compared to the reference camera sources. These tests show that the tiling mechanism implemented in the distributed environment preserves the image rendering quality of the serial implementation.