An efficient saliency driven approach for image manipulation

Abstract

With the increasing availability of low-cost high quality cameras, embedded vision systems, advanced computer vision algorithms, and proliferating solutions based on image/video data, the volume of visual content that is being captured, stored and transmitted is on the rise. Moreover, the image capturing hardware on mobile devices is also improving with a wide range of devices also housing a multiview camera setup. When combined with today’s user experience expectations, this poses a challenge to the editing process from which users expect more efficient results in the most automated possible way. Image editing is a multistage process that spans from the choice of the object or target region in the image for editing to the actual manipulation. We introduce a novel saliency-driven image content ranking approach that allows for automatic selection of objects without the need of training the model. Regions in an image can be selected according to the desired rank. This approach was compared with human behaviour when choosing the most salient object in an image within experiments that involved 2254 participants. The results obtained by the algorithm matched the behaviour of 91% of the human participants. The technique also scored a Fβ measure of 0.84 on the MSRA10k dataset and compares to normal saliency detection models that, unlike this technique, do not rank saliency. We also demonstrate how our saliency ranking model can be combined with segmentation techniques. The combined result of our saliency-driven ranking approach of segmentation masks compared well with the current deep learning state of the art methods that rank segmented objects. Once an object is selected for editing, users expect an efficient way to accurately manipulate images. This fundamental stage is explored in our work where we demonstrate the importance of object inpainting. The main challenge of image inpainting is its objective evaluation and this work presents a new structured approach to objectively evaluate inpainting algorithms. User studies with 2254 participants demonstrated that, on average, users take 3.67s to choose an object for editing in a screen. The combined saliency-driven image manipulation framework takes advantage of this physical limitation and efficiently pipelines processes to deliver an accurate and efficient result in image manipulation tasks such as attention re-targetting. A multi-purpose dataset was designed and built to serve all these functions and is also presented in this work.