Investigating neural style transfer techniques for chiaroscuro paintings

Abstract

In recent years, the capabilities and availability of Neural Style Transfer (NST) techniques have increased considerably. NST is an optimisation technique that applies the style from a target style image to a content image such that the content image appears to be painted in the style of the style image. Even with the advancements in this field, there is still a limit on the styles considered. One of the styles which has been neglected is chiaroscuro. Additionally, there is a lack of consistency in the evaluation of such techniques due to the nature of their outputs. In this dissertation, we compile a dataset which contains 641 chiaroscuro paintings of varying subjects and from it, we take a sub-set with 312 chiaroscuro paintings which are portraits. We manually select each opensource image within the dataset with care, with the intention of making the dataset publicly available. We use these datasets to train GAN-based NST models to perform style transfer of the artistic style chiaroscuro. We also adopt a CNN-based technique and incorporate post processing after the stylisation process to enhance the contrasts within the image. To develop the post processing we experiment with different image processing methods. This includes the generation of a mask to separate the background from the foreground, blurring filters, and brightening filters. Moreover, the evaluation of NST techniques is an ongoing issue within the field since art is inherently subjective. Therefore, we propose a robust evaluation to determine the effectiveness of the adopted techniques and compare them to state-of-the-art NST algorithms. The evaluation proposed is an amalgamation of methods selected after an extensive review of evaluations used in various NST research studies. We improve upon their approaches by making use of a style detection system which is able to predict 25 artistic styles with a 62% accuracy. In addition, we propose to compare the saliency maps of the content and stylised images using a quantitative technique as opposed to aesthetic judgement. Lastly, we perform an aesthetic evaluation through a questionnaire which we created specifically for this dissertation. The results obtained through this evaluation show that the techniques implemented are promising. The GAN-based technique outperformed the other methods with regards to retaining the integrity of the content images. This was confirmed through a comparison of saliency ranking results as well as a questionnaire with 45 participants. Additionally, this technique, along with the NST algorithm with post processing, attained the highest scores in terms of overall visual quality from the same questionnaire.