Inter-modal recognition of sketches

Abstract

Automatic face recognisers have made a lot progress these past few years. An important application of face recognisers is in law enforcement agencies, where automatic retrieval of photos of suspects can narrow down potential criminals quickly. In the case that a face photo is not available, investigators must rely on a face sketch which is produced based on an eyewitness’s description. The scope of this final year project is to implement an automatic face recogniser that is able to retrieve a photo based on the sketch input. Sketches and photos are inherently different, so in order to tackle this problem, an inter-modal approach is introduced. An inter-modal approach to sketch retrieval is done by taking common features from the sketch and the photo itself, without changing the modality of the images, and using this information as a basis for retrieval. Testing was carried out using the Chinese University of Hong Kong (CUHK) student database, which contains 188 photo-sketch pairs. This implementation makes use of an Active Orientation Model (AOM) to detect 68 predefined points on a query sketch and the photo dataset, and the Euclidean distance between the sketch and each photo in the gallery is calculated. At rank-100, this method achieved a recognition rate of 55.85%. To improve these results, Local Binary Patterns (LBP) were then introduced to extract features of the query sketch and each photo in the dataset. The distance between the sketch’s features and each photo’s features was obtained, and were then fused with the previously calculated point distances. Giving a higher weighting to the LBP histogram distances resulted in an increased recognition rate of 60.11% at rank-100.