Automatic recognition and translation of street signs from mobile phone images

Abstract

Mobile phone capabilities are becoming increasingly powerful enabling more demanding applications to be executed. They are also capable of accessing the Internet at higher speeds and sophisticated digital cameras are now being integrated. This dissertation demonstrates an implementation of a mobile application, which exploits the mentioned capabilities in order to extract and translate text from street signs. Street signs often have different illustrations, structures, font types, font sizes, symbols and colours. Furthermore, photos of street signs are often taken in an uncontrolled environment, thus problems such as different levels of luminance may occur. This dissertation essentially involves analyses and discussions tackling the above mentioned problems. As a result, an application was developed with the intention of extracting text from photos depicting street signs by overcoming the above mentioned problems. This process is divided into two main stages: character detection and recognition. The captured images will be transmitted to a server where image processing will take place. Text present on the street signs will be extracted and then translated into the target language which is then sent back to the mobile device. The key image processing stages include edge information extraction, character detection based on spatial cohesion and character recognition based on characters' semantic features. The latter uses a classifier together with a template matching technique to extract the respective ASCII values. Scene images pose new challenges in image processing. In order to demonstrate the effectiveness of the implemented solution, evaluation is done by using photos of street signs with complex and different backdrops.