Automated web site blacklisting

Abstract

Attacks on web browsers have increased considerably both in their popularity and sophistication. These threats come either from malicious or infected benign sites. At the perimeter level, these threats can be tackled by existing solutions such as manually configuring a URL blacklist or anti-malware solutions such as anti-virus. These solutions are limited ns they cannot detect malicious sites which are not listed in a blacklist or signature database. We aim to produce an enhanced secure web gateway providing perimeter-based protection against malicious sites through automatic configuration of the blacklists. To automate the blacklists, a solution is required that classifies previously unseen web pages as benign or malicious. After analysing related work, two solutions were considered. The first solution uses machine learning techniques and a classifier that is trained using both benign and malicious samples, otherwise known as a fully-supervised approach. The second solution is based on anomaly detection techniques which requires training on benign samples only, in this case a semisupervised approach. Both of these solutions make use of a number of features which can discriminate between malicious and benign pages. The chosen features are extremely important as they have an impact on the number of false positives and the detection rate. Evaluation results revealed the advantages and disadvantages of these two solutions, and their respective false positives and detection rates. The classifier-based fully-supervised approach achieved a better detection rate and is easily scalable. On the other hand, the anomaly-based detection semi-supervised approach requires less work for the initial setup, but requires fine-tuning to achieve acceptable detection rates.