iNotifyWeb : an adaptive web page classification, monitoring, and change notification system

Abstract

We live in the Information Age. The Internet gives users access to any information that they may be interested in whenever they want it. Often, users need to revisit web pages to check if they have been updated, and if these web pages are updated frequently, then users may need to revisit them frequently. However, although some people want to stay abreast of breaking news, it may cause a lot of distraction and waste of time to visit one or more web pages to see if there has been an interesting change. We have developed a prototype system to reduce the cognitive information overload problem (Kirsh, 2000), which results in a lot of unnecessary stress on internet users. The prototype allows the users to add web pages, or parts of web pages (referred to as blocks), for automatic monitoring. The blocks are categorised into categories (also being referred to as bins). Each bin contains related blocks. The user is then notified of changes. He may choose to be recommended either all the changes or the interesting changes only. These changes are presented to the user in the form of a virtual portal. Each bin in which some changes were encountered has a corresponding virtual page, containing two lists of links these being the recommended links and the other links. The system provides an automatic way of extracting the most important sections of a web page. Other methods that allow the addition of blocks that were not extracted by the system are also provided. This automatic extraction of important content is done by comparing two pages in the same domain. Here the next page is used for comparison with the first one. Sections that are common between the two pages are eliminated, leaving behind the important content from the first page. In order to automatically classify blocks into bins, the vector space model is used. Here each bin is represented by a centroid vector. When new blocks are to be classified, a representation vector is created. Cosine similarity measure is used in order to judge how similar the new vector is to any existent centroid vector. The user has the final say in deciding in which bin to actually place the new block. A Bayesian classifier is created for each bin. This classifier is supervised into learning the user's interests. This is used in order to classify the new links into one of two classes these being interesting or not interesting. The learning is done with the least user interruption as possible. During the evaluation, we managed to prove our hypothesis. Our aims and objectives were met. iNotifyWeb is able to help the users deal with the cognitive information overload problem. It is able to successfully extract the important content from a web page characterised by links. The bin classification feature also performs well. The Naive Bayesian classifier is able to learn user's interests. As expected, the users prefer to use a standalone Java application over a Mozilla Firefox extension.