An efficient distributed file system

Abstract

Today's corporations and government agencies are becoming more and more data-driven. Company's data can be made up of terabytes or even petabytes of files. Nowadays it is becoming more popular for companies to store their data in cloud storages, which can be located hundreds of miles away from the actual company's headquarters. Modifications of large data files, such as videos, over a normal Internet connection can be quite slow due to the design of the distributed file system being used. In this dissertation we first analyse the behaviour of other distributed file systems together with their week points and then present a system that focuses on distributing data amongst the nodes in an efficient way using commodity hardware and a basic Internet connection. The efficiency was achieved by adding cache on the client-side together with designing custom-made algorithms which try to transmit the least amount of data between the connected nodes. These algorithms try to exploit the similarities between versions of the same files and thus reduce the bandwidth consumption. Special data-structures were also designed so that files updating on the server-side is done as quickly as possible. An artefact was implemented using ANSI C on the Linux kernel. The client-side file system was programmed in user-space using a special library named FUSE while the communication between the nodes was performed through a network library named Message Passing Interface. The file system was than compared with an implementation of NFSv3 where, on certain aspects, it had a better performance.