Low-latency inter-thread communication over gigabit ethernet

Abstract

The increase in network capacity and computing power for commodity platforms has enabled the development of Networks of Workstations which have become affordable alternatives to dedicated parallel computers. The major challenge in using clusters of commodity platforms effectively is that commodity operating systems using conventional layered communication protocols such as TCP /IP were not originally designed for high performance applications. With the arrival of Gigabit and higher throughput networks, bandwidth degradation and lack of CPU resources available for user applications has become a common problem. Communication bottlenecks are mostly due to overheads imposed by interrupt generation, in-memory copying and protocol processing together with latencies induced by inefficient use of buses and I/O devices. In order for distributed computing to be feasible we must have an efficient communication architecture through which threads running on independent workstations can communicate. In this dissertation we tackle the issue of efficiency in intra-cluster communications based on Gigabit Ethernet. We identify the major bottlenecks in existing communication protocols and present a high-performance communication architecture that aims to provide high throughput, low-latency and low-overhead communication. Our approach is focused towards the design of more efficient systems software that can better exploit commodity hardware resources. We propose, implement and analyse a communication system that operates completely at the user level without any operating system interaction and is designed for integration with a user-level thread scheduler. We provide user-level multi-threaded applications with a zero-copy messaging system based on a CSP channel approach and reduce CPU overheads by replacing interrupts with local polling. Our proposed library offers a range of protocols each customized for different communication requirements whilst still providing transparency to the user application. Using our small message protocols for 1500-byte Ethernet payloads we achieved a throughput of 753Mbps with 38% CPU load on the sender and 43% CPU load on the receiving host. Using our large message protocol on the other hand we were able to increase the payload throughput to 941Mbps with a substantial reduction in CPU load. Thus with our architecture we provide user applications with substantial throughput increase over traditional kernel-based communication whilst retaining ample CPU resources that can be used for computation.