Efficient distributed execution of computationally intensive tasks

Abstract

Recent advances in computer and network technology have made distributed systems appropriate for the provision of high quality services in many application areas, especially when execution time is the critical factor. Many modern problems are extremely computationally intensive, leading to an unavoidable compromise to the quality of the generated results in order to best fit time and budgetary restrictions. This paper reports on the implementation of a distributed architecture, suitable for the efficient execution of computationally intensive parallel jobs that may require/generate large volume of data. A cluster of networked computing nodes as a physical layer to the distributed architecture is considered. We present a job model for the submitted tasks that is derived from real-life application areas. Overlapping techniques are proposed as a method of reducing network delays. We prove that the proposed distribution of work over the nodes of the execution network is optimal, in terms of minimizing the execution time, with respect to the availability of resources. Certain configuration and implementation details are discussed, mainly in the job management and submission and delivery manager layers. We conclude by presenting our experience of testing the environment for computing-intensive 3D rendering jobs derived from the ESPRIT project EROPPA.

Work partially supported by the EC ESPRIT 4th Framework Program on HPCN, R&D action 22503, EROPPA (Experiments on Remote Operations for Post Production Applications).