Load Balancing and Data Locality Via Fractiling: An Experimental Study

Abstract

In order to fully exploit the power of a parallel computer, an application must be distributed onto processors so that, as much as possible, each has an equal-sized, independent portion of the work. There is a tension between balancing processor loads and maximizing locality, as the dynamic re-assignment of work necessitates access to remote data. Fractiling is a dynamic scheduling scheme that simultaneously balances processor loads and maintains locality by exploiting the self-similarity properties of fractals.

Fractiling accommodates load imbalances caused by predictable phenomena, such as irregular data, and unpredictable phenomena, such as data-access latencies. Probabilistic analysis gives evidence that it should achieve close to optimal load-balance. We have applied fractiling to two applications, an N-body problem and dense matrix multiplication, running on shared-address space and on private-address space parallel machines, namely the Kendall Square KSR1 and the IBM SPI. Although the applications contained little or no algorithmic variance, fractiling improved performance over static scheduling due to systemic variance; however, artifacts of the memory subsystems of the two architectures impeded the scalability of the fractiled code.

Research supported by ARPA/USAF under Grant no. F30602–95-1-0008 and the New York State Science and Technology Foundation. The research was conducted using the resources of the Cornell Theory Center, which receives major funding from the National Science Foundation and New York State; additional funding comes from the Advanced Research Projects Agency, the National Institute of Health, IBM Corporation and other members of the center’s Corporate Research Institute. Susan Hummel was also supported in part by NSF Grant CCR-9321424; Joel Wein by NSF Grant CCR-92l 1494. We thank Bob Walkup for his assistance in programming the SP1.