Skip to main content
SpringerLink
Log in

Multidimensional array I/O in Panda 1.0

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Large multidimensional arrays are a common data type in high-performance scientific applications. Without special techniques for handling input and output, I/O can easily become a large fraction of execution time for applications using these arrays, especially on parallel platforms. Our research seeks to provide scientific programmers with simpler and more abstract interfaces for accessing persistent multidimensional arrays, and to produce advanced I/O libraries supporting more efficient layout alternatives for these arrays on disk and in main memory. We have created the Panda (Persistence AND Arrays) I/O library as a result of developing interfaces and libraries for applications in computational fluid dynamics in the areas of checkpoint, restart, and time-step output data. In the applications we have studied, we find that a simple, abstract interface can be used to insulate programmers from physical storage implementation details, while providing improved I/O performance at the same time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. J. L. Bell and G. S. Patterson Jr. Data organization in large numerical computations. The Journal of Supercomputing, 1(1):105–136, 1987.

    Google Scholar 

  2. R. Bordawekar, A. Choudhary, K. Kennedy, C. Koelbel, and M. Paleczny. A model and compilation strategy for out-of-core data parallel programs. Technical Report CRPC-TR94507-S, Center for Research on Parallel Computation, December 1994.

  3. R. Bordawekar, J. M. del Rosario, and A. Choudhary. Design and evaluation of primitives for parallel I/O. In Proceedings of Supercomputing '93, pages 452–461, Portland, Ore., November 1993.

  4. P. Brezany, M. Gernt, P. Mehrotra, and H. Zima. Concurrent file operations in a High Performance Fortran. In Proceedings of Supercomputing '92, pages 230–237, November 1992.

  5. P. Brezany, T. A. Mueck, and E. Schikuta. Language, compiler and parallel database support for I/O intensive applications. In Proceedings of the High Performance Computing and Networking 1995 Europe. Springer-Verlag, May 1995.

  6. P. Corbett, D. Feitelson, Y. Hsu, J.-P. Prost, M. Snir, S. Fineberg, B. Nitzberg, B. Traversat, and P. Wong. MPI-IO: A parallel file I/O interface for MPI. Technical Report NAS-95-002, NASA Ames Research Center, January 1995. Version 0.3.

  7. P. F. Corbett and D. G. Feitelson. Design and implementation of the Vesta parallel file system. In Proceedings of the Scalable High-Performance Computing Conference, pages 63–70, 1994.

  8. J. M. del Rosario, M. Harry, and A. Choudhary. The design of VIP-FS: A virtual, parallel file system for high performance parallel and distributed computing. Technical Report SCCS-628, Northeast Parallel Architectures Center, Syracuse University, May 1994.

  9. D. DeWitt, N. Kabra, J. Luo, J. M. Patel, and J.-B. Yu. Client-server Paradise. In Proceedings of the 20th VLDB Conference, Santiago, Chile, 1994.

  10. High Performance Fortran Forum. High Performance Fortran language specification, version 1.0. Technical Report CRPC-TR92225, Rice University, January 1993.

  11. G. Fox, S. Hiranandani, K. Kennedy, C. Koelbel, U. Kremer, C. Tseng, and M. Wu. Fortran D language specification. Technical Report TR 90-141, Dept. of Computer Science, Rice University, December 1990.

  12. N. Galbreath, W. Gropp, and D. Levine. Applications-driven parallel I/O. In Proceedings of Supercomputing '93, pages 462–471, Portland, Ore., November 1993.

  13. S. Hiranandani, K. Kennedy, and C. Tseng. Compiling Fortran D for MIMD distributed-memory machines. Communications of the ACM, 35(8): August 1992.

  14. J. V. Huber Jr., C. L. Elford, D. A. Reed, A. A. Chien, and D. S. Blumenthal. PPFS: A high performance portable parallel file system. In Proceedings of the International Conference on Supercomputing, June 1995.

  15. Interbase Software Corporation. Interbase Data Definition Guide. Boston, 1990.

  16. D. Kotz. Disk-directed I/O for MIMD multiprocessors. Technical Report PCS-TR94-226, Dept. of Computer Science, Dartmouth College, July 1994. Revised November 8, 1994.

  17. D. Kotz. Disk-directed I/O for an out-of-core computation. Technical Report PCS-TR95-251, Dept. of Computer Science, Dartmouth College, January 1995.

  18. D. Kotz. Expanding the potential for disk-directed I/O. Technical Report PCS-TR95-254, Dept. of Computer Science, Dartmouth College, March 1995.

  19. D. A. Lane. Visualization of time-dependent flow fields. In Proceedings of IEEE Visualization '93, pages 32–38, San Jose, Calif., October 1993.

  20. D. A. Lane. Ufat—A particle tracer for time-dependent flow fields. In Proceedings of IEEE Visualization '94, pages 257–264, Washington, D.C., October 1994.

  21. D. Maier and B. Vance. A call to order. In Proceedings of the Twelfth ACM Symposium on Principles of Database Systems, pages 1–16, Washington, D.C., May 1993.

  22. S. A. Moyer and V. S. Sunderam. PIOUS: A scalable parallel I/O system for distributed computing environments. In Proceedings of the Scalable High-Performance Computing Conference, pages 71–78, May 1994.

  23. H. Nag. Personal communication. Intel Corporation, March 1995.

  24. National Center for Supercomputing Applications. NCSA HDF Reference Manual, Version 3.3. University of Illinois, February 1994.

  25. B. Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.

  26. M. Paleczny, K. Kennedy, and C. Koelbel. Compiler support for out-of-core arrays on data parallel machines. In Proceedings of the Seventh Symposium on the Frontiers of Massively Parallel Computation, pages 110–118, McLean, Va., February 1995.

  27. J. S. Ryan and S. K. Weeratunga. Parallel computation of 3-D Navier-Stokes flowfields for supersonic vehicles. In 31st Aerospace Sciences Meeting and Exhibit, Reno, Nev., 1993. AIAA Paper 93-0064.

  28. S. Sarawagi and M. Stonebraker. Efficient organization of large multidimensional arrays. In Proceedings of the 10th International Conference on Data Engineering, pages 328–336, February 1994.

  29. Scientific and Engineering Software. Stratum Technical Reference Manual. Austin, Tex., October 1989.

  30. K. E. Seamons and M. Winslett. An efficient abstract interface for multidimensional array I/O. In Proceedings of Supercomputing '94, pages 650–659, Washington, D.C., November 1994.

  31. K. E. Seamons and M. Winslett. Physical schemas for large multidimensional arrays in scientific computing applications. In Proceedings of the 7th International Working Conference on Scientific and Statistical Database Management, pages 218–227, Charlottesville, Va., September 1994.

  32. K. E. Seamons and M. Winslett. A data management approach for handling large compressed arrays in high performance computing. In Proceedings of the Seventh Symposium on the Frontiers of Massively Parallel Computation, pages 119–128, McLean, Va., February 1995.

  33. R. Thakur and A. Choudhary. Accessing sections of out-of-core arrays using an extended two-phase method. Technical Report SCCS-685, Northeast Parallel Architectures Center, Syracuse University, January 1995.

  34. R. Thakur, E. Lusk, and W. Gropp. I/O characterization of a portable astrophysics application on the IBM SP and Intel Paragon. Technical Report MCS-P534-0895, Argonne National Laboratory, October 1995. Preprint.

  35. University Corporation for Atmospheric Research, Unidata Program Center. NetCDF User's Guide, Version 2.0. October 1992.

  36. P. Walatka, P. Buning, L. Pierce, and P. Elson. PLOT3D User's Manual, NASA Technical Memorandum 101067. NASA Ames Research Center, Unidata Program Center, July 1992.

  37. XIDAK Inc. Overview of Orion, Version 2 Release 3(4). Palo Alto, Calif., 1991.

  38. H. Zima, P. Brezany, B. Chapman, P. Mehrotra, and A. Schwald. Vienna Fortran - A language specification. Technical Report ICASE Interim Report 21, MS 132c, ICASE, 1992.

Download references

Author information

Authors and Affiliations

  1. Transarc Corporation The Gulf Tower, 15219, Pittsburgh, PA, USA

    Kent E. Seamons

  2. Department of Computer Science, University of Illinois, 61801, Urbana, IL, USA

    Marianne Winslett

Authors
  1. Kent E. Seamons
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marianne Winslett
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Seamons, K.E., Winslett, M. Multidimensional array I/O in Panda 1.0. J Supercomput 10, 191–211 (1996). https://doi.org/10.1007/BF00130709

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF00130709

Keywords

Search

Navigation