Skip to main content
SpringerLink
Log in

Fundamental issues in testing distributed real-time systems

  • Published:
Real-Time Systems Aims and scope Submit manuscript

Abstract

This paper examines the fundamental problems that one faces when testing a distributed, hard real-time system. It specifically identifies the influences of the distributedness and of the real-time requirements of the systems considered. We show how the usual problems of testing become more difficult, and which additional problems are introduced, because of these additional system characteristics.

We identify six such fundamental problems: Organization, Observability, Reproducibility, Host/Target Approach, Environment Simulation, and Representativity. These, as well as their interrelations, are presented in a general framework that is independent of a particular system architecture or application. This framework could serve as a starting point for all activities geared towards a particular system architecture or a specific application.

As an example, we illustrate how these test problems have been handled when developing a test methodology for the distributed real-time system MARS. Finally, the additional issues of test data generation and test result analysis are briefly discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Article Open access 05 August 2022

Roman Rietsche, Christian Dremel, … Jan-Marco Leimeister

References

  • ACM. Working papers from the ACM SIGSOFT rapid prototyping workshop.ACM SIGSOFT Software Engineering Notes, Special Issue on Rapid Prototyping, December 1982.

  • W. R. Adrion, M. A. Branstad, and J. C. Cherniavsky. Validation, verification, and testing of computer software.ACM Computing Surveys, 14(2):159–192, June 1982.

    Google Scholar 

  • R. L. Baber.The Spine of Software. John Wiley & Sons Ltd., 1987.

  • B. Beizer.Software Testing Techniques, 2nd edition, Van Nostrand Reinhold, New York, 1990.

    Google Scholar 

  • B. W. Boehm.Software Engineering Economics. Prentice Hall, 1981.

  • B. W. Boehm. A spiral model of software development and enhancement.ACM SIGSOFT SOftware Engineering Notes, 11(4):22–42, August 1986.

    Google Scholar 

  • P. Brinch Hansen. Testing a multiprogramming system.Software—Practice and Experience, 3(4):145–150, December 1973.

    Google Scholar 

  • P. Brinch Hansen. Reproducible testing of monitors.Software—Practice and Experience, 8(4):721–729, December 1978.

    Google Scholar 

  • K. M. Chandy and L. Lamport. Distributed snapshots: Determining global states of distributed systems.ACM Transactions on Computer Systems, 3(1):63–75, February 1985.

    Google Scholar 

  • J. E. Cooling.Software Design for Real-Time Systems. Chapman and Hall, London, 1991.

    Google Scholar 

  • R. A. DeMillo, W. M. McCracken, R. J. Martin, and J. F. Passafiume.Software Testing and Evaluation. Benjamin/Cummings Publ. Co., 1987.

  • P. S. Dodd and C. V. Ravishankar. Monitoring and debugging distributed real-time programs.Software—Practice and Experience, 22(10):863–877, October 1992.

    Google Scholar 

  • E. L. Duke. V&V of flight and mission-critical software.IEEE Software, 6(3):39–45, May 1989.

    Google Scholar 

  • M. E. Fagan. Design and code inspections to reduce errors in program development.IBM Systems Journal, 15(3):182–211, 1976.

    Google Scholar 

  • M. E. Fagan. Advances in software inspections.IEEE Transactions on Software Engineering, SE-12(7):744–751, July 1986.

    Google Scholar 

  • R. E. Fairley. Tutorial: Static analysis and dynamic testing of computer software.IEEE Computer, 11(4):14–23, April 1978.

    Google Scholar 

  • H. Fujiwara.Logic Testing and Design for Testability. MIT Press, 1985.

  • J. Gait. A debugger for concurrent programs.Software—Practice and Experience, 15(6):539–554, June 1985.

    Google Scholar 

  • J. Gait. A probe effect in concurrent programs.Software—Practice and Experience, 16(3):225–233, March 1986.

    Google Scholar 

  • H. Garcia-Molina, F. Germano, and W. H. Kohler. Debugging a distributed computing system.IEEE Transactions on Software Engineering, SE-10(2):210–219, March 1984.

    Google Scholar 

  • D. Gelperin and B. Hetzel. The growth of software testing.Communications of the ACM, 31(6):687–695, June 1988.

    Google Scholar 

  • R. L. Glass. Real-time: The “lost world” of software debugging and testing.Communications of the ACM, 23(5):264–271, May 1980.

    Google Scholar 

  • H. Gomaa. Software development of real-time systems.Communications of the ACM, 29(7):657–668, July 1986.

    Google Scholar 

  • M. M. Gorlick. The flight recorder: An architectural aid for system monitoring.ACM SIGPLAN Notices, 26(12):175–183, December 1991. Proceedings of the ACM/ONR Workshop on Parallel and Distributed Debugging, Santa Cruz, CA, USA, May 1991.

    Google Scholar 

  • D. Gries.The Science of Programming. Springer-Verlag, 1981.

  • K. Grimm. An effective strategy and automation concepts for systematic testing of safety related software. InIFAC Proceedings SAFECOMP 89, pp. 71–79, Vienna, Austria, December 1989.

  • B. Hetzel.The Complete Guide to Software Testing, 2nd edition. QED Information Sciences, 1988.

  • C. R. Hill. A real-time microprocessor debugging technique.ACM SIGSOFT Software Engineering Notes, 8(4):145–148, August 1983. Proceedings of the ACM SIGSOFT/SIGPLAN Software Engineering Symposium on High-Level Debugging.

    Google Scholar 

  • W. E. Howden. Life-cycle software validation.IEEE Computer, 15(2):71–78, February 1982.

    Google Scholar 

  • M. Hurfin, N. Plouzeau, and M. Raynal. EREBUS: A debugger for asynchronous distributed computing systems. InProc. 3rd IEEE Workshop on Future Trends in Distributed Computing Systems, Taipei, Taiwan, April 1992.

  • IEEE. IEEE Standard Glossary of Software Engineering Terminology, 1983. ANSI/IEEE Standard 729-1983.

  • H. Kopetz. Fault tolerance in real-time systems. InIFAC World Congress, Vol. 7, pp. 111–118. Tallinn, August 1990.

  • H. Kopetz, A. Damm, Ch. Koza, M. Mulazzani, W. Schwabl, Ch. Senft, and R. Zainlinger. Distributed fault-tolerant real-time systems: The MARS approach.IEEE Micro, 9(1):25–40, February 1989.

    Google Scholar 

  • H. Kopetz, R. Zainlinger, G. Fohler, H. Kantz, P. Puschner, and W. Schütz. An engineering approach to hard real-time system design. InProc. of the Third European Software Engineering Conference, ESEC '91, pp. 166–188, Milano, Italy, October 1991.

  • L. Lamport. Time, clocks, and the ordering of events in a distributed system.Communications of the ACM, 21(7):558–565, July 1978.

    Google Scholar 

  • G. Le Lann. Critical issues in distributed real-time computing. InWorkshop on Communication Networks and Distributed Operating Systems within the Space Environment, pp. 90–105. European Space Research and Technology Center, October 1989.

  • J.-C. Laprie. Dependability: A unifying concept for reliable computing and fault tolerance. In T. Anderson, editor,Dependability of Resilient Computers, pp. 1–28. BSP Professional Books, 1989.

  • J.-C. Laprie, editor.Dependability: Basic Concepts and Terminology, Vol. 5 ofDependable Computing and Fault Tolerance. Springer-Verlag, 1992.

  • J. Laski. Testing in the program development cycle.IEE Software Engineering Journal, 4(2):95–106, March 1989.

    Google Scholar 

  • M. Lutz (editor). Testing tools.IEEE Software, 7(3):53–57, May 1990.

  • A. D. Malony, D. A. Reed, and H. A. G. Wijshoff. Performance measurement intrusion and perturbation analysis.IEEE Transactions on Parallel and Distributed Systems, 3(4):433–450, July 1992.

    Google Scholar 

  • D. C. Marinescu, Jr., J. E. Lumpp, T. L. Casavant, and H. J. Siegel. A model for monitoring and debugging parallel and distributed software. InProc. 13th Ann. Int. Computer Software and Application Conference (COMPSAC '89), pp. 81–88, Orlando, Florida, USA, October 1989. IEEE Computer Society Press.

  • E. J. McCluskey. Design for testability. In D. K. Pradhan, editor,Fault-Tolerant Computing, Theory and Techniques (Vol. 1), Ch. 2, pp. 95–183. Prentice-Hall, 1986.

  • C. E. McDowell and D. P. Helmbold. Debugging concurrent programs.ACM Computing Surveys, 21(4):593–622, December 1989.

    Google Scholar 

  • G. J. Myers,The Art of Software Testing. John Wiley and Sons, New York 1979.

    Google Scholar 

  • K. M. Nichols (editor). Performance tools.IEEE Software, 7(3):21–30, May 1990.

  • K. Nielsen and K. Shumate.Designing Large Real-Time Systems with ADA. McGraw-Hill Book Company, 1988.

  • D. L. Parnas, A. J. van Schouwen, and S. P. Kwan. Evaluation of safety-critical software.Communications of the ACM, 33(6):636–648, June 1990.

    Google Scholar 

  • B. Plattner. Real-time execution monitoring.IEEE Transactions on Software Engineering, SE-10(6):756–764, November 1984.

    Google Scholar 

  • G. Pospischil, P. Puschner, A. Vrchoticky, and R. Zainlinger. Developing real-time tasks with predictable timing.IEEE Software, 9(5):35–44, September 1992.

    Google Scholar 

  • R. G. Sargent. Verification and validation of simulation models. In F. E. Cellier, editor,Progress in Modelling and Simulation, pp. 159–169. Academic Press, London, 1982.

    Google Scholar 

  • E. Schoitsch, E. Dittrich, S. Grasegger, D. Kropfitsch, A. Erb, P. Fritz, and H. Kopp. The ELEKTRA testbed: Architecture of a real-time test environment for high safety and reliability requirements. InIFAC Proceedings SAFECOMP 90, pp. 59–65, Gatwick, U.K., October/November 1990.

  • W. Schütz. Real-time simulation in the distributed real-time system MARS. InProc. 1990 European Simulation Multiconference, pp. 51–57, Nürnberg, FRG, June 1990. The Society for Computer Simulation International.

  • W. Schütz. A test strategy for the distributed real-time system MARS. InIEEE CompEuro 90, Computer Systems and Software Engineering, pp. 20–27, Tel Aviv, Israel, May 1990.

  • W. Schütz. On the testability of distributed real-time systems. InProc. 10th Symposium on Reliable Distributed Systems, pp. 52–61, Pisa, Italy, September 1991.

  • W. Schütz.The Testability of Distributed Real-Time Systems. Kluwer Academic Publishers, 1993.

  • K. G. Shin, D. D. Kandlur, D. L. Kiskis, P. S. Dodd, H. A. Rosenberg, and A. Indiresan. A distributed real-time operating system.IEEE Software, 9(5):58–68, September 1992.

    Google Scholar 

  • I. Sommerville.Software Engineering, 3rd edition. Addison-Wesley, 1989.

  • J. A. Stankovic. Misconceptions about real-time computing: A serious problem for next-generation systems.IEEE Computer, 21(10):10–19, October 1988.

    Google Scholar 

  • J. A. Stankovic and K. Ramamritham.IEEE Tutorial: Hard Real-Time Systems. IEEE Computer Society Press, 1988.

  • K. C. Tai. On testing concurrent programs. InProc. 9th Ann. Int. Computer Software and Application Conference (COMPSAC '85), pp. 310–317, Chicago, Illinois, USA, October 1985. IEEE Computer Society Press.

  • K. C. Tai and S. Ahuja. Reproducible testing of communication software. InProc. 11th Ann. Int. Computer Software and Application Conference (COMPSAC '87), pp. 331–337, Tokyo, Japan, October 1987. IEEE Computer Society Press.

  • K. C. Tai, R. H. Carver, and E. E. Obaid. Deterministic execution debugging of concurrent Ada programs. InProc. 13th Ann. Int. Computer Software and Application Conference (COMPSAC '89), pp. 102–109, Orlando, Florida, USA, October 1989. IEEE Computer Society Press.

  • K. C. Tai, R. H. Carver, and E. E. Obaid. Debugging concurrent Ada programs by deterministic execution.IEEE Transactions on Software Engineering, SE-17(1):45–63, January 1991.

    Google Scholar 

  • H. Tokuda, M. Kotera, and C. W. Mercer. A real-time monitor for a distributed real-time operating system.ACM SIGPLAN Notices, 24(1):68–77, January 1989. Proceedings of the ACM SIGPLAN/SIGOPS Workshop on Parallel and Distributed Debugging, Madison, Wisconsin, USA, May 1988.

    Google Scholar 

  • J. P. Tsai, K.-Y. Fang, H.-Y. Chen, and Y.-D. Bi. A noninterference monitoring and replay mechanism for real-time software testing and debugging.IEEE Transactions on Software Engineering, SE-16(8):897–916, August 1990.

    Google Scholar 

  • D. R. Wallace and R. U. Fujii. Software verification and validation: An overview.IEEE Software, 6(3):10–17, May 1989.

    Google Scholar 

  • P. T. Ward and S. J. Mellor.Structured Development for Real-Time Systems, Vols. 1–3. Yourdon Press, 1985.

  • S. N. Weiss. A formal framework for the study of concurrent program testing. InProc. Second Workshop on Software Testing, Verification, and Analysis, pp. 106–113, Banff, Canada, July 1988.

  • A. Wellings. Editorial: Real-time software.IEE Software Engineering Journal (Special Issue on Real-Time Software), 6(3):66–67, May 1991.

Download references

Author information

Author notes

  1. Werner Schütz

    Present address: Alcatel Austria AG, Wien, Austria

Authors and Affiliations

  1. LAAS-CNRS, F-31077, Toulouse Cedex, France

    Werner Schütz

Authors
  1. Werner Schütz
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Schütz, W. Fundamental issues in testing distributed real-time systems. Real-Time Syst 7, 129–157 (1994). https://doi.org/10.1007/BF01088802

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF01088802

Keywords

Search

Navigation