Performance and Robustness Engineering and the Role of Automated Software Development

Abstract

Performance engineering aims to demonstrate that the software being developed will meet the performance needs. The goal of robustness engineering is to prove that the system will work correctly in the presence or after occurence of faults or stress conditions. From this point of view robustness engineering complements performance engineering to cover the full range of conditions to which a system1 may be exposed. Performance and robustness properties need to be continuously monitored during the development process to ensure that the system will meet the user’s needs at the end. This paper will discuss aspects and problems of performance and robustness engineering. Also, it presents an approach, called “ISG” (Instantaneous System and Software Generation)2 which allows to continuously derive performance and robustness properties from the system-under-development. In case of ISG figures are derived from the real system right from the beginning. Therefore deviations from the desired functional, performance and robustness envelope can be corrected at an early stage. The capability for getting an immediate3 feedback from the system is obtained by the automated generation of the software from system engineering inputs. ISG builds the software by construction rules. This reduces the manual effort and allows for an immediate and representative feedback right after provision of inputs by the user. Due to automation the system can easily be instrumented on a user’s demand without requiring any additional programming effort. ISG automatically stimulates the system and exposes it to stress tests and fault injection, and records coverage and performance figures. By incremental development a smooth transition from the first idea to the final version is possible at low risk. The ISG approach has been applied to the domain of realtime, distributed, fault-tolerant systems and shall be extended towards other application domains in future such as databases and graphical user interfaces.

A “system” as it is understood here comprises hardware and software. System development in the context of this paper addresses the software part, possibly hardware emulated by software and all aspects related to hardware-software integration. Performance and robustness engineering deal with the quality of service under nominal and non-nominal conditions of the software when executed on the system’s hardware.

© The ISG approach is property of Dr. Rainer Gerlich BSSE System and Software Engineering. All features of ISG are protected by international copyright, all rights are reserved (1999–2000)

“Immediate” means a range of about 20 minutes up to 2 hours depending on the complexity of the system on an UltraSparc I/143 or PC 200.