Devising optimal integration test orders using cost–benefit analysis

Abstract

Integration testing is an integral part of software testing. Prior studies have focused on reducing test cost in integration test order generation. However, there are no studies concerning the testing priorities of critical classes when generating integration test orders. Such priorities greatly affect testing efficiency. In this study, we propose an effective strategy that considers both test cost and efficiency when generating test orders. According to a series of dynamic execution scenarios, the software is mapped into a multi-layer dynamic execution network (MDEN) model. By analyzing the dynamic structural complexity, an evaluation scheme is proposed to quantify the class testing priority with the defined class risk index. Cost—benefit analysis is used to perform cycle-breaking operations, satisfying two principles: assigning higher priorities to higher-risk classes and minimizing the total complexity of test stubs. We also present a strategy to evaluate the effectiveness of integration test order algorithms by calculating the reduction of software risk during their testing process. Experiment results show that our approach performs better across software of different scales, in comparison with the existing algorithms that aim only to minimize test cost. Finally, we implement a tool, ITOsolution, to help practitioners automatically generate test orders.

摘要

集成测试是面向对象软件测试的重要组成部分。传统的类级集成测试顺序研究策略大多围绕如何降低测试成本开展工作, 并未考虑赋予可靠性风险较大的节点较高测试优先级, 从而影响软件测试效率。本文提出一种兼顾测试成本与测试效率的方法生成集成测试序列。根据软件在不同场景下的运行状态, 将其映射成多层动态执行网络 (multi-layer dynamic execution network, MDEN) 。借助该网络模型与概率风险评估方法为软件中每一个类赋予风险权重。利用成本收益分析方法, 在生成测试用例的过程中保证两条原则: 为高风险的类赋予较高权重, 同时最小化测试桩复杂度。在此基础上, 分析测试序列对软件系统总体运行风险的影响, 从而提出评估测试序列优劣的度量方案。通过与现有算法的实验对比分析, 证明所提算法生成的类级集成测试序列能有效降低测试代价。最后, 将所提算法实现为自动生成集成测试序列的开源工具ITOsolution。

Supplementary materials for

Devising optimal integration test orders using cost–benefit analysis