New Trends and Ideas
SIT: Sampling-based interactive testing for self-adaptive apps

https://doi.org/10.1016/j.jss.2016.07.002Get rights and content

Highlights

  • We propose an interactive application model for understanding interactions between an application and its running environment.

  • We propose a test generation technique for systematically sampling an application’s input space in a light-weight way.

  • We conduct experiments to evaluate our approach with real-world self-adaptive applications.

Abstract

Self-adaptive applications (“apps” for short) are useful but error-prone. This stems from developers’ inadequate consideration of environmental dynamics and uncertainty. Two features of self-adaptive apps, infinite reaction loop and uncertain interaction, bring additional challenges to software testing and make existing approaches ineffective. In this article, we propose a novel approach SIT (Sample-based Interactive Testing) to testing self-adaptive apps effectively and in a light-weight way. Our key insight is that a self-adaptive app’s input space can be systematically split, adaptively explored, and mapped to the testing of the app’s different behavior. This is achieved by our approach’s two components, an interactive app model and a test generation technique. The former captures characteristics of interactions between an app and its environment, and the latter uses adaptive sampling to explore an app’s input space and test its behavior. We experimentally evaluated our approach with real-world self-adaptive apps. The experimental results reported that our SIT improved the bug detection by 22.4–42.2%, but with a smaller time cost. Besides, SIT is also scalable with our tailored optimization techniques.

Introduction

Self-adaptive applications (“apps” for short) are gaining increasing attention. Their programs contain adaptation logic, which decide ways of delivering services based on varying environmental conditions (McKinley et al., 2004). Such apps typically run on embedded systems or smartphone platforms, using their sensors to collect environmental conditions. Different from traditional programs, a self-adaptive app involves a closed interaction loop that connects the app to its running environment, in which the app senses environmental changes, makes decisions based on its adaptation logic and performs adaptation to cope with the sensed changes. Examples of such self-adaptive apps include robot- car (Yang et al., 2014), which controls an intelligent automatic car to explore an unknown area, phone-adaptor (Sama et al., 2008)1, which adapts the working mode of a smartphone according to its sensed changes in its user contexts, and Locale,2 which provides location-based services (e.g., reminding book-returning near library or meeting schedules in office) automatically.

While self-adaptive apps offer flexible functionalities, they have to address complexities incurred by environmental dynamics and uncertainty. Empirical evidence shows that building self-adaptive apps is challenging and they are easily error-prone (Kulkarni, Tripathi, 2010, Sama, Elbaum, Raimondi, Rosenblum, Wang, 2010, Xu, Cheung, Ma, Cao, Lu, 2012). Regarding this, we identify two major causes:

A self-adaptive app executes in an infinite reaction loop with its environment (Brun et al., 2009). This execution model differs from those of many traditional apps, which take finite values (as input) and return results (as output). A self-adaptive app’s reaction loop incurs an infinite series of input/output pairs, and this forms a large state space, in which the app’s behavior can hardly be adequately tested. Besides, due to physical constraints, this series of input/output pairs has inherent correlations on their values. For example, a specific output (say, action of moving a robot-car forward) changes an environment’s status, which then affects the concerned app’s next input (say, shortening the distance between the car and its facing obstacle). As such, testing self-adaptive apps is non-trivial in that it has to take environment into consideration. In this article, we use iteration to denote one pass in executing an app’s reaction loop (i.e., sensing-decision-adaptation).

A self-adaptive app also interacts with its environment in an uncertain way (Cheng, 2009). This refers to both internal uncertainty (difficulty of deciding the impact of adaptation on an app’s goal realization) and external uncertainty (difficulty of deciding the error in measuring environmental conditions) (Esfahani et al., 2011). While the former concerns more on adaptation algorithms, we in this article focus on the latter, which affects how a self-adaptive app understands its environment and takes normal or abnormal behavior.

These two causes together decide the differences between the bugs in self-adaptive apps and those in traditional programs. While a traditional program’s bugs depend mostly on the program itself (e.g., developers’ mistakes in writing programs), a self-adaptive app’s bugs would rely on three factors, namely, the app (program) itself, its running environment, and the uncertainty that affects app-environment interactions. First, the app may itself have implementation defects in dealing with app-environment interactions. Second, the app’s running environment follows certain physical constraints and this may trigger program bugs in a way independent of the app itself. Finally, uncertainty in app-environment interactions can also cause the app’s execution to deviate from its supposed path, leading to unexpected failure.

The uniqueness of bugs in self-adaptive app contributes to the challenges in effectively testing self-adaptive apps. An app can fail after accumulating multiple executions of its reaction loop with its environment. These executions are different iterations of the same program loop but with different input/output pairs. Testing such an app has to connect all these iterations, and this forms a large space, in which failure can be easily missed. Uncertainty further worsens the testing practice by making the input/output pairs deviate from their ideal values. For example, a robot-car can have its sensed distance contain unpredicted error, and its turning-direction action can also be imprecise due to uncontrollable rub with the ground. Such uncertainty can cause new problems to testing self-adaptive apps. For example, if the car senses its environment once per second and it runs for only one minute, then its app’s execution can contain up to 260 possibilities, even if uncertainty causes only two values to each input. Searching for a bug in such a large space is difficult. Thus the testing has to be extremely efficient to be useful.

Existing work proposed various random testing techniques to address the large space problem for self-adaptive apps, e.g., by considering metamorphic relations (Tse et al., 2004), context-switching points (Wang et al., 2007) and adverse environmental conditions (Ramirez et al., 2011). While they improve the testing efficiency, a systematic exploration of an app’s space is not guaranteed. Moreover, the uncertainty issue is also overlooked. Our later evaluation reports that random testing could miss 39.2–64.2% bugs in self-adaptive apps.

Some other work focuses on guided testing for systematic exploration of an app’s space. Dynamic symbolic execution (DSE) or concolic testing (Godefroid, Klarlund, Sen, 2005, Sen, Marinov, Agha, 2005) and their variants are typical examples. When applied for testing self-adaptive apps, these techniques are also inadequate due to their low-efficiency caused by constraint solving. Moreover, physical constraints from environment (named environmental constraints) are usually, but such constraints are usually missing, and this causes solving-based techniques unable to explore an app’s space completely. Our later evaluation reports that DSE could miss 25–40.5% bugs in self-adaptive apps. Besides, DSE itself is very time-consuming and it could explore very limited reaction interactions given the same time budget.

Self-adaptive app resembles mobile app in that both of them relate the environment. However, our work is essentially testing self-adaptive apps that interact with their environments, which can act in an uncertain way, rather than testing mobile apps without considering their environments and uncertainty explicitly (although such apps can also be our subjects in some situations as shown in our evaluation). Testing self-adaptive apps differs from testing mobile apps. For example, the former needs to consider an app’s environment for a complete interaction-and-adaptation loop and this loop can go for multiple iterations, while the latter focuses mostly on an app itself. In other words, when considering sensing inputs from multiple iterations between an app and its environment, testing a mobile app seldom considers the inherent constraints of the sequences of sensing inputs of the app. This might cause problems to testing self-adaptive apps that interact with their environments, if one uses the same testing approach. For example, our later evaluation reports that bugs detected for self-adaptive apps by a mobile app testing approach (Liang et al., 2014) can contain up to 78.5% false positives.

In this article, we propose a novel approach SIT (Sample-based Interactive Testing) for testing self-adaptive apps. It consists of an interactive app model and a test generation technique. The model captures the characteristics of interactions between an app and its environment, and the technique uses adaptive sampling to systematically explore an app’s space. They together contribute to our SIT approach’s effectiveness. On one hand, our interactive app model considers the impact of environmental constraints and uncertainty on an app’s input/output pairs, and enables a systematic exploration of its space. This exploration is guided, as compared to random testing. On the other hand, our test generation technique is light-weight by only sampling those inputs required by the exploration. Besides, the sampling does not rely on constraint solving, as compared to DSE. Therefore, it is highly efficient.

We experimentally evaluated our SIT approach with real-world self-adaptive apps and compared it with existing work. The experimental results consistently showed our SIT’s effectiveness and efficiency. In particular, SIT improved the bug detection by 22.4–42.2% but with a smaller time cost. In summary, we make the following contributions in this article:

  • Proposed an interactive app model for understanding interactions between an app and its running environment;

  • Proposed a test generation technique for systematically sampling an app’s space in a light-weight way;

  • Conducted experiments to evaluate our approach with real-world self-adaptive apps.

The remainder of this article is organized as follows. Section 2 introduces our interactive app model for understanding how an app interacts with its environment, and presents a motivating example. Section 3 elaborates on our sampling-based test generation based on our interactive app model. Section 4 evaluates our SIT approach and compares it with existing work. Section 5 discusses related work, and finally Section 6 concludes this article and discusses future work.

Section snippets

Preliminaries

In this section, we introduce our interactive app model and present our motivating example based on this model.

Sampling-based interactive testing

In this section, we present our SIT approach for testing self-adaptive apps suffering uncertainty.

Evaluation

In this section, we discuss the evaluation of our SIT approach with real-world self-adaptive apps. Section 4.1 introduces the implementation of our SIT approach. Section 4.2 presents our research questions for study, as well as variables and metrics for measuring these variables derived from the questions. Section 4.3 introduces the selected subjects. Section 4.4 discusses issues relating to our preparations for experiments. Section 4.5 elaborates on our experimental design and procedure.

Related work

In this section, we discuss related work on handling uncertainty for self-adaptive apps and testing for self-adaptive, context-aware, mobile or numeric programs.

Conclusion and future work

In this article, we focus on testing self-adaptive apps. We analyzed characteristics of such apps and studied their challenges to software testing. We proposed a novel approach, named SIT, to testing self-adaptive apps in a systematic and light-weight way. Our experimental evaluation reported promising results, showing that SIT can detect more bugs by covering more code and exploring more iterations, but with smaller time cost, as compared with existing work.

Our work still has limitations. It

Acknowledgment

This work was supported in part by National Basic Research 973 Program (Grant no. 2015CB352202), and National Natural Science Foundation (Grant nos. 61472174, 91318301, 61321491) of China.

Yi Qin is a Ph.D. student with the State Key Laboratory for Novel Software Technology and the Department of Computer Science and Technology of Nanjing University. His research interests include self-adaptive application testing.

References (50)

  • ChenT.Y. et al.

    Adaptive random testing: the art of test case diversity

    J. Syst. Softw.

    (Jan. 2010)
  • T. Toftkjær et al.

    The impact of sensor errors and building structures on particle filter-based inertial positioning

    Pervasive Mob. Comput.

    (2012)
  • XuC. et al.

    Adam: identifying defects in context-aware adaptation

    J. Syst. Softw.

    (2012)
  • C.Q. Adamsen et al.

    Systematic execution of android test suites in adverse conditions

    Proceedings of the 2015 International Symposium on Software Testing and Analysis, ISSTA ’15

    (2015)
  • D. Amalfitano et al.

    Considering context events in event-based testing of mobile applications

    Proceedings of IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops, ICSTW’ 13

    (2013)
  • S. Anand et al.

    Automated concolic testing of smartphone apps

    Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, FSE ’12

    (2012)
  • BaoT. et al.

    White box sampling in uncertain data processing enabled by program analysis

    Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications, OOPSLA ’12

    (2012)
  • Y. Brun et al.

    Engineering self-adaptive systems through feedback loops

    Software eEgineering for Self-adaptive Systems

    (2009)
  • Caiipa, 2014,...
  • S. Chaudhuri et al.

    Proving programs robust

    Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, ser. ESEC/FSE ’11

    (2011)
  • ChengB.H.e. a.

    Software engineering for self-adaptive systems

    Software Engineering for Self-Adaptive Systems: A Research Roadmap

    (2009)
  • L. De Moura et al.

    Z3: an efficient smt solver

    Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, ser. TACAS’08/ETAPS’08

    (2008)
  • C. Duntgen et al.

    Berlinmod: a benchmark for moving object databases

    VLDB J.

    (2009)
  • N. Esfahani

    A framework for managing uncertainty in self-adaptive software systems

    Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’11

    (2011)
  • N. Esfahani et al.

    Taming uncertainty in selfadaptive software

    Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, ser. ESEC/FSE ’11

    (2011)
  • M. Famelis et al.

    Partial models: towards modeling and reasoning with uncertainty

    Proceedings of the 34th International Conference on Software Engineering, ICSE ’12

    (2012)
  • Java Path Finder, 2013....
  • G. Fraser et al.

    Mutation-driven generation of unit tests and oracles

    Proceedings of the 19th International Symposium on Software Testing and Analysis, ISSTA ’10

    (2010)
  • E.M. Fredericks et al.

    Towards runtime adaptation of test cases for self-adaptive systems in the face of uncertainty

    Proceedings of the 9th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, SEAMS ’14

    (2014)
  • E.M. Fredericks et al.

    Towards run-time testing of dynamic adaptive systems

    Proceedings of the 8th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, SEAMS ’13

    (2013)
  • C. Ghezzi et al.

    Managing nonfunctional uncertainty via model-driven adaptivity

    Proceedings of the 2013 International Conference on Software Engineering, ICSE ’13

    (2013)
  • P. Godefroid et al.

    Dart: Directed automated random testing

    Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’05

    (2005)
  • T. Griebe et al.

    A model-based approach to test automation for context-aware mobile applications

    Proceedings of the 29th Annual ACM Symposium on Applied Computing, SAC’14

    (2014)
  • T. Griebe et al.

    A model-based approach to test automation for context-aware mobile applications

    Proceedings of the 29th Annual ACM Symposium on Applied Computing, SAC ’14

    (2014)
  • M. Jang et al.

    Simulation framework for testing context-aware ubiquitous applications

    Proceedings of the 7th International Conference on Advanced Communication Technology, ICACT ’05

    (2005)
  • Cited by (31)

    • Alternatives for testing of context-aware software systems in non-academic settings: results from a Rapid Review

      2022, Information and Software Technology
      Citation Excerpt :

      S11. Qin et al. [51] present an approach for selecting test cases for self-adaptive context-aware software systems. Context variation is considered in terms of uncertainty in the adaptation rules and uncertainty in the sensor's capacity to measure the environment accurately.

    • Self-adaptive systems: A systematic literature review across categories and domains

      2022, Information and Software Technology
      Citation Excerpt :

      Mobile systems most likely also share the same similarities with overlap as networking and robotics to IoT and IaaS. They have been a consistent theme across the timelines and have a presence in this one with dynamic decisions [156], input space mapping [267] and emotion measurement [298]. A review of self-adaptive systems in the context of mobile systems is given in [213], one on monitoring self-adaptive applications within edge computing frameworks in [209], and one on learning in self-adaptive systems in [214].

    • Tigris: A DSL and framework for monitoring software systems at runtime

      2021, Journal of Systems and Software
      Citation Excerpt :

      These analysis methods support fundamental software engineering tasks such as the addition of features, debugging, validation of quality requirements, performance engineering, or optimization (Feng et al., 2018). Fully dynamic and online monitoring is the core of self-adaptive systems (SAS), which are those capable of adapting their behavior to keep satisfying their requirements in dynamic environments (Qin et al., 2016; Tanabe et al., 2017; Mertz and Nunes, 2018). This typically involves collecting and storing runtime data from software components and the environment in such a way that the collected data can be analyzed and used to guide adaptations in the system.

    • Runtime testing of context-aware variability in adaptive systems

      2021, Information and Software Technology
      Citation Excerpt :

      Additionally, RETAkE is a grey-box approach that must be part of the DAS under test, combining verification techniques instead of using them on isolation. Qin et al. [38] propose an approach called SIT (Sample-based Interactive Testing) for testing self-adaptive applications. This approach explores the input space through systematic sampling and returns a set of sequences of value assignments to the context-aware application’s input parameters that exercise different application behaviors.

    • Adaptive generation of challenging scenarios for testing and evaluation of autonomous vehicles

      2018, Journal of Systems and Software
      Citation Excerpt :

      One testing method is to simply stimulate the system with an enormous amount of inputs for fault detection (Christensen et al., 2008; Meinke and Nycander, 2015), often utilizing optimization-based combination testing (Mahmoud and Ahmed, 2015) to minimize the size of the test suite while maximizing coverage. Additionally, sampling-based methods have been used to discover different performance modes for a car control system (Qin et al., 2016). All of these testing methods provide run-time assurances for the robustness of the software, but they do not provide information about the how the autonomy will perform when executing a mission.

    View all citing articles on Scopus

    Yi Qin is a Ph.D. student with the State Key Laboratory for Novel Software Technology and the Department of Computer Science and Technology of Nanjing University. His research interests include self-adaptive application testing.

    Chang Xu is an associate professor with the State Key Laboratory for Novel Software Technology and the Department of Computer Science and Technology of Nanjing University. He received his Ph.D. degree in computer science and engineering from the Hong Kong University of Science and Technology. His research interests include big data software engineering, software testing and analysis, and adaptive and embedded system.

    Ping Yu is an assistant professor with the State Key Laboratory for Novel Software Technology and the Department of Computer Science and Technology of Nanjing University. She received her Ph.D. degree in computer science and technology from Nanjing University. Her research interests include software engineering and methodology.

    Jian Lu is a full professor with the State Key Laboratory for Novel Software Technology and the Department of Computer Science and Technology of Nanjing University. He received his Ph.D. degree in computer science and technology from Nanjing University. Her research interests include software engineering and methodology.

    View full text