Skip to main content

Advertisement

Springer Nature Link
Log in

Simulation Might Change Your Results: A Comparison of Context-Aware System Input Validation in Simulated and Physical Environments

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

Context-aware systems (a.k.a. CASs) integrate cyber and physical space to provide adaptive functionalities in response to changes in context. Building context-aware systems is challenging due to the uncertain running environment. Therefore, many input validation approaches have been proposed to protect context-aware systems from uncertainty and keep them executing safely. However, in contrast to context-aware systems’ prevailing in physical environments, most of those academic solutions (83%) are purely evaluated in simulated environments. In this article, we study whether this evaluation setting could lead to biased conclusions. We build a testing platform, RM-Testing, based on DJI RoboMaster robot car, to conduct the physical-environment based experiments. We select three up-to-date input validation approaches, and compare their performance in the simulated environment and in the physical environment. The experimental results show that all three approaches’ performance in simulated environments (improving task success rate by 82% compared with the system without the support of input validation) does differ from their performance in a physical environment (improving the task success rate by 50%). We also recognize three factors (scenario setting, physical platform and environmental model) that affect the performance of input validation approaches, based on an execution model of the context-aware system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Lü J, Ma X, Huang Y, Cao C, Xu F. Internetware: A shift of software paradigm. In Proc. the 1st Asia-Pacific Symposium on Internetware, October 2009, Article No. 7. https://doi.org/10.1145/1640206.1640213.

  2. Sama M, Elbaum S, Raimondi F, Rosenblum D S, Wang Z. Context-aware adaptive applications: Fault patterns and their automated identification. IEEE Transactions on Software Engineering, 2010, 36(5): 644-661. https://doi.org/10.1109/TSE.2010.35.

    Article  Google Scholar 

  3. Qin Y, Xu C, Chen Z, Lü J. Software testing for cyber-physical systems suffering uncertainty. SCIENTIA SINICA Informationis, 2019, 49(11): 1428-1450. https://doi.org/10.1360/N112018-00305. (in Chinese)

  4. Sama M, Rosenblum D S, Wang Z, Elbaum S. Model-based fault detection in context-aware adaptive applications. In Proc. the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering, November 2008, pp.261-271. https://doi.org/10.1145/1453101.1453136.

  5. Xu C, Cheung S C, Ma X, Cao C, Lu J. Adam: Identifying defects in context-aware adaptation. Journal of Systems and Software, 2012, 85(12): 2812-2828. https://doi.org/10.1016/j.jss.2012.04.078.

    Article  Google Scholar 

  6. Xu C, Cheung S C, Chan W K, Ye C. Partial constraint checking for context consistency in pervasive computing. ACM Transactions on Software Engineering and Methodology, 2010, 19(3): Article No. 9. https://doi.org/10.1145/1656250.1656253.

  7. Qin Y, Xie T, Xu C, Astorga A, Lu J. CoMID: Context-based multiinvariant detection for monitoring cyber-physical software. IEEE Transactions on Reliability, 2019, 69(1): 106-123. https://doi.org/10.1109/TR.2019.2933324.

    Article  Google Scholar 

  8. Wang H, Xu J, Xu C, Ma X, Lu J. DISSECTOR: Input validation for deep learning applications by crossing-layer dissection. In Proc. the 42nd ACM/IEEE International Conference on Software Engineering, Oct. 2020, pp.727-738. https://doi.org/10.1145/3377811.3380379.

  9. Ramirez A J, Jensen A C, Cheng B H, Knoester D B. Automatically exploring how uncertainty impacts behavior of dynamically adaptive systems. In Proc. the 26th IEEE/ACM International Conference on Automated Software Engineering, November 2011, pp.568-571. https://doi.org/10.1109/ASE.2011.6100127.

  10. Aliabadi M R, Kamath A A, Gascon-Samson J, Pattabiraman K. ARTINALI: Dynamic invariant detection for cyberphysical system security. In Proc. the 11th Joint Meeting on Foundations of Software Engineering, Sept. 2017, pp.349-361. https://doi.org/10.1145/3106237.3106282.

  11. Wang Z, Elbaum S, Rosenblum D S. Automated generation of context-aware tests. In Proc. the 29th International Conference on Software Engineering, May 2007, pp.406-415. https://doi.org/10.1109/ICSE.2007.18.

  12. Nentwich C, Capra L, Emmerich W, Finkelsteiin A. xlinkit: A consistency checking and smart link generation service. ACM Transactions on Internet Technology, 2002, 2(2): 151-185. https://doi.org/10.1145/514183.514186.

    Article  Google Scholar 

  13. Egyed A. Fixing inconsistencies in UML design models. In Proc. the 29th International Conference on Software Engineering, May 2007, pp.292-301. https://doi.org/10.1109/ICSE.2007.38.

  14. Chu X, Morcos J, Ilyas I F, Ouzzani M, Papotti P, Tang N, Ye Y. KATARA: A data cleaning system powered by knowledge bases and crowdsourcing. In Proc. the 2015 ACM SIGMOD International Conference on Management of Data, May 31-June 4, 2015, pp.1247-1261. https://doi.org/10.1145/2723372.2749431.

  15. Pei K, Cao Y, Yang J, Jana S. DeepXplore: Automated whitebox testing of deep learning systems. In Proc. the 26th Symposium on Operating Systems Principles, October 2017, pp.1-18. https://doi.org/10.1145/3132747.3132785.

  16. Tian Y, Pei K, Jana S, Ray B. DeepTest: Automated testing of deep-neural-network-driven autonomous cars. In Proc. the 40th International Conference on Software Engineering, May 27–June 3, 2018, pp.303-314. https://doi.org/10.1145/3180155.3180220.

  17. Luo C, Goncalves J, Velloso E, Kostakos V. A survey of context simulation for testing mobile context-aware applications. ACM Computing Surveys, 2020, 53(1): Article No. 21. https://doi.org/10.1145/3372788.

  18. Chen J, Qin Y, Wang H, Xu C. Simulated or physical? An empirical study on input validation for context-aware systems in different environments. In Proc. the 12th Asia-Pacific Symposium on Internetware, Nov. 2021, pp.146-155. https://doi.org/10.1145/3457913.3457919.

  19. Paden B, Čáp M, Yong S Z, Yershov D, Frazzoli E. A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Transactions on Intelligent Vehicles, 2016, 1(1): 33-55. https://doi.org/10.1109/TIV.2016.2578706.

    Article  Google Scholar 

  20. Badue C, Guidolini R, Carneiro R V et al. Self-driving cars: A survey. Expert Systems with Applications, 2021, 165: Article No. 113816. https://doi.org/10.1016/j.eswa.2020.113816.

  21. Ma N, Gao Y, Li J, Li D. Interactive cognition in self-driving. SCIENTIA SINICA Informationis, 2018, 48(8): 1083-1096. https://doi.org/10.1360/N112018-00028. (in Chinese)

  22. Xi W, Xu C, Yang W, Hong X. How context inconsistency and its resolution impact context-aware applications. Journal of Frontiers of Computer Science and Technology, 2014, 8(4): 427-437. https://doi.org/10.3778/j.issn.1673-9418.1311013. (in Chinese)

  23. Esfahani N, Malek S. Uncertainty in self-adaptive software systems. In Proc. the International Seminar on Software Engineering for Self-Adaptive Systems, Oct. 2013, pp.214-238. https://doi.org/10.1007/978-3-642-35813-5_9.

  24. Fredericks E M, DeVries B, Cheng B H C. Towards run-time adaptation of test cases for self-adaptive systems in the face of uncertainty. In Proc. the 9th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, June 2014, pp.17-26. https://doi.org/10.1145/2593929.2593937.

  25. Jiang H, Elbaum S, Detweiler C. Reducing failure rates of robotic systems though inferred invariants monitoring. In Proc. the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, November 2013, pp.1899-1906. https://doi.org/10.1109/IROS.2013.6696608.

  26. Al-Garadi M A, Mohamed A, Al-Ali A K, Du X, Ali I, Guizani M. A survey of machine and deep learning methods for Internet of Things (IoT) security. IEEE Communications Surveys & Tutorials, 2020, 22(3): 1646-1685. https://doi.org/10.1109/COMST.2020.2988293.

  27. Li Z, Ma X, Xu C, Cao C, Xu J, Lü J. Boosting operational DNN testing efficiency through conditioning. In Proc. the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, August 2019, pp.499-509. https://doi.org/10.1145/3338906.3338930.

  28. Li Z, Ma X, Xu C, Xu J, Cao C, Lü J. Operational calibration: Debugging confidence errors for DNNs in the field. In Proc. the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, November 2020, pp.901-913. https://doi.org/10.1145/3368089.3409696.

  29. Nentwich C, Emmerich W, Finkelsteiin A, Ellmer E. Flexible consistency checking. ACM Transactions on Software Engineering and Methodology, 2003, 12(1): 28-63. https://doi.org/10.1145/839268.839271.

    Article  Google Scholar 

  30. Gehrke J, Madden S. Query processing in sensor networks. IEEE Pervasive Computing, 2004, 3(1): 46-55. https://doi.org/10.1109/MPRV.2004.1269131.

    Article  Google Scholar 

  31. Tong Y, Qin Y, Jiang Y, Xu C, Cao C, Ma X. Timely and accurate detection of model deviation in self-adaptive software-intensive systems. In Proc. the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, August 2021, pp.168-180. https://doi.org/10.1145/3468264.3468548.

  32. Ernst M D, Perkins J H, Guo P J, McCamant S, Pacheco C, Tschantz M S, Xiao C. The Daikon system for dynamic detection of likely invariants. Science of Computer Programming, 2007, 69(1/2/3): 35-45. https://doi.org/10.1016/j.scico.2007.01.015.

    Article  MathSciNet  MATH  Google Scholar 

  33. Jiang H, Elbaum S, Detweiler C. Inferring and monitoring invariants in robotic systems. Autonomous Robots, 2017, 41(4): 1027-1046. https://doi.org/10.1007/s10514-016-9576-y.

    Article  Google Scholar 

  34. Bai X, Yang M, Shi B, Liao M. Deep learning for scene text detection and recognition. SCIENTIA SINICA Informationis, 2018, 48(5): 531-544. https://doi.org/10.1360/N112018-00003. (in Chinese)

  35. Matalonga S, Travassos G H. Testing context-aware software systems: Unchain the context, set it free! In Proc. the 31st Brazilian Symposium on Software Engineering, September 2017, pp.250-254. https://doi.org/10.1145/3131151.3131190.

  36. Dourish P. What we talk about when we talk about context. Personal and Ubiquitous Computing, 2004, 8(1): 19-30. https://doi.org/10.1007/s00779-003-0253-8.

    Article  Google Scholar 

  37. Ramirez A J, Jensen A C, Cheng B H C. A taxonomy of uncertainty for dynamically adaptive systems. In Proc. the 7th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, June 2012, pp.99-108. https://doi.org/10.1109/SEAMS.2012.6224396.

  38. Yang W, Xu C, Liu Y, Cao C, Ma X, Lu J. Verifying self-adaptive applications suffering uncertainty. In Proc. the 29th ACM/IEEE International Conference on Automated Software Engineering, September 2014, pp.199-210. https://doi.org/10.1145/2642937.2642999.

  39. Wohlin C, Runeson P, Höst M, Ohlsson M C, Regnell B, Wesslén A. Experimentation in Software Engineering. Springer, 2000. https://doi.org/10.1007/978-3-642-29044-2.

  40. Zhang L, Xu C, Ma X, Gu T, Hong X, Cao C, Lu J. Resynchronizing model-based self-adaptive systems with environments. In Proc. the 19th Asia-Pacific Software Engineering Conference, December 2012, pp.184-193. https://doi.org/10.1109/APSEC.2012.62.

  41. Dey A K, Abowd G D, Salber D. A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Human-Computer Interaction, 2001, 16(2): 97-166. https://doi.org/10.1207/S15327051HCI16234_02.

    Article  Google Scholar 

  42. Magnusson S. Similarities and differences in psychophysiological reactions between simulated and real air-to-ground missions. The International Journal of Aviation Psychology, 2002, 12(1): 49-61. https://doi.org/10.1207/S15327108IJAP1201_5.

    Article  Google Scholar 

  43. Bishop I D, Rohrmann B. Subjective responses to simulated and real environments: A comparison. Landscape and Urban Planning, 2003, 65(4): 261-277. https://doi.org/10.1016/S0169-2046(03)00070-7.

    Article  Google Scholar 

  44. Qin Y, Xu C, Yu P, Lu J. SIT: Sampling-based interactive testing for self-adaptive apps. Journal of Systems and Software, 2016, 120: 70-88. https://doi.org/10.1016/j.jss.2016.07.002.

    Article  Google Scholar 

  45. Weyns D, Schmerl B, Grassi V, Malek S, Mirandola R, Prehofer C, Wuttke J, Andersson J, Giese H, Göschka K M. On patterns for decentralized control in self-adaptive systems. Software Engineering for Self-Adaptive Systems II, 2013, pp.76-107. https://doi.org/10.1007/978-3-642-35813-5_4.

  46. Kephart J O, Chess D M. The vision of autonomic computing. Computer, 2003, 36(1): 41-50. https://doi.org/10.1109/MC.2003.1160055.

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, 210023, China

    Jin-Chi Chen, Yi Qin, Hui-Yan Wang & Chang Xu

Authors
  1. Jin-Chi Chen
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Yi Qin
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Hui-Yan Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Chang Xu
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding authors

Correspondence to Yi Qin or Chang Xu.

Additional information

Yi Qin mainly contributed to the empirical study on input validation’s effectiveness in two different environments. Chang Xu mainly contributed to the comparison between the physical and simulated environments.

Supplementary Information

ESM 1

(PDF 177 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, JC., Qin, Y., Wang, HY. et al. Simulation Might Change Your Results: A Comparison of Context-Aware System Input Validation in Simulated and Physical Environments. J. Comput. Sci. Technol. 37, 83–105 (2022). https://doi.org/10.1007/s11390-021-1669-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-021-1669-1

Keywords