Skip to main content

Search-Based Temporal Testing in an Embedded Multicore Platform

  • Conference paper
  • First Online:
Applications of Evolutionary Computation (EvoApplications 2018)

Abstract

Multicore processors have now become the norm. However, for many embedded real-time systems their use introduces challenges in verification as their shared components are potential channels for interference. Of particular interest is the determination for each task of its worst case (longest) execution time (WCET). In this paper, we investigate the effectiveness of a variety of metaheuristic search algorithms for dynamically finding extreme execution times of tasks executing on a multicore processor. Over finite search spaces, these are shown to perform considerably better than randomly generated test inputs and the work reveals significant performance differences between the various algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    The GSL used in this paper is version 2.1, while the current one is version 2.4 (released on 19 June 2017 [30]).

  2. 2.

    Single-state methods have recently been included in ECJ version 25.

  3. 3.

    The time from another clock source, namely CLOCK_REALTIME, may leap forward or even backward after a time adjustment.

References

  1. Burns, A., McDermid, J.A.: Real-time safety-critical systems: analysis and synthesis. Softw. Eng. J. 9(6), 267–281 (1994)

    Article  Google Scholar 

  2. Engel, A.: Verification, Validation and Testing of Engineered Systems. Wiley Series in Systems Engineering and Management, 2nd edn. John Wiley & Sons, Hoboken (2010)

    Book  Google Scholar 

  3. Pohlheim, H., Wegener, J.: Testing the temporal behavior of real-time software modules using extended evolutionary algorithms. In: Proceedings of the 1st Annual Conference on Genetic and Evolutionary Computation - vol. 2, GECCO 1999, p. 1795. Morgan Kaufmann Publishers Inc., San Francisco (1999)

    Google Scholar 

  4. Graydon, P., Bate, I.: Realistic safety cases for the timing of systems. Comput. J. 57(5), 759–774 (2014)

    Article  Google Scholar 

  5. Wilhelm, R., Engblom, J., Ermedahl, A., Holsti, N., Thesing, S., Whalley, D., Bernat, G., Ferdinand, C., Heckmann, R., Mitra, T., Mueller, F., Puaut, I., Puschner, P., Staschulat, J., Stenström, P.: The worst-case execution-time problem - overview of methods and survey of tools. ACM Trans. Embed. Comput. Syst. 7(3), 36:1–36:53 (2008). http://doi.acm.org/10.1145/1347375.1347389

    Article  Google Scholar 

  6. Tracey, N., Clark, J., McDermid, J., Mander, K.: A search-based automated test-data generation framework for safety-critical systems. In: Henderson, P. (ed.) Systems Engineering for Business Process Change: New Directions, pp. 174–213. Springer, New York (2002). https://doi.org/10.1007/978-1-4471-0135-2_12. http://dl.acm.org/citation.cfm?id=763012.763025

    Chapter  Google Scholar 

  7. McMinn, P.: Search-based software testing: past, present and future. In: Proceedings of the 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2011, pp. 153–163. IEEE Computer Society, Washington (2011). http://dx.doi.org/10.1109/ICSTW.2011.100

  8. Poulding, S., Clark, J.A.: The problems with multi-core: Challenges for the critical systems community. Technical report, Department of Computer Science, University of York, York, United Kingdom (2012). version 0.1

    Google Scholar 

  9. Saidi, S., Ernst, R., Uhrig, S., Theiling, H., de Dinechin, B.D.: The shift to multicores in real-time and safety-critical systems. In: 2015 International Conference on Hardware/Software Codesign and System Synthesis (CODES+ISSS), pp. 220–229, October 2015

    Google Scholar 

  10. Boussad, I., Lepagnot, J., Siarry, P.: A survey on optimization metaheuristics. Inf. Sci. 237, 82–117 (2013). http://www.sciencedirect.com/science/article/pii/S0020025513001588

    Article  MathSciNet  MATH  Google Scholar 

  11. Afzal, W., Torkar, R., Feldt, R.: A systematic review of search-based testing for non-functional system properties. Inf. Softw. Technol. 51(6), 957–976 (2009). http://www.sciencedirect.com/science/article/pii/S0950584908001833

    Article  Google Scholar 

  12. Wegener, J., Grimm, K., Grochtmann, M., Sthamer, H., Jones, B.: Systematic testing of real-time systems. In: 4th International Conference on Software Testing Analysis and Review (EuroSTAR 1996) (1996)

    Google Scholar 

  13. Wegener, J., Sthamer, H., Jones, B.F., Eyres, D.E.: Testing real-time systems using genetic algorithms. Softw. Qual. J. 6(2), 127–135 (1997). https://doi.org/10.1023/A:1018551716639

    Article  Google Scholar 

  14. Tracey, N., Clark, J.A., Mander, K.: The way forward for unifying dynamic test-case generation: The optimisation-based approach. In: Proceedings of the IFIP International Workshop on Dependable Computing and Its Applications (DCIA), York (1998)

    Google Scholar 

  15. Alander, J.T., Mantere, T., Moghadampour, G., Matila, J.: Searching protection relay response time extremes using genetic algorithm-software quality by optimization. In: 1997 Fourth International Conference on Advances in Power System Control, Operation and Management, APSCOM-1997, (Conf. Publ. No. 450), vol. 1, pp. 95–99, November 1997

    Google Scholar 

  16. Gross, H.G., Jones, B.F., Eyres, D.E.: Structural performance measure of evolutionary testing applied to worst-case timing of real-time systems. IEE Proc. Softw. 147(2), 25–30 (2000)

    Article  Google Scholar 

  17. Wegener, J., Mueller, F.: A comparison of static analysis and evolutionary testing for the verification of timing constraints. Real-Time Syst. 21(3), 241–268 (2001). https://doi.org/10.1023/A:1011132221066

    Article  MATH  Google Scholar 

  18. Tlili, M., Wappler, S., Sthamer, H.: Improving evolutionary real-time testing. In: Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, GECCO 2006, pp. 1917–1924. ACM, New York (2006). http://doi.acm.org/10.1145/1143997.1144316

  19. O’Sullivan, M., Vössner, S., Wegener, J.: Testing temporal correctness of real-time systems-a new approach using genetic algorithms and cluster analysis. In: Proceedings of the 6th European Conference on Software Testing, Analysis and Review (EuroSTAR 1998), Munich, Germany (1998)

    Google Scholar 

  20. Groß, H.G.: A prediction system for dynamic optimisation-based execution time analysis. In: Proceedings of the First International Workshop on Software Engineering using Metaheuristic Innovative Algorithms (SEMINAL), ISCE 2001, Toronto, Canada (2001)

    Google Scholar 

  21. Harman, M., Jia, Y., Zhang, Y.: Achievements, open problems and challenges for search based software testing. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pp. 1–12, April 2015

    Google Scholar 

  22. Bate, I., Khan, U.: Wcet analysis of modern processors using multi-criteria optimisation. Empirical Softw. Eng. 16(1), 5–28 (2011). https://doi.org/10.1007/s10664-010-9133-9

    Article  Google Scholar 

  23. Burger, D., Austin, T.M.: The simplescalar tool set, version 2.0. SIGARCH Comput. Archit. News. 25(3), 13–25 (1997). http://doi.acm.org/10.1145/268806.268810

    Article  Google Scholar 

  24. Khan, U., Bate, I.: WCET analysis of modern processors using multi-criteria optimisation. In: 2009 1st International Symposium on Search Based Software Engineering, pp. 103–112, May 2009

    Google Scholar 

  25. Burke, E.K., Gendreau, M., Hyde, M., Kendall, G., Ochoa, G., Özcan, E., Qu, R.: Hyper-heuristics: a survey of the state of the art. J. Oper. Res. Soc. 64(12), 1695–1724 (2013). https://doi.org/10.1057/jors.2013.71

    Article  Google Scholar 

  26. Talbi, E.G.: Metaheuristics: From Design to Implementation. Wiley Publishing, Hoboken (2009)

    Book  MATH  Google Scholar 

  27. Luke, S.: Essentials of Metaheuristics, 2nd edn. Lulu (2013). http://cs.gmu.edu/~sean/book/metaheuristics/

  28. Galassi, M., Davies, J., Theiler, J., Gough, B., Jungman, G., Alken, P., Booth, M., Rossi, F., Ulerich, R.: GNU Scientific Library Reference Manual, 3rd edn. Network Theory Limited, UK (2015). The GSL Team

    Google Scholar 

  29. The Apache Software Foundation: Commons math: The apache commons mathematics library (2016). http://commons.apache.org/proper/commons-math/. Accessed 29 Mar 2017

  30. Free Software Foundation Inc: Gsl - gnu scientific library (2017). https://www.gnu.org/software/gsl/. Accessed 21 Oct 2017

  31. Breshears, C.: The Art of Concurrency: A Thread Monkey’s Guide to Writing Parallel Applications. O’Reilly Media Inc., Sebastopol (2009)

    Google Scholar 

  32. IEEE: Standard for information technology-portable operating system interface (posix(r)) base specifications, issue 7. IEEE Std 1003.1, 2016 Edition (incorporates IEEE Std 1003.1-2008, IEEE Std 1003.1-2008/Cor 1–2013, and IEEE Std 1003.1-2008/Cor 2–2016), pp. 1–3957, September 2016

    Google Scholar 

  33. Sparger, J.: COSC 462 - Parallel Programming, Department of Electrical Engineering & Computer Science at the University of Tennessee, Knoxville. http://web.eecs.utk.edu/~jsparger/cosc462/pp/parallelProject3/. Accessed 09 Sept 2017

  34. Weems, B.: CSE 4351: Parallel Processing, Department of Computer Science and Engineering at the University of Texas, Arlington. http://ranger.uta.edu/~weems/NOTES4351/cse4351.html. Accessed 09 Sept 2017

  35. Stough, J.: Strategies for Introducing Parallelism with Python, Department of Computer Science at Washington and Lee University, Lexington. http://home.wlu.edu/~stoughj/SC13/. Accessed 09 Sept 2017

  36. Freescale semiconductor: P4080 Development System User’s Guide. Technical report (2011)

    Google Scholar 

  37. Levy, M., Conte, T.M.: Embedded multicore processors and systems. IEEE Micro 29(3), 7–9 (2009)

    Article  Google Scholar 

  38. White, D.R.: Software review: the ECJ toolkit. Genetic Program. Evol. Mach. 13(1), 65–67 (2012). http://dx.doi.org/10.1007/s10710-011-9148-z

    Article  MathSciNet  Google Scholar 

  39. Kirkpatrick, S., Gelatt, C.D., Vecchi, M.P.: Optimization by simulated annealing. Science 220(4598), 671–680 (1983). http://science.sciencemag.org/content/220/4598/671

    Article  MathSciNet  MATH  Google Scholar 

  40. Abramson, D., Krishnamoorthy, M., Dang, H.: Simulated annealing cooling schedules for the school timetabling problem. Asia Pac. J. Oper. Res. 16 (1998)

    Google Scholar 

Download references

Acknowledgements

Komsan Srivisut’s work is sponsored by the Royal Thai Government. John A. Clark is sponsored by the Engineering and Physical Sciences Research Council (EPSRC) under the Dynamic Adaptive Automated Software Engineering (DAASE) project EP/J017515/1. The authors would also like to thank Nathan Burles (also sponsored by DAASE) for his contribution to the development of execution infrastructure on the P4080.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Komsan Srivisut .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Srivisut, K., Clark, J.A., Paige, R.F. (2018). Search-Based Temporal Testing in an Embedded Multicore Platform. In: Sim, K., Kaufmann, P. (eds) Applications of Evolutionary Computation. EvoApplications 2018. Lecture Notes in Computer Science(), vol 10784. Springer, Cham. https://doi.org/10.1007/978-3-319-77538-8_53

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-77538-8_53

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-77537-1

  • Online ISBN: 978-3-319-77538-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics