skip to main content
10.1145/3307630.3342400acmotherconferencesArticle/Chapter ViewAbstractPublication PagessplcConference Proceedingsconference-collections
research-article

Answering the Call of the Wild?: Thoughts on the Elusive Quest for Ecological Validity in Variability Modeling

Published:09 September 2019Publication History

ABSTRACT

Ecological validity is a term commonly used in several disciplines to refer to the fact that in a research study, the methods, the materials, and the settings must approximate the real world, i.e. what happens in everyday life. Variability modeling is no exception, it has striven for this form of validity by looking at two main sources, industrial projects and open source projects. Despite their unquestionable value, industrial projects inherently pose limitations; for instance, in terms of open access or results replication, which are two important tenets for any scientific endeavor. In this paper, we present our first findings on the use of open source projects in variability modeling research, and identify trends and avenues for further research.

References

  1. Iago Abal, Claus Brabrand, and Andrzej Wasowski. 2014. 42 variability bugs in the linux kernel: a qualitative analysis. In ACM/IEEE International Conference on Automated Software Engineering, ASE '14, Vasteras, Sweden - September 15 - 19, 2014, Ivica Crnkovic, Marsha Chechik, and Paul Grünbacher (Eds.). ACM, 421--432. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Iago Abal, Jean Melo, Stefan Stanciulescu, Claus Brabrand, Márcio Ribeiro, and Andrzej Wasowski. 2018. Variability Bugs in Highly Configurable Systems: A Qualitative Analysis. ACM Trans. Softw. Eng. Methodol. 26, 3 (2018), 10:1--10:34. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Mathieu Acher, Anthony Cleve, Philippe Collet, Philippe Merle, Laurence Duchien, and Philippe Lahire. 2014. Extraction and evolution of architectural variability models in plugin-based systems. Software and System Modeling 13, 4 (2014), 1367--1394. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Nele Andersen, Krzysztof Czarnecki, Steven She, and Andrzej Wasowski. 2012. Efficient synthesis of feature models. In 16th International Software Product Line Conference, SPLC '12, Salvador, Brazil - September 2--7, 2012, Volume 1. 106--115. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Paolo Arcaini, Angelo Gargantini, and Paolo Vavassori. 2017. Automated Repairing of Variability Models. In Proceedings of the 21st International Systems and Software Product Line Conference, SPLC 2017, Volume A, Sevilla, Spain, September 25--29, 2017. 9--18. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Wesley K. G. Assunção, Roberto E. Lopez-Herrejon, Lukas Linsbauer, Silvia R. Vergilio, and Alexander Egyed. 2017. Multi-objective reverse engineering of variability-safe feature models based on code dependencies of system variants. Empirical Software Engineering 22, 4 (2017), 1763--1794. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. David Benavides, Sergio Segura, and Antonio Ruiz Cortés. 2010. Automated analysis of feature models 20 years later: A literature review. Inf. Syst. 35, 6 (2010), 615--636. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Thorsten Berger, Steven She, Rafael Lotufo, Krzysztof Czarnecki, and Andrzej Wasowski. 2010. Feature-to-Code Mapping in Two Large Product Lines. In Software Product Lines: Going Beyond - 14th International Conference, SPLC 2010, Jeju Island, South Korea, September 13--17, 2010. Proceedings. 498--499. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Thorsten Berger, Steven She, Rafael Lotufo, Andrzej Wasowski, and Krzysztof Czarnecki. 2010. Variability modeling in the real: a perspective from the operating systems domain. In ASE 2010, 25th IEEE/ACM International Conference on Automated Software Engineering, Antwerp, Belgium, September 20--24, 2010, Charles Pecheur, Jamie Andrews, and Elisabetta Di Nitto (Eds.). ACM, 73--82. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Thorsten Berger, Steven She, Rafael Lotufo, Andrzej Wasowski, and Krzysztof Czarnecki. 2013. A Study of Variability Models and Languages in the Systems Software Domain. IEEE Trans. Software Eng. 39, 12 (2013), 1611--1640. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Claus Brabrand, Márcio Ribeiro, Társis Tolêdo, and Paulo Borba. 2012. Intraprocedural dataflow analysis for software product lines. In Proceedings of the 11th International Conference on Aspect-oriented Software Development, AOSD 2012, Potsdam, Germany, March 25--30, 2012, Robert Hirschfeld, Éric Tanter, Kevin J. Sullivan, and Richard P. Gabriel (Eds.). ACM, 13--24. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Andrade Chittaranjan. 2018. Internal, External, and Ecological Validity in Research Design, Conduct, and Evaluation. Indian journal of psychological medicine 40, 5 (2018), 498--499. https://doi.org/Google ScholarGoogle Scholar
  13. Jürgen Cito and Harald C. Gall. 2016. Using docker containers to improve reproducibility in software engineering research. In Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14--22, 2016 - Companion Volume, Laura K. Dillon, Willem Visser, and Laurie Williams (Eds.). ACM, 906--907. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Robertas Damasevicius, Paulius Paskevicius, Eimutis Karciauskas, and Romas Marcinkevicius. 2012. Automatic Extraction of Features and Generation of Feature Models from Java Programs. ITC 41, 4 (2012), 376--384.Google ScholarGoogle ScholarCross RefCross Ref
  15. Nicolas Dintzner, Arie van Deursen, and Martin Pinzger. 2014. Extracting feature model changes from the Linux kernel using FMDiff. In The Eighth International Workshop on Variability Modelling of Software-intensive Systems, VaMoS '14, Sophia Antipolis, France, January 22--24, 2014. 22:1--22:8. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Nicolas Dintzner, Arie van Deursen, and Martin Pinzger. 2017. Analysing the Linux kernel feature model changes using FMDiff. Software and System Modeling 16, 1 (2017), 55--76. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Sascha El-Sharkawy, Adam Krafczyk, and Klaus Schmid. 2015. Analysing the Kconfig semantics and its analysis tools. In Proceedings of the 2015 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences, GPCE 2015, Pittsburgh, PA, USA, October 26--27, 2015, Christian Kästner and Aniruddha S. Gokhale (Eds.). ACM, 45--54. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Sascha El-Sharkawy, Adam Krafczyk, and Klaus Schmid. 2017. An Empirical Study of Configuration Mismatches in Linux. In Proceedings of the 21st International Systems and Software Product Line Conference, SPLC 2017, Volume A, Sevilla, Spain, September 25--29, 2017. 19--28. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. José A. Galindo, David Benavides, Pablo Trinidad, Antonio Manuel Gutiérrez-Fernández, and Antonio Ruiz-Cortés. 2019. Automated analysis of feature models: Quo vadis? Computing 101, 5 (2019), 387--433. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Christopher Henard, Mike Papadakis, Mark Harman, and Yves Le Traon. 2015. Combining Multi-Objective Search and Constraint Solving for Configuring Large Software Product Lines. In 37th IEEE/ACM International Conference on Software Engineering, ICSE 2015, Florence, Italy, May 16--24, 2015, Volume 1. 517--528. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, Patrick Heymans, and Yves Le Traon. 2014. Bypassing the Combinatorial Explosion: Using Similarity to Generate and Prioritize T-Wise Test Configurations for Software Product Lines. IEEE Trans. Software Eng. 40, 7 (2014), 650--670. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, and Yves Le Traon. 2013. Assessing Software Product Line Testing Via Model-Based Mutation: An Application to Similarity Testing. In Sixth IEEE International Conference on Software Testing, Verification and Validation, ICST 2013 Workshops Proceedings, Luxembourg, Luxembourg, March 18--22, 2013. 188--197. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, and Yves Le Traon. 2013. Towards automated testing and fixing of re-engineered feature models. In 35th International Conference on Software Engineering, ICSE '13, San Francisco, CA, USA, May 18--26, 2013. 1245--1248. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Ruben Heradio, Hector Perez-Morago, David Fernández-Amorós, Francisco Javier Cabrerizo, and Enrique Herrera-Viedma. 2016. A bibliometric analysis of 20 years of research on software product lines. Information & Software Technology 72 (2016), 1--15. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Robert M. Hierons, Miqing Li, Xiaohui Liu, Sergio Segura, and Wei Zheng. 2016. SIP: Optimal Product Selection from Feature Models Using Many-Objective Evolutionary Optimization. ACM Trans. Softw. Eng. Methodol. 25, 2 (2016), 17:1--17:39. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Martin Fagereng Johansen, Øystein Haugen, and Franck Fleurey. 2012. An algorithm for generating t-wise covering arrays from large feature models. In 16th International Software Product Line Conference, SPLC '12, Salvador, Brazil - September 2--7, 2012, Volume 1. 46--55. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M. Germán, and Daniela E. Damian. 2016. An in-depth study of the promises and perils of mining GitHub. Empirical Software Engineering 21, 5 (2016), 2035--2071. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Alexander Knüppel, Thomas Thüm, Stephan Mennicke, Jens Meinicke, and Ina Schaefer. 2017. Is there a mismatch between real-world feature models and product-line research?. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4--8, 2017, Eric Bodden, Wilhelm Schäfer, Arie van Deursen, and Andrea Zisman (Eds.). ACM, 291--302. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Jia Hui (Jimmy) Liang, Vijay Ganesh, Krzysztof Czarnecki, and Venkatesh Raman. 2015. SAT-based analysis of large real-world feature models is easy. In Proceedings of the 19th International Conference on Software Product Line, SPLC 2015, Nashville, TN, USA, July 20--24, 2015, Douglas C. Schmidt (Ed.). ACM, 91--100. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Jörg Liebig, Alexander von Rhein, Christian Kästner, Sven Apel, Jens Dörre, and Christian Lengauer. 2013. Scalable analysis of variable software. In Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE'13, Saint Petersburg, Russian Federation, August 18--26, 2013, Bertrand Meyer, Luciano Baresi, and Mira Mezini (Eds.). ACM, 81--91. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Lukas Linsbauer, Roberto Erick Lopez-Herrejon, and Alexander Egyed. 2017. Variability extraction and modeling for product variants. Software and System Modeling 16, 4 (2017), 1179--1199. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Roberto Erick Lopez-Herrejon, Sheny Illescas, and Alexander Egyed. 2018. A systematic mapping study of information visualization for software product line engineering. Journal of Software: Evolution and Process 30, 2 (2018).Google ScholarGoogle ScholarCross RefCross Ref
  33. Roberto E. Lopez-Herrejon, Lukas Linsbauer, and Alexander Egyed. 2015. A systematic mapping study of search-based software engineering for software product lines. Information & Software Technology 61 (2015), 33--51. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Roberto E. Lopez-Herrejon, Leticia Montalvillo-Mendizabal, and Alexander Egyed. 2011. From Requirements to Features: An Exploratory Study of Feature-Oriented Refactoring. In Software Product Lines - 15th International Conference, SPLC 2011, Munich, Germany, August 22--26, 2011, Eduardo Santana de Almeida, Tomoji Kishi, Christa Schwanninger, Isabel John, and Klaus Schmid (Eds.). IEEE Computer Society, 181--190. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Rafael Lotufo, Steven She, Thorsten Berger, Krzysztof Czarnecki, and Andrzej Wasowski. 2010. Evolution of the Linux Kernel Variability Model. In Software Product Lines: Going Beyond - 14th International Conference, SPLC 2010, Jeju Island, South Korea, September 13--17, 2010. Proceedings. 136--150. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Sarah Nadi, Thorsten Berger, Christian Kästner, and Krzysztof Czarnecki. 2014. Mining configuration constraints: static analyses and empirical results. In 36th International Conference on Software Engineering, ICSE '14, Hyderabad, India - May 31 -- June 07, 2014, Pankaj Jalote, Lionel C. Briand, and André van der Hoek (Eds.). ACM, 140--151. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Sarah Nadi, Thorsten Berger, Christian Kästner, and Krzysztof Czarnecki. 2015. Where Do Configuration Constraints Stem From? An Extraction Approach and an Empirical Study. IEEE Trans. Software Eng. 41, 8 (2015), 820--841.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Ganesh Khandu Narwane, José A. Galindo, Shankara Narayanan Krishna, David Benavides, Jean-Vivien Millo, and S. Ramesh. 2016. Traceability Analyses between Features and Assets in Software Product Lines. Entropy 18, 8 (2016), 269.Google ScholarGoogle ScholarCross RefCross Ref
  39. José Antonio Parejo, Ana B. Sánchez, Sergio Segura, Antonio Ruiz Cortés, Roberto E. Lopez-Herrejon, and Alexander Egyed. 2016. Multi-objective test case prioritization in highly configurable systems: A case study. Journal of Systems and Software 122 (2016), 287--310. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Hector Perez-Morago, Ruben Heradio, David Fernández-Amorós, Roberto Bean, and Carlos Cerrada. 2015. Efficient Identification of Core and Dead Features in Variability Models. IEEE Access 3 (2015), 2333--2340.Google ScholarGoogle ScholarCross RefCross Ref
  41. Andreas Pleuss and Goetz Botterweck. 2012. Visualization of variability and configuration options. STTT 14, 5 (2012), 497--510. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Andreas Pleuss, Goetz Botterweck, Deepak Dhungana, Andreas Polzer, and Stefan Kowalewski. 2012. Model-driven support for product line evolution on feature level. Journal of Systems and Software 85, 10 (2012), 2261--2274. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. K. Pohl, G. Bockle, and F. J. van der Linden. 2005. Software Product Line Engineering: Foundations, Principles and Techniques. Springer. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Rick Rabiser, Klaus Schmid, Martin Becker, Goetz Botterweck, Matthias Galster, Iris Groher, and Danny Weyns. 2018. A study and comparison of industrial vs. academic software product line research published at SPLC. In Proceeedings of the 22nd International Systems and Software Product Line Conference - Volume 1, SPLC 2018, Gothenburg, Sweden, September 10--14, 2018. 14--24. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Ana B. Sánchez, Sergio Segura, and Antonio Ruiz Cortés. 2014. The Drupal framework: a case study to evaluate variability testing techniques. In The Eighth International Workshop on Variability Modelling of Software-intensive Systems, VaMoS '14, Sophia Antipolis, France, January 22--24, 2014.11:1--11:8. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Ana B. Sánchez, Sergio Segura, José Antonio Parejo, and Antonio Ruiz Cortés. 2017. Variability testing in the wild: the Drupal case study. Software and System Modeling 16, 1 (2017), 173--194. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Abdel Salam Sayyad, Joseph Ingram, Tim Menzies, and Hany Ammar. 2013. Scalable product line configuration: A straw to break the camel's back. In 2013 28th IEEE/ACM International Conference on Automated Software Engineering, ASE 2013, Silicon Valley, CA, USA, November 11--15, 2013, Ewen Denney, Tevfik Bultan, and Andreas Zeller (Eds.). IEEE, 465--474. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Steven She, Rafael Lotufo, Thorsten Berger, Andrzej Wasowski, and Krzysztof Czarnecki. 2010. The Variability Model of The Linux Kernel. In Fourth International Workshop on Variability Modelling of Software-Intensive Systems, Linz, Austria, January 27--29, 2010. Proceedings (ICB-Research Report), David Benavides, Don S. Batory, and Paul Grünbacher (Eds.), Vol. 37. Universität Duisburg-Essen, 45--51. http://www.vamos-workshop.net/proceedings/VaMoS_2010_Proceedings.pdfGoogle ScholarGoogle Scholar
  49. Steven She, Rafael Lotufo, Thorsten Berger, Andrzej Wasowski, and Krzysztof Czarnecki. 2011. Reverse engineering feature models. In Proceedings of the 33rd International Conference on Software Engineering, ICSE 2011, Waikiki, Honolulu, HI, USA, May 21--28, 2011, Richard N. Taylor, Harald C. Gall, and Nenad Medvidovic (Eds.). ACM, 461--470. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Norbert Siegmund, Sergiy S. Kolesnikov, Christian Kästner, Sven Apel, Don S. Batory, Marko Rosenmüller, and Gunter Saake. 2012. Predicting performance via automated feature-interaction detection. In 34th International Conference on Software Engineering, ICSE 2012, June 2--9, 2012, Zurich, Switzerland, Martin Glinz, Gail C. Murphy, and Mauro Pezzè (Eds.). IEEE Computer Society, 167--177. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Xhevahire Tërnava and Philippe Collet. 2017. Early Consistency Checking between Specification and Implementation Variabilities. In Proceedings of the 21st International Systems and Software Product Line Conference, SPLC 2017, Volume A, Sevilla, Spain, September 25--29, 2017. 29--38. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Alexander von Rhein, Alexander Grebhahn, Sven Apel, Norbert Siegmund, Dirk Beyer, and Thorsten Berger. 2015. Presence-Condition Simplification in Highly Configurable Systems. In 37th IEEE/ACM International Conference on Software Engineering, ICSE 2015, Florence, Italy, May 16--24, 2015, Volume 1. 178--188. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, and Björn Regnell. 2012. Experimentation in Software Engineering. Springer. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Yinxing Xue, Jinghui Zhong, Tian Huat Tan, Yang Liu, Wentong Cai, Manman Chen, and Jun Sun. 2016. IBED: Combining IBEA and DE for optimal feature selection in software product line engineering. Appl. Soft Comput. 49 (2016), 1215--1231. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Tewfik Ziadi, Luz Frias, Marcos Aurélio Almeida da Silva, and Mikal Ziane. 2012. Feature Identification from the Source Code of Product Variants. In 16th European Conference on Software Maintenance and Reengineering, CSMR 2012, Szeged, Hungary, March 27--30, 2012. 417--422. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Answering the Call of the Wild?: Thoughts on the Elusive Quest for Ecological Validity in Variability Modeling

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in
    • Published in

      cover image ACM Other conferences
      SPLC '19: Proceedings of the 23rd International Systems and Software Product Line Conference - Volume B
      September 2019
      252 pages
      ISBN:9781450366687
      DOI:10.1145/3307630

      Copyright © 2019 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 9 September 2019

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article

      Acceptance Rates

      Overall Acceptance Rate167of463submissions,36%

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader