ABSTRACT
Ecological validity is a term commonly used in several disciplines to refer to the fact that in a research study, the methods, the materials, and the settings must approximate the real world, i.e. what happens in everyday life. Variability modeling is no exception, it has striven for this form of validity by looking at two main sources, industrial projects and open source projects. Despite their unquestionable value, industrial projects inherently pose limitations; for instance, in terms of open access or results replication, which are two important tenets for any scientific endeavor. In this paper, we present our first findings on the use of open source projects in variability modeling research, and identify trends and avenues for further research.
- Iago Abal, Claus Brabrand, and Andrzej Wasowski. 2014. 42 variability bugs in the linux kernel: a qualitative analysis. In ACM/IEEE International Conference on Automated Software Engineering, ASE '14, Vasteras, Sweden - September 15 - 19, 2014, Ivica Crnkovic, Marsha Chechik, and Paul Grünbacher (Eds.). ACM, 421--432. Google ScholarDigital Library
- Iago Abal, Jean Melo, Stefan Stanciulescu, Claus Brabrand, Márcio Ribeiro, and Andrzej Wasowski. 2018. Variability Bugs in Highly Configurable Systems: A Qualitative Analysis. ACM Trans. Softw. Eng. Methodol. 26, 3 (2018), 10:1--10:34. Google ScholarDigital Library
- Mathieu Acher, Anthony Cleve, Philippe Collet, Philippe Merle, Laurence Duchien, and Philippe Lahire. 2014. Extraction and evolution of architectural variability models in plugin-based systems. Software and System Modeling 13, 4 (2014), 1367--1394. Google ScholarDigital Library
- Nele Andersen, Krzysztof Czarnecki, Steven She, and Andrzej Wasowski. 2012. Efficient synthesis of feature models. In 16th International Software Product Line Conference, SPLC '12, Salvador, Brazil - September 2--7, 2012, Volume 1. 106--115. Google ScholarDigital Library
- Paolo Arcaini, Angelo Gargantini, and Paolo Vavassori. 2017. Automated Repairing of Variability Models. In Proceedings of the 21st International Systems and Software Product Line Conference, SPLC 2017, Volume A, Sevilla, Spain, September 25--29, 2017. 9--18. Google ScholarDigital Library
- Wesley K. G. Assunção, Roberto E. Lopez-Herrejon, Lukas Linsbauer, Silvia R. Vergilio, and Alexander Egyed. 2017. Multi-objective reverse engineering of variability-safe feature models based on code dependencies of system variants. Empirical Software Engineering 22, 4 (2017), 1763--1794. Google ScholarDigital Library
- David Benavides, Sergio Segura, and Antonio Ruiz Cortés. 2010. Automated analysis of feature models 20 years later: A literature review. Inf. Syst. 35, 6 (2010), 615--636. Google ScholarDigital Library
- Thorsten Berger, Steven She, Rafael Lotufo, Krzysztof Czarnecki, and Andrzej Wasowski. 2010. Feature-to-Code Mapping in Two Large Product Lines. In Software Product Lines: Going Beyond - 14th International Conference, SPLC 2010, Jeju Island, South Korea, September 13--17, 2010. Proceedings. 498--499. Google ScholarDigital Library
- Thorsten Berger, Steven She, Rafael Lotufo, Andrzej Wasowski, and Krzysztof Czarnecki. 2010. Variability modeling in the real: a perspective from the operating systems domain. In ASE 2010, 25th IEEE/ACM International Conference on Automated Software Engineering, Antwerp, Belgium, September 20--24, 2010, Charles Pecheur, Jamie Andrews, and Elisabetta Di Nitto (Eds.). ACM, 73--82. Google ScholarDigital Library
- Thorsten Berger, Steven She, Rafael Lotufo, Andrzej Wasowski, and Krzysztof Czarnecki. 2013. A Study of Variability Models and Languages in the Systems Software Domain. IEEE Trans. Software Eng. 39, 12 (2013), 1611--1640. Google ScholarDigital Library
- Claus Brabrand, Márcio Ribeiro, Társis Tolêdo, and Paulo Borba. 2012. Intraprocedural dataflow analysis for software product lines. In Proceedings of the 11th International Conference on Aspect-oriented Software Development, AOSD 2012, Potsdam, Germany, March 25--30, 2012, Robert Hirschfeld, Éric Tanter, Kevin J. Sullivan, and Richard P. Gabriel (Eds.). ACM, 13--24. Google ScholarDigital Library
- Andrade Chittaranjan. 2018. Internal, External, and Ecological Validity in Research Design, Conduct, and Evaluation. Indian journal of psychological medicine 40, 5 (2018), 498--499. https://doi.org/Google Scholar
- Jürgen Cito and Harald C. Gall. 2016. Using docker containers to improve reproducibility in software engineering research. In Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14--22, 2016 - Companion Volume, Laura K. Dillon, Willem Visser, and Laurie Williams (Eds.). ACM, 906--907. Google ScholarDigital Library
- Robertas Damasevicius, Paulius Paskevicius, Eimutis Karciauskas, and Romas Marcinkevicius. 2012. Automatic Extraction of Features and Generation of Feature Models from Java Programs. ITC 41, 4 (2012), 376--384.Google ScholarCross Ref
- Nicolas Dintzner, Arie van Deursen, and Martin Pinzger. 2014. Extracting feature model changes from the Linux kernel using FMDiff. In The Eighth International Workshop on Variability Modelling of Software-intensive Systems, VaMoS '14, Sophia Antipolis, France, January 22--24, 2014. 22:1--22:8. Google ScholarDigital Library
- Nicolas Dintzner, Arie van Deursen, and Martin Pinzger. 2017. Analysing the Linux kernel feature model changes using FMDiff. Software and System Modeling 16, 1 (2017), 55--76. Google ScholarDigital Library
- Sascha El-Sharkawy, Adam Krafczyk, and Klaus Schmid. 2015. Analysing the Kconfig semantics and its analysis tools. In Proceedings of the 2015 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences, GPCE 2015, Pittsburgh, PA, USA, October 26--27, 2015, Christian Kästner and Aniruddha S. Gokhale (Eds.). ACM, 45--54. Google ScholarDigital Library
- Sascha El-Sharkawy, Adam Krafczyk, and Klaus Schmid. 2017. An Empirical Study of Configuration Mismatches in Linux. In Proceedings of the 21st International Systems and Software Product Line Conference, SPLC 2017, Volume A, Sevilla, Spain, September 25--29, 2017. 19--28. Google ScholarDigital Library
- José A. Galindo, David Benavides, Pablo Trinidad, Antonio Manuel Gutiérrez-Fernández, and Antonio Ruiz-Cortés. 2019. Automated analysis of feature models: Quo vadis? Computing 101, 5 (2019), 387--433. Google ScholarDigital Library
- Christopher Henard, Mike Papadakis, Mark Harman, and Yves Le Traon. 2015. Combining Multi-Objective Search and Constraint Solving for Configuring Large Software Product Lines. In 37th IEEE/ACM International Conference on Software Engineering, ICSE 2015, Florence, Italy, May 16--24, 2015, Volume 1. 517--528. Google ScholarDigital Library
- Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, Patrick Heymans, and Yves Le Traon. 2014. Bypassing the Combinatorial Explosion: Using Similarity to Generate and Prioritize T-Wise Test Configurations for Software Product Lines. IEEE Trans. Software Eng. 40, 7 (2014), 650--670. Google ScholarDigital Library
- Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, and Yves Le Traon. 2013. Assessing Software Product Line Testing Via Model-Based Mutation: An Application to Similarity Testing. In Sixth IEEE International Conference on Software Testing, Verification and Validation, ICST 2013 Workshops Proceedings, Luxembourg, Luxembourg, March 18--22, 2013. 188--197. Google ScholarDigital Library
- Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, and Yves Le Traon. 2013. Towards automated testing and fixing of re-engineered feature models. In 35th International Conference on Software Engineering, ICSE '13, San Francisco, CA, USA, May 18--26, 2013. 1245--1248. Google ScholarDigital Library
- Ruben Heradio, Hector Perez-Morago, David Fernández-Amorós, Francisco Javier Cabrerizo, and Enrique Herrera-Viedma. 2016. A bibliometric analysis of 20 years of research on software product lines. Information & Software Technology 72 (2016), 1--15. Google ScholarDigital Library
- Robert M. Hierons, Miqing Li, Xiaohui Liu, Sergio Segura, and Wei Zheng. 2016. SIP: Optimal Product Selection from Feature Models Using Many-Objective Evolutionary Optimization. ACM Trans. Softw. Eng. Methodol. 25, 2 (2016), 17:1--17:39. Google ScholarDigital Library
- Martin Fagereng Johansen, Øystein Haugen, and Franck Fleurey. 2012. An algorithm for generating t-wise covering arrays from large feature models. In 16th International Software Product Line Conference, SPLC '12, Salvador, Brazil - September 2--7, 2012, Volume 1. 46--55. Google ScholarDigital Library
- Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M. Germán, and Daniela E. Damian. 2016. An in-depth study of the promises and perils of mining GitHub. Empirical Software Engineering 21, 5 (2016), 2035--2071. Google ScholarDigital Library
- Alexander Knüppel, Thomas Thüm, Stephan Mennicke, Jens Meinicke, and Ina Schaefer. 2017. Is there a mismatch between real-world feature models and product-line research?. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4--8, 2017, Eric Bodden, Wilhelm Schäfer, Arie van Deursen, and Andrea Zisman (Eds.). ACM, 291--302. Google ScholarDigital Library
- Jia Hui (Jimmy) Liang, Vijay Ganesh, Krzysztof Czarnecki, and Venkatesh Raman. 2015. SAT-based analysis of large real-world feature models is easy. In Proceedings of the 19th International Conference on Software Product Line, SPLC 2015, Nashville, TN, USA, July 20--24, 2015, Douglas C. Schmidt (Ed.). ACM, 91--100. Google ScholarDigital Library
- Jörg Liebig, Alexander von Rhein, Christian Kästner, Sven Apel, Jens Dörre, and Christian Lengauer. 2013. Scalable analysis of variable software. In Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE'13, Saint Petersburg, Russian Federation, August 18--26, 2013, Bertrand Meyer, Luciano Baresi, and Mira Mezini (Eds.). ACM, 81--91. Google ScholarDigital Library
- Lukas Linsbauer, Roberto Erick Lopez-Herrejon, and Alexander Egyed. 2017. Variability extraction and modeling for product variants. Software and System Modeling 16, 4 (2017), 1179--1199. Google ScholarDigital Library
- Roberto Erick Lopez-Herrejon, Sheny Illescas, and Alexander Egyed. 2018. A systematic mapping study of information visualization for software product line engineering. Journal of Software: Evolution and Process 30, 2 (2018).Google ScholarCross Ref
- Roberto E. Lopez-Herrejon, Lukas Linsbauer, and Alexander Egyed. 2015. A systematic mapping study of search-based software engineering for software product lines. Information & Software Technology 61 (2015), 33--51. Google ScholarDigital Library
- Roberto E. Lopez-Herrejon, Leticia Montalvillo-Mendizabal, and Alexander Egyed. 2011. From Requirements to Features: An Exploratory Study of Feature-Oriented Refactoring. In Software Product Lines - 15th International Conference, SPLC 2011, Munich, Germany, August 22--26, 2011, Eduardo Santana de Almeida, Tomoji Kishi, Christa Schwanninger, Isabel John, and Klaus Schmid (Eds.). IEEE Computer Society, 181--190. Google ScholarDigital Library
- Rafael Lotufo, Steven She, Thorsten Berger, Krzysztof Czarnecki, and Andrzej Wasowski. 2010. Evolution of the Linux Kernel Variability Model. In Software Product Lines: Going Beyond - 14th International Conference, SPLC 2010, Jeju Island, South Korea, September 13--17, 2010. Proceedings. 136--150. Google ScholarDigital Library
- Sarah Nadi, Thorsten Berger, Christian Kästner, and Krzysztof Czarnecki. 2014. Mining configuration constraints: static analyses and empirical results. In 36th International Conference on Software Engineering, ICSE '14, Hyderabad, India - May 31 -- June 07, 2014, Pankaj Jalote, Lionel C. Briand, and André van der Hoek (Eds.). ACM, 140--151. Google ScholarDigital Library
- Sarah Nadi, Thorsten Berger, Christian Kästner, and Krzysztof Czarnecki. 2015. Where Do Configuration Constraints Stem From? An Extraction Approach and an Empirical Study. IEEE Trans. Software Eng. 41, 8 (2015), 820--841.Google ScholarDigital Library
- Ganesh Khandu Narwane, José A. Galindo, Shankara Narayanan Krishna, David Benavides, Jean-Vivien Millo, and S. Ramesh. 2016. Traceability Analyses between Features and Assets in Software Product Lines. Entropy 18, 8 (2016), 269.Google ScholarCross Ref
- José Antonio Parejo, Ana B. Sánchez, Sergio Segura, Antonio Ruiz Cortés, Roberto E. Lopez-Herrejon, and Alexander Egyed. 2016. Multi-objective test case prioritization in highly configurable systems: A case study. Journal of Systems and Software 122 (2016), 287--310. Google ScholarDigital Library
- Hector Perez-Morago, Ruben Heradio, David Fernández-Amorós, Roberto Bean, and Carlos Cerrada. 2015. Efficient Identification of Core and Dead Features in Variability Models. IEEE Access 3 (2015), 2333--2340.Google ScholarCross Ref
- Andreas Pleuss and Goetz Botterweck. 2012. Visualization of variability and configuration options. STTT 14, 5 (2012), 497--510. Google ScholarDigital Library
- Andreas Pleuss, Goetz Botterweck, Deepak Dhungana, Andreas Polzer, and Stefan Kowalewski. 2012. Model-driven support for product line evolution on feature level. Journal of Systems and Software 85, 10 (2012), 2261--2274. Google ScholarDigital Library
- K. Pohl, G. Bockle, and F. J. van der Linden. 2005. Software Product Line Engineering: Foundations, Principles and Techniques. Springer. Google ScholarDigital Library
- Rick Rabiser, Klaus Schmid, Martin Becker, Goetz Botterweck, Matthias Galster, Iris Groher, and Danny Weyns. 2018. A study and comparison of industrial vs. academic software product line research published at SPLC. In Proceeedings of the 22nd International Systems and Software Product Line Conference - Volume 1, SPLC 2018, Gothenburg, Sweden, September 10--14, 2018. 14--24. Google ScholarDigital Library
- Ana B. Sánchez, Sergio Segura, and Antonio Ruiz Cortés. 2014. The Drupal framework: a case study to evaluate variability testing techniques. In The Eighth International Workshop on Variability Modelling of Software-intensive Systems, VaMoS '14, Sophia Antipolis, France, January 22--24, 2014.11:1--11:8. Google ScholarDigital Library
- Ana B. Sánchez, Sergio Segura, José Antonio Parejo, and Antonio Ruiz Cortés. 2017. Variability testing in the wild: the Drupal case study. Software and System Modeling 16, 1 (2017), 173--194. Google ScholarDigital Library
- Abdel Salam Sayyad, Joseph Ingram, Tim Menzies, and Hany Ammar. 2013. Scalable product line configuration: A straw to break the camel's back. In 2013 28th IEEE/ACM International Conference on Automated Software Engineering, ASE 2013, Silicon Valley, CA, USA, November 11--15, 2013, Ewen Denney, Tevfik Bultan, and Andreas Zeller (Eds.). IEEE, 465--474. Google ScholarDigital Library
- Steven She, Rafael Lotufo, Thorsten Berger, Andrzej Wasowski, and Krzysztof Czarnecki. 2010. The Variability Model of The Linux Kernel. In Fourth International Workshop on Variability Modelling of Software-Intensive Systems, Linz, Austria, January 27--29, 2010. Proceedings (ICB-Research Report), David Benavides, Don S. Batory, and Paul Grünbacher (Eds.), Vol. 37. Universität Duisburg-Essen, 45--51. http://www.vamos-workshop.net/proceedings/VaMoS_2010_Proceedings.pdfGoogle Scholar
- Steven She, Rafael Lotufo, Thorsten Berger, Andrzej Wasowski, and Krzysztof Czarnecki. 2011. Reverse engineering feature models. In Proceedings of the 33rd International Conference on Software Engineering, ICSE 2011, Waikiki, Honolulu, HI, USA, May 21--28, 2011, Richard N. Taylor, Harald C. Gall, and Nenad Medvidovic (Eds.). ACM, 461--470. Google ScholarDigital Library
- Norbert Siegmund, Sergiy S. Kolesnikov, Christian Kästner, Sven Apel, Don S. Batory, Marko Rosenmüller, and Gunter Saake. 2012. Predicting performance via automated feature-interaction detection. In 34th International Conference on Software Engineering, ICSE 2012, June 2--9, 2012, Zurich, Switzerland, Martin Glinz, Gail C. Murphy, and Mauro Pezzè (Eds.). IEEE Computer Society, 167--177. Google ScholarDigital Library
- Xhevahire Tërnava and Philippe Collet. 2017. Early Consistency Checking between Specification and Implementation Variabilities. In Proceedings of the 21st International Systems and Software Product Line Conference, SPLC 2017, Volume A, Sevilla, Spain, September 25--29, 2017. 29--38. Google ScholarDigital Library
- Alexander von Rhein, Alexander Grebhahn, Sven Apel, Norbert Siegmund, Dirk Beyer, and Thorsten Berger. 2015. Presence-Condition Simplification in Highly Configurable Systems. In 37th IEEE/ACM International Conference on Software Engineering, ICSE 2015, Florence, Italy, May 16--24, 2015, Volume 1. 178--188. Google ScholarDigital Library
- Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, and Björn Regnell. 2012. Experimentation in Software Engineering. Springer. Google ScholarDigital Library
- Yinxing Xue, Jinghui Zhong, Tian Huat Tan, Yang Liu, Wentong Cai, Manman Chen, and Jun Sun. 2016. IBED: Combining IBEA and DE for optimal feature selection in software product line engineering. Appl. Soft Comput. 49 (2016), 1215--1231. Google ScholarDigital Library
- Tewfik Ziadi, Luz Frias, Marcos Aurélio Almeida da Silva, and Mikal Ziane. 2012. Feature Identification from the Source Code of Product Variants. In 16th European Conference on Software Maintenance and Reengineering, CSMR 2012, Szeged, Hungary, March 27--30, 2012. 417--422. Google ScholarDigital Library
Index Terms
Answering the Call of the Wild?: Thoughts on the Elusive Quest for Ecological Validity in Variability Modeling
Recommendations
Using Open Source Projects in Higher Education: A Two-Way Certification Framework
Revised Selected Papers of the SEFM 2012 Satellite Events on Information Technology and Open Source: Applications for Education, Innovation, and Sustainability - Volume 7991The paper proposes a framework for the certification of free/libre open source software FLOSS projects. The process is two-fold and is based on our experience over the last four years of using FLOSS for teaching software engineering in Aristotle ...
Creating and growing healthy community open source projects
PLoP '20: Proceedings of the 27th Conference on Pattern Languages of ProgramsThis article presents a succinct and minimal handbook of best practices of how to create and grow community open source projects. We start with the assumption that the handbook's user has a minimal but useful piece of software at hand that they want to ...
MetricHaven: More than 23,000 Metrics for Measuring Quality Attributes of Software Product Lines
SPLC '19: Proceedings of the 23rd International Systems and Software Product Line Conference - Volume BVariability-aware metrics are designed to measure qualitative aspects of software product lines. As we identified in a prior SLR [6], there exist already many metrics that address code or variability separately, while the combination of both has been ...
Comments