skip to main content
10.1145/2597008.2597139acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
Article

Do architectural design decisions improve the understanding of software architecture? two controlled experiments

Published: 02 June 2014 Publication History

Abstract

Architectural design decision (ADD) and its design rationale, as a paradigm shift on documenting and enriching architecture design description, is supposed to facilitate the understanding of architecture and the reasoning behind the design rationale, which consequently improves the architecting process and gets better architecture design results. But the lack of empirical evaluation that supports this statement is one of the major reasons that prevent industrial practitioners from using ADDs in their daily architecting activities. In this paper, we conducted two controlled experiments, as a family of experiments, to investigate how presence of ADDs can improve the understanding of architecture. The main results of our experiments are: (i) using ADDs and their rationale in architecture documentation does not affect the time needed for completing architecture design tasks; (ii) one experiment and the family of experiments achieved a significantly better understanding of architecture design when using ADDs; and (iii) with regard to the correctness of architecture understanding, more experienced participants benefited more from ADDs in comparison with less experienced ones.

References

[1]
Kruchten, P., Lago, P., and van Vliet, H. 2006. Building up and reasoning about architectural knowledge. In Proceedings of the 2nd International Conference on the Quality of Software-Architectures (QoSA), Springer LNCS, 43-58.
[2]
Jansen, A. and Bosch, J. 2005. Software architecture as a set of architectural design decisions. In Proceedings of the 5th Working IEEE/IFIP Conference on Software Architecture (WICSA), IEEE, 109-120.
[3]
Shahin, M., Liang, P., and Mohammad, R. K. 2010. Improving understandability of architecture design through visualization of architectural design decision, In Proceedings of the 5th Workshop on SHAring and Reusing architectural Knowledge (SHARK), ACM, 88-95.
[4]
Liang, P. and Avgeriou, P. 2009. Tools and Technologies for Architecture Knowledge Management. In Software Architecture Knowledge Management: Theory and Practice, Springer, 91-111.
[5]
Shahin, M., Liang, P., and Mohammad, R. K. 2009. Architectural design decision: Existing models and tools. In Proceedings of the 8th Working IEEE/IFIP Conference on Software Architecture (WICSA), IEEE, 293-296.
[6]
Telea, A., Voinea, L., and Sassenburg, H. 2010. Visual tools for software architecture understanding: A stakeholder perspective, IEEE Software, 27(6):46-53.
[7]
Knodel, J., Muthig, D., and Naab, M. 2008. An experiment on the role of graphical elements in architecture visualization, Empirical Software Engineering, 13(6):693-726.
[8]
Haitzer, T. and Zdun, U. 2013. Controlled experiment on the supportive effect of architectural component diagrams for design understanding of novice architects, In Proceedings of the 7th European conference on Software Architecture (ECSA), Springer LNCS, 54-71.
[9]
Ugai, T., Hayashi, S., and Saeki, M. 2010. Visualizing stakeholder concerns with anchored map, In Proceedings of the 5th International Workshop on Requirements Engineering Visualization (REV), IEEE, 20-24.
[10]
Jedlitschka, A. and Pfahl, D. 2005. Reporting guidelines for controlled experiments in software engineering. In Proceedings of the 4th International Symposium on Empirical Software Engineering (ISESE), IEEE, 95-104.
[11]
ISO, 2011. ISO/IEC/IEEE 42010:2011 Systems and software engineering - Architecture description, http://www.iso.org/iso/catalogue_detail.htm?csnumber=5050 8
[12]
Ruxton, G. D. 2006. The unequal variance t-test is an underused alternative to Student's t-test and the Mann-Whitney U test. Behavioral Ecology, 17(4):688-690.
[13]
Tang, A., Tran, M. H., Han, J., and van Vliet, H. 2008. Design reasoning improves software design quality, In Proceeding of the 4th International Conference on Quality of Software-Architectures: Models and Architectures (QoSA), Springer LNCS, 28-42.
[14]
Jansen, A., van der Ven, J., Avgeriou, P., and Hammer, D. K. 2007. Tool support for architectural decisions, In Proceedings of the 6th Working IEEE/IFIP Conference on Software Architecture (WICSA), IEEE, 4-13.
[15]
Shahin, M., Liang, P., and Mohammad, R. K. 2010. Rationale visualization of software architectural design decision using Compendium, In Proceedings of the 25th ACM Symposium on Applied Computing (SAC), ACM, 2358- 2359.
[16]
Mirakhorli, M. 2011. Tracing architecturally significant requirements: A decision-centric approach, In Proceedings of the 33rd International Conference on Software Engineering (ICSE), ACM, 1126-1127.
[17]
Kunz, W. and Rittel, H. 1970. Issues as Elements of Information Systems. Studiengruppe fur Systemforschung.
[18]
Wohlin, C., Host, M., and Henningsson, K. 2003. Empirical research methods in software engineering. Empirical Methods and Studies in Software Engineering, Springer, 145-165.
[19]
Royston, P. 1982. An Extension of Shapiro and Wilk’s W Test for Normality to Large Samples, Applied Statistics, 31:115-124.
[20]
Tang, A. and Lago, P. 2010. Notes on Design Reasoning Techniques (V1.4), SUTICT-TR2010.01, Swinburne University of Technology & VU University Amsterdam.
[21]
Basili, V. R., Shull, F., and Lanubile, F. 1999. Building knowledge through families of experiments, IEEE Transactions on Software Engineering, 25(4):456-473.
[22]
Reggio, G., Ricca, F., Scanniello, G., Di Cerbo, F., and Dodero, G. 2013. On the comprehension of workflows modeled with a precise style: Results from a family of controlled experiments, Journal on Software & Systems Modeling.
[23]
Hadar, I., Reinhartz-Berger, I., Kuflik, T., Perini, A., Ricca, F., and Susi, A. 2013. Comparing the comprehensibility of requirements models expressed in Use Case and Tropos: Results from a family of experiments, Information and Software Technology, 55(10):1823-1843.
[24]
Bass, L., Clements, P., and Kazman, R. 2012. Software Architecture in Practice, 3rd Edition, Addison Wesley.
[25]
Kruchten, P., Lago, P., van Vliet, H., and Wolf, T. 2005. Building up and exploiting architectural knowledge. In Proceedings of the 5th Working IEEE/IFIP Conference on Software Architecture (WICSA), IEEE, 291-292.
[26]
Babar, M. A., Dingsøyr, T., Lago, P., and van Vliet, H. 2009. Software Architecture Knowledge Management: Theory and Practice. Springer-Verlag New York.
[27]
Cruz-Lemus, J. A., Genero, M., Caivano, D., Abrahão, S., Insfrán, E., and Carsí, J. A. 2011. Assessing the influence of stereotypes on the comprehension of UML sequence diagrams: A family of experiments, Information and Software Technology, 53(12):1391-1403.
[28]
Shull, F., Carver, J. C., Vegas, S, and Juristo, N. 2008. The role of replications in empirical software engineering, Empirical Software Engineering, 13(2):211-218.
[29]
Li, Z., Liang, P., and Avgeriou, P. 2013. Application of knowledge-based approaches in software architecture: A systematic mapping study, Information and Software Technology, 55(5):777-794.
[30]
Sjoberg, D. I. K., Hannay, J. E., Hansen, O., Kampenes, V. B., Karahasanovic, A., Liborg, N., and Rekdal, A. C. 2005. A survey of controlled experiments in software engineering, IEEE Transactions on Software Engineering, 31(9):733-753.
[31]
Kitchenham, B., Pfleeger, S., Pickard, L., Jones, P., Hoaglin, D., El Emam, K., and Rosenberg, J. 2002, Preliminary guidelines for empirical research in software engineering, IEEE Transactions on Software Engineering, 28(8):721-734.
[32]
Host, M., Regnell, B., and Wohlin, C. 2000. Using students as subjects - A comparative study of students and professionals in lead-time impact assessment. Empirical Software Engineering, 5(3):201-214.
[33]
Selic, B. 2003. The pragmatics of model-driven development, IEEE Software, 20(5):19-25.
[34]
Rello, L. and Baeza-Yates, R. 2012. Lexical quality as a proxy for web text understandability, In Proceedings of the 21st International Conference Companion on World Wide Web (WWW), ACM, 591-592.
[35]
Lu, S. Y., Cai, J., Burkett, W., and Udwadia, F. 2000. A methodology for collaborative design process and conflict analysis, CIRP Annals-Manufacturing Technology, 49(1):69- 73.
[36]
Girczyc, E. and Carlson, S. 1993. Increasing design quality and engineering productivity through design reuse, In Proceedings of the 30th international Design Automation Conference (DAC), ACM, 48-53.
[37]
Mirakhorli, M. and Cleland-Huang, J. 2011. Transforming trace information in architectural documents into re-usable and effective traceability links, In Proceedings of the 6th Workshop on SHAring and Reusing architectural Knowledge (SHARK), ACM, 45-52.
[38]
Shahin, M., Liang, P., and Li, Z. 2011. Architectural design decision visualization for architecture design: Preliminary results of a controlled experiment, In Proceedings of the 5th European Conference on Software Architecture: Companion Volume (ECSA), ACM, 2-9.
[39]
Pickarda, L. M., Kitchenhama, B. A., and Jonesb, P. W. 1998. Combining empirical results in software engineering, Information and Software Technology, 40(14):811-821.
[40]
Cohen, J. 1988. Statistical Power Analysis for the Behavioral Sciences, 2nd Edition, Lawrence Earlbaum Associates, Hillsdale, NJ.
[41]
Aranda, J., Ernst, N., Horkoff, J., and Easterbrook, S. 2007. A framework for empirical evaluation of model comprehensibility, In Proceeding of Workshop on Modeling in Software Engineering (MiSE), IEEE, 7-13.
[42]
van Heesch, U. and Avgeriou, P. 2010. Naive architecting - Understanding the reasoning process of students - A descriptive survey, In Proceedings of the 4th European conference on Software Architecture (ECSA), Springer LNCS, 24-37.
[43]
Lago, P., Avgeriou, P., Capilla, R., and Kruchten, P. 2008. Wishes and boundaries for a software architecture knowledge community. In Proceedings of the 7th Working IEEE/IFIP Conference on Software Architecture (WICSA), IEEE, 271- 274.
[44]
Hedges, L. V, and Olkin, I. 1985. Statistical Methods for Meta-analysis. Academic Press Orlando.

Cited By

View all
  • (2024)Mining architectural information: A systematic mapping studyEmpirical Software Engineering10.1007/s10664-024-10480-629:4Online publication date: 4-Jun-2024
  • (2022)A Review on Architectural Design and its ProcessSustainable Architecture and Building Materials10.46632/sabm/1/1/11:1(01-07)Online publication date: 1-Apr-2022
  • (2021)Evaluating Model-Driven Development Claims with Respect to Quality: A Family of ExperimentsIEEE Transactions on Software Engineering10.1109/TSE.2018.288470647:1(130-145)Online publication date: 1-Jan-2021
  • Show More Cited By

Index Terms

  1. Do architectural design decisions improve the understanding of software architecture? two controlled experiments

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      ICPC 2014: Proceedings of the 22nd International Conference on Program Comprehension
      June 2014
      325 pages
      ISBN:9781450328791
      DOI:10.1145/2597008
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      In-Cooperation

      • TCSE: IEEE Computer Society's Tech. Council on Software Engin.

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 02 June 2014

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. architectural design decision
      2. controlled experiment
      3. design rationale
      4. meta-analysis
      5. software architecture

      Qualifiers

      • Article

      Conference

      ICSE '14
      Sponsor:

      Upcoming Conference

      ICSE 2025

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)12
      • Downloads (Last 6 weeks)1
      Reflects downloads up to 20 Jan 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Mining architectural information: A systematic mapping studyEmpirical Software Engineering10.1007/s10664-024-10480-629:4Online publication date: 4-Jun-2024
      • (2022)A Review on Architectural Design and its ProcessSustainable Architecture and Building Materials10.46632/sabm/1/1/11:1(01-07)Online publication date: 1-Apr-2022
      • (2021)Evaluating Model-Driven Development Claims with Respect to Quality: A Family of ExperimentsIEEE Transactions on Software Engineering10.1109/TSE.2018.288470647:1(130-145)Online publication date: 1-Jan-2021
      • (2020)The Evolution of Architectural Decision Making as a Key Focus Area of Software Architecture Research: A Semi-Systematic Literature Study2020 IEEE International Conference on Software Architecture (ICSA)10.1109/ICSA47634.2020.00015(69-80)Online publication date: Mar-2020
      • (2020)Replication of Studies in Empirical Software Engineering: A Systematic Mapping Study, From 2013 to 2018IEEE Access10.1109/ACCESS.2019.29521918(26773-26791)Online publication date: 2020
      • (2019)Why do they ask? An exploratory study of crowd discussions about Android application programming interface in stack overflow在 Stack Overflow 平台上关于安卓应用程序接口群体讨论的探索性研究Journal of Central South University10.1007/s11771-019-4185-526:9(2432-2446)Online publication date: 14-Oct-2019
      • (2015)Software Architecture Design Assistants Need Controlled Efficiency ExperimentsProceedings of the 1st International Workshop on Future of Software Architecture Design Assistants10.1145/2751491.2751492(19-24)Online publication date: 6-May-2015

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media