Article

Experiments on quality evaluation of embedded software in Japan robot software design contest

Authors:

Hironori Washizaki,

Yasuhide Kobayashi,

Hiroyuki Watanabe,

Kenji Hiranabe,

Kazuya FukudaAuthors Info & Claims

ICSE '06: Proceedings of the 28th international conference on Software engineering

Pages 551 - 560

https://doi.org/10.1145/1134285.1134363

Published: 28 May 2006 Publication History

Abstract

As a practical opportunity for educating Japanese young developers in the field of embedded software development, a software design contest involving the design of software to automatically control a line-trace robot, and conduct running performance tests was held. In this paper,we give the results of the contest from the viewpoint of software quality evaluation. We create a framework for evaluating the software quality which integrated design model quality and the final system performance, and conduct analysis using the framework. As a result of analysis,it is found that the quantitative measurement of the structural complexity of the design models bears a strong relationship to qualitative evaluation of the design conducted by judges. It is also found that there is no strong correlation between design model quality evaluated by the judges and the final system performance. For embedded software development, it is particularly important to estimate and verify reliability and performance in the early stages,using the model. Based on the analysis result,we consider possible remedies with respect to the models submitted,the evaluation methods used and the contest specifications. In order to adequately measure several non-functional quality characteristics including performance on the model,it is necessary to improve the way of developing robot software (such as applying model driven development)and reexamine the evaluation methods.

References

[1]

Japan System House Association: Embedded Technology (ET) Software Design Robot Contest 2005, http: //www.etrobo. jp/

[2]

The LEGO Group, LEGO Mindstorms, http: //mindstorms. lego. com/

[3]

B. W. Boehm: A Spiral Model of Software Development and Enhancement, Computer, Vol. 21, No. 5, 1988.

Digital Library

[4]

OMG: UML 2. 0 Specification, 2004, http: //www.uml. org/

[5]

Y. Kobayashi, T. Yamazaki, T. Futagami and H. Watanabe: Robot Contest as Software Engineering Education, Proc. IPSJ/SIGSE Embedded Software Symposium (ESS2004), 2004. (in Japanese)

[6]

D. West: Metaphor, Architecture, and XP, Proc. 3rd International Conference on Extreme Programming and Agile Processes in Software Engineering, 2002.

[7]

J. Herbsleb, D. Root and J. Tomayko: The eXtreme Programming (XP) Metaphor and Software Architecture, CMU-CS-03-167, Carnegie Mellon University, 2003.

[8]

OMG: Software Process Engineering Metamodel Specification, Version 1. 1, 2005.

[9]

ISO/IEC TR 9126-1: Software engineering -Product Quality -Part 1: Quality model, 2001.

[10]

ISO/IEC TR 9126-2: Software engineering -Product Quality -Part 2: External metrics, 2003.

[11]

ISO/IEC TR 9126-3: Software engineering -Product Quality -Part 3: Internal metrics, 2003.

[12]

K. Lee and S. J. Lee: A Quantitative Software Quality Evaluation Model for the Artifacts of Component Based Development, Proc. 6th International Conference on Software Engineering, Artificial Intelligence, Networking and Paralle/Distributed Computing, and 1st ACIS International Workshop on Self-Assembling Wireless Networks, 2005.

Digital Library

[13]

M. Lorenz and J. Kidd: Object-Oriented Software Metrics, Prentice Hall, 1994.

Digital Library

[14]

F. B. Abreu, M. Gonlao and R. Esteves: Toward the Design Quality Evaluation of Object-Oriented Software Systems, Proc. 5th International Conference on Software Quality, 1995.

[15]

R. Banker, S. Datar, C. Kemerer and D. Zweig: Software Complexity and Maintenance Costs, Communications of the ACM, Vol. 36, No. 11, 1993.

Digital Library

[16]

W. P. Stevens, G. J. Myers and L. L. Constantine: Structured Design, IBM Systems Journal, Vol. 13, No. 2, 1974.

Digital Library

[17]

A. Brown: An introduction to Model Driven Architecture, IBM developerWorks, May, 2004, http: //www-128. ibm. com/developerworks/rational/library/3100. html

[18]

B. Selic: The Pragmatics of Model-Driven Development, IEEE Software, Vol. 20, No. 5, 2003.

Digital Library

[19]

K. Kang, S. Cohen, J. Hess, W. Novak and A. Peterson: Feature-Oriented Domain Analysis (FODA) Feasibility Study, CMU/SEI-90-TR-21, Carnegie Mellon University, 1990.

[20]

OMG: UML Pro. le for Schedulability, Performance, and Time (version 1. 1), 2005.

[21]

M. Marzolla and S. Balsamo: UML-PSI: The UML Performance SImulator, Proc. 1st International Conference on the Quantitative Evaluation of Systems, 2004.

Digital Library

[22]

S. Balsamo, A. Di. Marco, P. Inverardiand M. Simeoni: Model-Based Performance Prediction in Software Development: A Survey, IEEE Transactions on Software Engineering, Vol. 30, No. 5, 2004.

Digital Library

[23]

M. Woodside, D. C. Petriu, D. B. Petriu, H. Shen, T. Israr and J. Merseguer: Performance by Unified Model Analysis (PUMA), Proc. 5th ACM Workshop on Software and Performance, 2005.

Digital Library

[24]

T. Winn and P. Calder: Is This a Pattern?, IEEE Software, Vol. 19, No. 1, 2002.

Digital Library

[25]

H. Washizaki, A. Kubo, A. Takasu and Y. Fukazawa: Relation Analysis among Patterns on Software Development Process, Proc. 6th International Conference on Product Focused Software Process Improvement, LNCS Vol. 3547, 2005.

Digital Library

[26]

W. J. Brown, R. C. Malveau, H. W. McCormick and T. J. Mowbray: AntiPatterns: Refactoring Software, Architectures, and Projects in Crisis, Wiley, 1998.

Digital Library

[27]

C. U. Smith and L. G. Williams: Software Performance AntiPatterns, Proc. 2nd International Workshop on Software and Performance, 2000.

Digital Library

[28]

B. Selic, G. Gullekson and P. Ward, Real-Time Object Oriented Modelling, John Wiley &Sons, 1994.

Digital Library

[29]

S. M. Yacoub, H. H. Ammar and R. Robinson: Dynamic Metrics for Object Oriented Designs, Proc. 6th International Software Metrics Symposium, 1999.

Digital Library

[30]

M. Genero, D. Miranda and M. Piattini: Defining Metrics for UML Statechart Diagrams in a Methodological Way, International Workshop on Conceptual Modeling Quality, 2003.

Cited By

Washizaki H(2022)Model-based Robot Control Education and Software Quality in ET Software Design Robot ContestETロボコンにおけるモデルを取り入れたロボット制御の教育とソフトウェア品質Journal of the Robotics Society of Japan10.7210/jrsj.40.20840:3(208-211)Online publication date: 2022
https://doi.org/10.7210/jrsj.40.208
Nakai HTsuda NHonda KWashizaki HFukazawa Y(2016)A SQuaRE-based software quality evaluation framework and its case study2016 IEEE Region 10 Conference (TENCON)10.1109/TENCON.2016.7848750(3704-3707)Online publication date: Nov-2016
https://doi.org/10.1109/TENCON.2016.7848750
Nakai HTsuda NHonda KWashizaki HFukazawa Y(2016)Initial Framework for Software Quality Evaluation Based on ISO/IEC 25022 and ISO/IEC 250232016 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C)10.1109/QRS-C.2016.66(410-411)Online publication date: Aug-2016
https://doi.org/10.1109/QRS-C.2016.66
Show More Cited By

Index Terms

Recommendations

SQUALE - Software QUALity Enhancement
CSMR '09: Proceedings of the 2009 European Conference on Software Maintenance and Reengineering

The Squale project was born from industrial effort to control software quality. Its goals are to reﬁne and enhance Qualixo Model, a software-metric based quality model already used by large companies in France (Air France-KLM, PSA Peugeot-Citroën) and ...
Quality evaluation and analysis for domain software

Establishing attribute model to illustrate attributes to be looked into.Establishing evaluation model to illustrate computation logic of evaluation.Establishing classifies model to illustrate quality levels and mapping conditions.Analyzing quality ...
Victor R. Basili's Contributions to Software Quality

On 16 May 2005, the worldwide software engineering community honored Victor R. Basili of the University of Maryland and Fraunhofer Center for Experimental Software Engineering, Maryland, for his achievements in empirical software engineering. The ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '06: Proceedings of the 28th international conference on Software engineering

May 2006

1110 pages

ISBN:1595933751

DOI:10.1145/1134285

General Chair:
Leon J. Osterweil
University of Massachusetts, USA
,
Program Chairs:
Dieter Rombach
TU Kaiserslautern, Germany
,
Mary Lou Soffa
University of Virginia, USA

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 May 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICSE06

Sponsor:

ICSE06: International Conference on Software Engineering

May 20 - 28, 2006

Shanghai, China

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
665
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Washizaki H(2022)Model-based Robot Control Education and Software Quality in ET Software Design Robot ContestETロボコンにおけるモデルを取り入れたロボット制御の教育とソフトウェア品質Journal of the Robotics Society of Japan10.7210/jrsj.40.20840:3(208-211)Online publication date: 2022
https://doi.org/10.7210/jrsj.40.208
Nakai HTsuda NHonda KWashizaki HFukazawa Y(2016)A SQuaRE-based software quality evaluation framework and its case study2016 IEEE Region 10 Conference (TENCON)10.1109/TENCON.2016.7848750(3704-3707)Online publication date: Nov-2016
https://doi.org/10.1109/TENCON.2016.7848750
Nakai HTsuda NHonda KWashizaki HFukazawa Y(2016)Initial Framework for Software Quality Evaluation Based on ISO/IEC 25022 and ISO/IEC 250232016 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C)10.1109/QRS-C.2016.66(410-411)Online publication date: Aug-2016
https://doi.org/10.1109/QRS-C.2016.66
Kadoya MNakano TOzawa TWada MItoh HWashizaki HFukazawa YKöppe C(2013)A pattern language for the ET robot contestProceedings of the 20th Conference on Pattern Languages of Programs10.5555/2725669.2725699(1-4)Online publication date: 23-Oct-2013
https://dl.acm.org/doi/10.5555/2725669.2725699
Nakamura YSakamoto KInoue KWashizaki HFukazawa Y(2011)Evaluation of Understandability of UML Class Diagrams by Using Word SimilarityProceedings of the 2011 Joint Conference of the 21st International Workshop on Software Measurement and the 6th International Conference on Software Process and Product Measurement10.1109/IWSM-MENSURA.2011.50(178-187)Online publication date: 3-Nov-2011
https://dl.acm.org/doi/10.1109/IWSM-MENSURA.2011.50
Ormandjieva OAlagar VZheng M(2008)Early quality monitoring in the development of real-time reactive systemsJournal of Systems and Software10.1016/j.jss.2007.12.80881:10(1738-1753)Online publication date: 1-Oct-2008
https://dl.acm.org/doi/10.1016/j.jss.2007.12.808
Washizaki HNamiki RFukuoka THarada YWatanabe H(2007)A framework for measuring and evaluating program source code qualityProceedings of the 8th international conference on Product-Focused Software Process Improvement10.5555/2394450.2394483(284-299)Online publication date: 2-Jul-2007
https://dl.acm.org/doi/10.5555/2394450.2394483
WASHIZAKI HYasuhide KHiroyuki WNAKAJIMA EHAGIWARA YHIRANABE KFUKUDA K(2007)Quality Evaluation of Embedded Software in Robot Software Design ContestProgress in Informatics10.2201/NiiPi.2007.4.6(63)Online publication date: Mar-2007
https://doi.org/10.2201/NiiPi.2007.4.6
Washizaki HNamiki RFukuoka THarada YWatanabe H(2007)A Framework for Measuring and Evaluating Program Source Code QualityProceedings of the 8th international conference on Product-Focused Software Process Improvement10.1007/978-3-540-73460-4_26(284-299)Online publication date: 2-Jul-2007
https://dl.acm.org/doi/10.1007/978-3-540-73460-4_26

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten