skip to main content
10.1145/3273934.3273937acmotherconferencesArticle/Chapter ViewAbstractPublication PagespromiseConference Proceedingsconference-collections
research-article

An Empirical Study of Metric-based Comparisons of Software Libraries

Published: 10 October 2018 Publication History

Abstract

BACKGROUND: Software libraries provide a set of reusable functionality, which helps developers write code in a systematic and timely manner. However, selecting the appropriate library to use is often not a trivial task. AIMS: In this paper, we investigate the usefulness of software metrics in helping developers choose libraries. Different developers care about different aspects of a library and two developers looking for a library in a given domain may not necessarily choose the same library. Thus, instead of directly recommending a library to use, we provide developers with a metric-based comparison of libraries in the same domain to empower them with the information they need to make an informed decision. METHOD: We use software data analytics from several sources of information to create quantifiable metric-based comparisons of software libraries. For evaluation, we select 34 open-source Java libraries from 10 popular domains and extract nine metrics related to these libraries. We then conduct a survey of 61 developers to evaluate whether our proposed metric-based comparison is useful, and to understand which metrics developers care about. RESULTS: Our results show that developers find that the proposed technique provides useful information when selecting libraries. We observe that developers care the most about metrics related to the popularity, security, and performance of libraries. We also find that the usefulness of some metrics may vary according to the domain. CONCLUSIONS: Our survey results showed that our proposed technique is useful. We are currently building a public website for metric-based library comparisons, while incorporating the feedback we obtained from our survey participants.

References

[1]
2018. Git Tags. (2018). https://git-scm.com/book/en/v2/Git-Basics-Tagging.
[2]
2018. Library Metric Comparison Online Artifact. (2018). https://github.com/ualberta-smr/LibraryMetricScripts.
[3]
2018. MVNRepository. (2018). https://mvnrepository.com/.
[4]
H. Borges, A. C. Hora, and M. T. Valente. 2016. Understanding the factors that impact the popularity of GitHub repositories. CoRR abs/1606.04984 (2016). arXiv:1606.04984
[5]
Juliet Corbin and Anselm Strauss. 2008. Basics of qualitative research. Techniques and procedures for developing grounded theory (3rd ed.).
[6]
Daniel Alencar da Costa, Shane McIntosh, Uirá Kulesza, and Ahmed E. Hassan. 2016. The impact of switching to a rapid release cycle on integration delay of addressed Issues: an empirical study of the Mozilla Firefox project. In Proc. of the International Conference on Mining Software Repositories (MSR). 374--385.
[7]
Fernando López de la Mora and Sarah Nadi. 2018. Which library should I use? A metric-based comparison of software libraries. In Proceedings of the 40th International Conference on Software Engineering New Ideas and Emerging Results Track (ICSENIER '18).
[8]
Robert Dyer, Hoan Anh Nguyen, Hridesh Rajan, and Tien N. Nguyen. 2013. Boa: a language and infrastructure for analyzing ultra-large-scale software repositories. In Proceedings of the 2013 International Conference on Software Engineering (ICSE '13). 422--431.
[9]
Emanuel Giger, Martin Pinzger, and Harald Gall. 2010. Predicting the fix time of bugs. In Proceedings of the 2nd International Workshop on Recommendation Systems for Software Engineering (RSSE '10). 52--56.
[10]
Leo A. Goodman. 1961. Snowball Sampling. Ann. Math. Statist. 32, 1 (03 1961), 148--170.
[11]
Robert J Grissom and John J Kim. 2005. Effect sizes for research: a broad practical approach. Mahwah, N.J.; London: Lawrence Erlbaum Associates. Formerly CIP.
[12]
Emitza Guzman, David Azócar, and Yang Li. 2014. Sentiment analysis of commit comments in GitHub: an empirical study. In Proceedings of the 11th Working Conference on Mining Software Repositories (MSR 2014). 352--355.
[13]
Andre Hora and Marco Tulio Valente. 2015. Apiwave: keeping track of API popularity and migration. In Proceedings of the 31st IEEE International Conference on Software Maintenance and Evolution (ICSME '15). 321--323.
[14]
Suhas Kabinna, Cor-Paul Bezemer, Weiyi Shang, and Ahmed E. Hassan. 2016. Logging library migrations: a case study for the Apache Software Foundation projects. In Proceedings of the 13th International Conference on Mining Software Repositories (MSR '16). 154--164.
[15]
Foutse Khomh, Tejinder Dhaliwal, Ying Zou, and Bram Adams. 2012. Do faster releases improve software quality?: an empirical case study of Mozilla Firefox. In Proceedings of the 9th IEEE Working Conference on Mining Software Repositories (MSR '12). 179--188.
[16]
A. Lamkanfi, S. Demeyer, E. Giger, and B. Goethals. 2010. Predicting the severity of a reported bug. In 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010). 1--10.
[17]
Mario Linares-Vásquez, Gabriele Bavota, Carlos Bernal-Cárdenas, Massimiliano Di Penta, Rocco Oliveto, and Denys Poshyvanyk. 2013. API change and fault proneness: a threat to the success of Android apps. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2013). 477--487.
[18]
Walid Maalej and Martin P Robillard. 2013. Patterns of knowledge in API reference documentation. IEEE Trans. on Software Engineering 39, 9 (2013), 1264--1282.
[19]
Yana Momchilova Mileva, Valentin Dallmeier, Martin Burger, and Andreas Zeller. 2009. Mining trends of library usage. In Proceedings of the Joint International and Annual ERCIM Workshops on Principles of Software Evolution (IWPSE) and Software Evolution (Evol) Workshops (IWPSE-Evol '09). 57--62.
[20]
Yana Momchilova Mileva, Valentin Dallmeier, and Andreas Zeller. 2010. Mining API popularity. In Proceedings of the 5th International Academic and Industrial Conference on Testing - Practice and Research Techniques (TAIC PART' 10). 173--180.
[21]
Shaikh Mostafa, Rodney Rodriguez, and Xiaoyin Wang. 2017. Experience paper: a study on behavioral backward incompatibilities of Java software libraries. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2017). 215--225.
[22]
Masao Ohira, Yutaro Kashiwa, Yosuke Yamatani, Hayato Yoshiyuki, Yoshiya Maeda, Nachai Limsettho, Keisuke Fujino, Hideaki Hata, Akinori Ihara, and Kenichi Matsumoto. 2015. A dataset of high impact bugs: manually-classified issue reports. In Proceedings of the 12th Working Conference on Mining Software Repositories (MSR '15). 518--521.
[23]
Marco Ortu, Bram Adams, Giuseppe Destefanis, Parastou Tourani, Michele Marchesi, and Roberto Tonelli. 2015. Are bullies more productive?: empirical study of affectiveness vs. issue fixing time. In Proceedings of the 12th Working Conference on Mining Software Repositories (MSR '15). 303--313.
[24]
Nitish Pandey, Abir Hudait, Debarshi Kumar Sanyal, and Amitava Sen. 2018. Automated classification of issue reports from a software issue tracker. 423--430.
[25]
Chris Parnin and Christoph Treude. 2011. Measuring API documentation on the web. In Proceedings of the 2Nd International Workshop on Web 2.0 for Software Engineering (Web2SE '11). ACM, 25--30.
[26]
M. M. Rahman, C. K. Roy, and D. Lo. 2016. RACK: automatic API recommendation using crowdsourced knowledge. In 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Vol. 1. 349--359.
[27]
M. P. Robillard, E. Bodden, D. Kawrykow, M. Mezini, and T. Ratchford. 2013. Automated API property inference techniques. IEEE Trans. on Software Engineering 39, 5 (May 2013), 613--637.
[28]
Martin P Robillard, Andrian Marcus, Christoph Treude, Gabriele Bavota, Oscar Chaparro, Neil Ernst, Marco Aurélio Gerosa, Michael Godfrey, Michele Lanza, Mario Linares-Vásquez, et al. 2017. On-demand developer documentation. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 479--483.
[29]
Siddharth Subramanian, Laura Inozemtseva, and Reid Holmes. 2014. Live API documentation. In Proceedings of the 36th International Conference on Software Engineering. ACM, 643--652.
[30]
C. Teyton, J. R. Falleri, and X. Blanc. 2012. Mining library migration graphs. In 2012 19th Working Conference on Reverse Engineering. 289--298.
[31]
Cédric Teyton, Jean-Rémy Falleri, Marc Palyart, and Xavier Blanc. 2014. A study of library migrations in Java. Journal of Software: Evolution and Process 26, 11 (Nov. 2014), 1030--1052.
[32]
Ferdian Thung. 2016. API recommendation system for software development. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE 2016). 896--899.
[33]
Gias Uddin and Foutse Khomh. 2017. Automatic summarization of API reviews. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE '17).
[34]
C. Vendome, M. Linares-Vásquez, G. Bavota, M. Di Penta, D. German, and D. Poshyvanyk. 2017. Machine learning-based detection of open source license exceptions. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). 118--129.
[35]
Laerte Xavier, Aline Brito, Andre Hora, and Marco Tulio Valente. 2017. Historical and impact analysis of API breaking changes: a large scale study. In 24th International Conference on Software Analysis, Evolution and Reengineering (SANER '17). 138--147.

Cited By

View all
  • (2024)A Multi-Metric Ranking with Label Correlations Approach for Library Migration Recommendations2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER60148.2024.00026(183-191)Online publication date: 12-Mar-2024
  • (2024)Development of a Test Automation Framework based on a Comparison of different Approaches for Test Automation in the Embedded Systems Area2024 IEEE 7th International Conference on Industrial Cyber-Physical Systems (ICPS)10.1109/ICPS59941.2024.10640021(1-6)Online publication date: 12-May-2024
  • (2024)Studying the Evolution of Library Utilization in Maven Projects: A Metric-Based ApproachSoftware Architecture. ECSA 2023 Tracks, Workshops, and Doctoral Symposium10.1007/978-3-031-66326-0_22(359-374)Online publication date: 30-Jul-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
PROMISE'18: Proceedings of the 14th International Conference on Predictive Models and Data Analytics in Software Engineering
October 2018
97 pages
ISBN:9781450365932
DOI:10.1145/3273934
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

  • Univ. of Oulu: University of Oulu

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 October 2018

Permissions

Request permissions for this article.

Check for updates

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

Conference

PROMISE'18

Acceptance Rates

Overall Acceptance Rate 98 of 213 submissions, 46%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)41
  • Downloads (Last 6 weeks)6
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)A Multi-Metric Ranking with Label Correlations Approach for Library Migration Recommendations2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER60148.2024.00026(183-191)Online publication date: 12-Mar-2024
  • (2024)Development of a Test Automation Framework based on a Comparison of different Approaches for Test Automation in the Embedded Systems Area2024 IEEE 7th International Conference on Industrial Cyber-Physical Systems (ICPS)10.1109/ICPS59941.2024.10640021(1-6)Online publication date: 12-May-2024
  • (2024)Studying the Evolution of Library Utilization in Maven Projects: A Metric-Based ApproachSoftware Architecture. ECSA 2023 Tracks, Workshops, and Doctoral Symposium10.1007/978-3-031-66326-0_22(359-374)Online publication date: 30-Jul-2024
  • (2023)Analyzing Maintenance Activities of Software LibrariesProceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering10.1145/3593434.3593474(313-318)Online publication date: 14-Jun-2023
  • (2023)"Always Contribute Back": A Qualitative Study on Security Challenges of the Open Source Supply Chain2023 IEEE Symposium on Security and Privacy (SP)10.1109/SP46215.2023.10179378(1545-1560)Online publication date: May-2023
  • (2023)Evaluating Software Documentation Quality2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR)10.1109/MSR59073.2023.00023(67-78)Online publication date: May-2023
  • (2023)Where to Go Now? Finding Alternatives for Declining Packages in the npm EcosystemProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE56229.2023.00119(1628-1639)Online publication date: 11-Nov-2023
  • (2023)Selecting third-party libraries: the data scientist’s perspectiveEmpirical Software Engineering10.1007/s10664-022-10241-328:1Online publication date: 1-Jan-2023
  • (2022)Concept-Annotated Examples for Library ComparisonProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology10.1145/3526113.3545647(1-16)Online publication date: 29-Oct-2022
  • (2022)A Semantic Web-Enabled Approach for Dependency ManagementInternational Journal of Software Engineering and Knowledge Engineering10.1142/S021819402250049832:09(1307-1343)Online publication date: 29-Sep-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media