research-article

On evaluating recommender systems for API usages

Authors:

Thorsten Schäfer,

Mira MeziniAuthors Info & Claims

RSSE '08: Proceedings of the 2008 international workshop on Recommendation systems for software engineering

Pages 16 - 20

https://doi.org/10.1145/1454247.1454254

Published: 09 November 2008 Publication History

Abstract

To ease framework understanding, tools have been developed that analyze existing framework instantiations to extract API usage patterns and present them to the user. However, detailed quantitative evaluations of such recommender systems are lacking. In this paper we present an automated evaluation process which extracts queries and expected results from existing code bases. This enables the validation of recommendation systems with large test beds in an objective manner by means of precision and recall measures. We demonstrate the applicability of our approach by evaluating an improvement of an existing API recommender tool that takes into account the framework-method context for recommendations.

References

[1]

Overview of the Third Text REtrieval Conference (TREC-3), Gaithersburg, MD, USA, 1990. NIST.

[2]

R. Agrawal, T. Imielinski, and A. Swami. Mining association rules between sets of items in large databases. In SIGMOD Int'l Conf. Management of Data, pages 207--216. ACM, 1993.

Digital Library

[3]

J. Bosch, P. Molin, M. Mattsson, and P. Bengtsson. Object-oriented framework-based software development: Problems and experiences. ACM Computing Surveys, 32(1):3--8, 2000.

Digital Library

[4]

M. Bruch, T. Schäfer, and M. Mezini. FrUiT: IDE support for framework understanding. In OOPSLA Workshop Eclipse Technology Exchange, pages 55--59. ACM, 2006.

Digital Library

[5]

W. T. Eadie, D. Drijard, and F. E. James. Statistical methods in experimental physics. Amsterdam: North-Holland, 1971.

[6]

R. Kohavi. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pages 1137--1145, 1995.

Digital Library

[7]

Z. Li and Y. Zhou. PR-Miner: Automatically extracting implicit programming rules and detecting violations in large software code. In FSE, pages 306--315. ACM, 2005.

Digital Library

[8]

B. Livshits and T. Zimmermann. Dynamine: finding common error patterns by mining software revision histories. In FSE, pages 296--305. ACM, 2005.

Digital Library

[9]

D. Mandelin, L. Xu, R. Bodík, and D. Kimelman. Jungloid mining: helping to navigate the api jungle. In PLDI, pages 48--61. ACM, 2005.

Digital Library

[10]

A. Michail. Data mining library reuse patterns using generalized association rules. In ICSE, pages 167--176. ACM, 2000.

Digital Library

[11]

N. Sahavechaphan and K. Claypool. Xsnippet: Mining for sample code. In OOPSLA, pages 413--430. ACM, 2006.

Digital Library

[12]

F. Shull, F. Lanubile, and V. R. Basili. Investigating reading techniques for object-oriented framework learning. IEEE TSE, 26(11):1101--1118, 2000.

Digital Library

[13]

S. Thummalapenta and T. Xie. Parseweb: a programmer assistant for reusing open source code on the web. In ASE, pages 204--213. ACM, 2007.

Digital Library

[14]

A. Wasylkowski, A. Zeller, and C. Lindig. Detecting object usage anomalies. In FSE, pages 35--44. ACM, 2007.

Digital Library

[15]

T. Xie and J. Pei. MAPO: mining api usages from open source repositories. In MSR, pages 54--57. ACM, 2006.

Digital Library

Cited By

Desku ARaufi BLuma ASelimi B(2022)Recommender Model for Secure Software Engineering using Cosine Similarity MeasuresInternational Journal of Engineering and Advanced Technology10.35940/ijeat.E3628.061152211:5(144-148)Online publication date: 30-Jun-2022
https://doi.org/10.35940/ijeat.E3628.0611522
Di Rocco JDi Ruscio DDi Sipio CNguyen PRubei R(2021)Development of recommendation systems for software engineering: the CROSSMINER experienceEmpirical Software Engineering10.1007/s10664-021-09963-726:4Online publication date: 14-May-2021
https://doi.org/10.1007/s10664-021-09963-7
Li HXie RKong XWang LLi B(2020)An Analysis of Utility for API Recommendation: Do the Matched Results Have the Same Efforts?2020 IEEE 20th International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS51102.2020.00067(479-488)Online publication date: Dec-2020
https://doi.org/10.1109/QRS51102.2020.00067
Show More Cited By

Index Terms

On evaluating recommender systems for API usages

Recommendations

Evaluating collaborative filtering recommender systems

Recommender systems have been evaluated in many, often incomparable, ways. In this article, we review the key decisions in evaluating collaborative filtering recommender systems: the user tasks being evaluated, the types of analysis and datasets being ...
Evaluating Recommender Systems: Survey and Framework
The comprehensive evaluation of the performance of a recommender system is a complex endeavor: many facets need to be considered in configuring an adequate and effective evaluation setting. Such facets include, for instance, defining the specific goals of ...
Evaluating Decision-Aware Recommender Systems
RecSys '17: Proceedings of the Eleventh ACM Conference on Recommender Systems

The main goal of a Recommender System is to suggest relevant items to users, although other utility dimensions - such as diversity, novelty, confidence, possibility of providing explanations - are often considered. In this work, in order to increase the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

RSSE '08: Proceedings of the 2008 international workshop on Recommendation systems for software engineering

November 2008

62 pages

ISBN:9781605582283

DOI:10.1145/1454247

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 November 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

SIGSOFT '08/FSE-16

Sponsor:

SIGSOFT

SIGSOFT '08/FSE-16: SIGSOFT 2008 -16th International Symposium on the Foundations of Software Engineering

November 9, 2008

Georgia, Atlanta

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
444
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Desku ARaufi BLuma ASelimi B(2022)Recommender Model for Secure Software Engineering using Cosine Similarity MeasuresInternational Journal of Engineering and Advanced Technology10.35940/ijeat.E3628.061152211:5(144-148)Online publication date: 30-Jun-2022
https://doi.org/10.35940/ijeat.E3628.0611522
Di Rocco JDi Ruscio DDi Sipio CNguyen PRubei R(2021)Development of recommendation systems for software engineering: the CROSSMINER experienceEmpirical Software Engineering10.1007/s10664-021-09963-726:4Online publication date: 14-May-2021
https://doi.org/10.1007/s10664-021-09963-7
Li HXie RKong XWang LLi B(2020)An Analysis of Utility for API Recommendation: Do the Matched Results Have the Same Efforts?2020 IEEE 20th International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS51102.2020.00067(479-488)Online publication date: Dec-2020
https://doi.org/10.1109/QRS51102.2020.00067
Niu HKeivanloo IZou Y(2017)API usage pattern recommendation for software developmentJournal of Systems and Software10.1016/j.jss.2016.07.026129:C(127-139)Online publication date: 1-Jul-2017
https://dl.acm.org/doi/10.1016/j.jss.2016.07.026
Niu HKeivanloo IZou Y(2017)Learning to rank code examples for code search enginesEmpirical Software Engineering10.1007/s10664-015-9421-522:1(259-291)Online publication date: 1-Feb-2017
https://dl.acm.org/doi/10.1007/s10664-015-9421-5
D'Souza AYang DLopes C(2016)Collective Intelligence for Smarter API Recommendations in Python2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation (SCAM)10.1109/SCAM.2016.22(51-60)Online publication date: Oct-2016
https://doi.org/10.1109/SCAM.2016.22
Asaduzzaman MRoy CSchneider KHou D(2016)A Simple, Efficient, Context-sensitive Approach for Code CompletionJournal of Software: Evolution and Process10.1002/smr.179128:7(512-541)Online publication date: 1-Jul-2016
https://dl.acm.org/doi/10.1002/smr.1791
Nagy CCleve A(2015)Mining Stack Overflow for discovering error patterns in SQL queriesProceedings of the 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSM.2015.7332505(516-520)Online publication date: 29-Sep-2015
https://dl.acm.org/doi/10.1109/ICSM.2015.7332505
Asaduzzaman MRoy CMonir SSchneider K(2015)Exploring API method parameter recommendationsProceedings of the 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSM.2015.7332473(271-280)Online publication date: 29-Sep-2015
https://dl.acm.org/doi/10.1109/ICSM.2015.7332473
Asaduzzaman MRoy CSchneider KHou D(2014)CSCCProceedings of the 2014 IEEE International Conference on Software Maintenance and Evolution10.1109/ICSME.2014.29(71-80)Online publication date: 29-Sep-2014
https://dl.acm.org/doi/10.1109/ICSME.2014.29
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten