Skip to main content
SpringerLink
Log in

An analysis of correctness for API recommendation: are the unmatched results useless?

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

API recommendation is a promising approach which is widely used during software development. However, the evaluation of API recommendation is not explored with sufficient rigor. The current evaluation of API recommendation mainly focuses on correctness, the measurement is conducted by matching recommended results with ground-truth results. In most cases, there is only one set of ground-truth APIs for each recommendation attempt, but the object code can be implemented in dozens of ways. The neglect of code diversity results in a possible defect in the evaluation. To address the problem, we invite 15 developers to analyze the unmatched results in a user study. The online evaluation confirms that some unmatched APIs can also benefit to programming due to the functional correlation with ground-truth APIs. Then we measure the API functional correlation based on the relationships extracted from API knowledge graph, API method name, and API documentation. Furthermore, we propose an approach to improve the measurement of correctness based on API functional correlation. Our measurement is evaluated on a dataset of 6141 requirements and historical code fragments from related commits. The results show that 28.2% of unmatched APIs can contribute to correctness in our experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Xia X, Bao L F, Lo D, et al. What do developers search for on the web? Empir Softw Eng, 2017, 22: 3149–3185

    Article  Google Scholar 

  2. Bao L F, Xing Z C, Wang X Y, et al. Tracking and analyzing cross-cutting activities in developers’ daily work. In: Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering, 2015. 277–282

  3. Lv C, Jiang W, Liu Y, et al. APISynth: a new graph-based API recommender system. In: Proceedings of the 36th International Conference on Software Engineering, 2014. 596–597

  4. Thung F. API recommendation system for software development. In: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, 2016. 896–899

  5. Yu H B, Song W H, Mine T. APIBook: an effective approach for finding APIs. In: Proceedings of the 8th Asia-Pacific Symposium on Internetware, 2016. 45–53

  6. Robillard M, Walker R, Zimmermann T. Recommendation systems for software engineering. IEEE Softw, 2010, 27: 80–86

    Article  Google Scholar 

  7. Rahman M M, Roy C K, Lo D. Rack: automatic API recommendation using crowdsourced knowledge. In: Proceedings of the 23rd International Conference on Software Analysis, Evolution, and Reengineering, 2016. 349–359

  8. Raghothaman M, Wei Y, Hamadi Y. Swim: synthesizing what I mean-code search and idiomatic snippet synthesis. In: Proceedings of the 38th International Conference on Software Engineering, 2016. 357–367

  9. Xu C Y, Sun X B, Li B, et al. MULAPI: improving API method recommendation with API usage location. J Syst Softw, 2018, 142: 195–205

    Article  Google Scholar 

  10. Proksch S, Amann S, Nadi S, et al. Evaluating the evaluations of code recommender systems: a reality check. In: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, 2016. 111–121

  11. Nguyen A T, Nguyen T N. Graph-based statistical language model for code. In: Proceedings of the 37th IEEE International Conference on Software Engineering, 2015. 858–868

  12. Asaduzzaman M, Roy C K, Schneider K A, et al. A simple, efficient, context-sensitive approach for code completion. J Softw Evol Proc, 2016, 28: 512–541

    Article  Google Scholar 

  13. Thung F, Lo D, Lawall J. Automated library recommendation. In: Proceedings of the 20th Working Conference on Reverse Engineering, 2013. 182–191

  14. Li J, Wang Y, Lyu M R, et al. Code completion with neural attention and pointer networks. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence, 2018. 4159–4165

  15. Yuan W Z, Nguyen H H, Jiang L X, et al. LibraryGuru: API recommendation for Android developers. In: Proceedings of the 40th International Conference on Software Engineering, 2018. 364–365

  16. Cao B Q, Liu X Q, Rahman M, et al. Integrated content and network-based service clustering and web APIs recommendation for mashup development. IEEE Trans Serv Comput, 2020, 13: 99–113

    Article  Google Scholar 

  17. Beel J, Genzmehr M, Langer S, et al. A comparative analysis of offline and online evaluations and discussion of research paper recommender system evaluation. In: Proceedings of International Workshop on Reproducibility and Replication in Recommender Systems Evaluation, 2013. 7–14

  18. Manning C, Raghavan P, Schütze H. Introduction to information retrieval. Nat Lang Eng, 2010, 16: 100–103

    Article  Google Scholar 

  19. Thung F, Wang S W, Lo D, et al. Automatic recommendation of API methods from feature requests. In: Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering, 2013. 290–300

  20. Thung F, Oentaryo R J, Lo D, et al. WebAPIRec: recommending web APIs to software projects via personalized ranking. IEEE Trans Emerg Top Comput Intell, 2017, 1: 145–156

    Article  Google Scholar 

  21. Nguyen A T, Hilton M, Codoban M, et al. API code recommendation using statistical learning from fine-grained changes. In: Proceedings of ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2016. 511–522

  22. Shani G, Gunawardana A. Evaluating recommendation systems. In: Recommender systems handbook. Berlin: Springer, 2011. 257–297

  23. Huang Q, Xia X, Xing Z C, et al. API method recommendation without worrying about the task-API knowledge gap. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, 2018. 293–304

  24. Nguyen A T, Hilton M, Codoban M, et al. API code recommendation using statistical learning from fine-grained changes. In: Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2016. 511–522

  25. Bruch M, Monperrus M, Mezini M. Learning from examples to improve code completion systems. In: Proceedings of the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, 2009. 213–222

  26. Gu X D, Zhang H Y, Zhang D M, et al. Deep API learning. In: Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2016. 631–642

  27. Weimer M, Karatzoglou A, Le V, et al. COFIRANK: maximum margin matrix factorization for collaborative ranking. In: Proceedings of Annual Conference on Neural Information Processing Systems, 2007. 222–230

  28. Weimer W, Fry Z P, Forrest S. Leveraging program equivalence for adaptive program repair: models and first results. In: Proceedings of the 28th International Conference on Automated Software Engineering, 2013. 356–366

  29. Levenshtein V. Binary codes capable of correcting insertions and reversals. Sov Phys Dokl, 1966, 10: 707–710

    MathSciNet  Google Scholar 

Download references

Acknowledgements

This work was supported in part by National Key R&D Program of China (Grant No. 2018YFB100-3900), in part by National Natural Science Foundation of China (Grant Nos. 61402103, 61572126, 61872078), in part by Open Research Fund of Key Laboratory of Safety-Critical Software Fund (Nanjing University of Aeronautics and Astronautics) (Grant No. NJ2019006), and in part by Key Laboratory of Computer Network and Information Integration of the Ministry of Education of China (Grant No. 93K-9).

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Southeast University, Nanjing, 211189, China

    Xianglong Kong, Weina Han, Li Liao & Bixin Li

Authors
  1. Xianglong Kong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weina Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Li Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bixin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bixin Li.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kong, X., Han, W., Liao, L. et al. An analysis of correctness for API recommendation: are the unmatched results useless?. Sci. China Inf. Sci. 63, 190103 (2020). https://doi.org/10.1007/s11432-019-2929-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-019-2929-9

Keywords

Search

Navigation