skip to main content
research-article

Retrieving API Knowledge from Tutorials and Stack Overflow Based on Natural Language Queries

Published: 22 July 2023 Publication History

Abstract

When encountering unfamiliar APIs, developers tend to seek help from API tutorials and Stack Overflow (SO). API tutorials help developers understand the API knowledge in a general context, while SO often explains the API knowledge in a specific programming task. Thus, tutorials and SO posts together can provide more API knowledge. However, it is non-trivial to retrieve API knowledge from both API tutorials and SO posts based on natural language queries. Two major problems are irrelevant API knowledge in two different resources and the lexical gap between the queries and documents. In this article, we regard a fragment in tutorials and a Question and Answering (Q&A) pair in SO as a knowledge item (KI). We generate ⟨ API, FRA⟩ pairs (FRA stands for fragment) from tutorial fragments and APIs and build ⟨ API, QA⟩ pairs based on heuristic rules of SO posts. We fuse ⟨ API, FRA⟩ pairs and ⟨ API, QA⟩ pairs to generate API knowledge (AK for short) datasets, where each data item is an ⟨ API, KI⟩ pair. We propose a novel approach, called PLAN, to automatically retrieve API knowledge from both API tutorials and SO posts based on natural language queries. PLAN contains three main stages: (1) API knowledge modeling, (2) query mapping, and (3) API knowledge retrieving. It first utilizes a deep-transfer-metric-learning-based relevance identification (DTML) model to effectively find relevant ⟨ API, KI⟩ pairs containing two different knowledge items (⟨ API, QA⟩ pairs and ⟨ API, FRA⟩ pairs) simultaneously. Then, PLAN generates several potential APIs as a way to reduce the lexical gap between the query and ⟨ API, KI⟩ pairs. According to potential APIs, we can select relevant ⟨ API, KI⟩ pairs to generate potential results. Finally, PLAN returns a list of ranked ⟨ API, KI⟩ pairs that are related to the query. We evaluate the effectiveness of PLAN with 270 queries on Java and Android AK datasets containing 10,072 ⟨ API, KI⟩ pairs. Our experimental results show that PLAN is effective and outperforms the state-of-the-art approaches. Our user study further confirms the effectiveness of PLAN in locating useful API knowledge.

References

[8]
2022. PLAN’s replication package and datasets. https://zenodo.org/record/6944137#.YuVEFurP1Jw.
[9]
Shams Azad, Peter C. Rigby, and Latifa Guerrouj. 2017. Generating API call rules from version history and stack overflow posts. ACM Transactions on Software Engineering and Methodology 25, 4 (2017), 1–22.
[10]
Alberto Bacchelli, Luca Ponzanelli, and Michele Lanza. 2012. Harnessing stack overflow for the IDE. In International Workshop on Recommendation Systems for Software Engineering. 26–30.
[11]
Lingfeng Bao, Zhenchang Xing, Xin Xia, David Lo, Minghui Wu, and Xiaohu Yang. 2020. psc2code: Denoising code extraction from programming screencasts. ACM Transactions on Software Engineering and Methodology 29, 3 (2020), 21:1–21:38.
[12]
Shaiful Alam Chowdhury and Abram Hindle. 2015. Mining StackOverflow to filter out off-topic IRC discussion. In Working Conference on Mining Software Repositories. 422–425.
[13]
Norman Cliff. 2014. Ordinal Methods for Behavioral Data Analysis. Psychology Press.
[14]
Barthélémy Dagenais and Martin P. Robillard. 2012. Recovering traceability links between an API and its learning resources. In International Conference on Software Engineering. 47–57.
[15]
Guoxian Dai, Jin Xie, Fan Zhu, and Yi Fang. 2017. Deep correlated metric learning for sketch-based 3D shape retrieval. In Conference on Artificial Intelligence. 4002–4008.
[16]
Lucas Batista Leite de Souza, Eduardo Cunha Campos, and Marcelo de Almeida Maia. 2014. Ranking crowd knowledge to assist software development. In International Conference on Program Comprehension. 72–82.
[17]
Wei Fu and Tim Menzies. 2017. Easy over hard: A case study on deep learning. In Proceedings of Joint Meeting on Foundations of Software Engineering. 49–60.
[18]
Zhipeng Gao, Xin Xia, John Grundy, David Lo, and Yuan-Fang Li. 2020. Generating question titles for stack overflow from mined code snippets. ACM Transactions on Software Engineering and Methodology 29, 4 (2020), 1–37.
[19]
Zhipeng Gao, Xin Xia, David Lo, and John Grundy. 2020. Technical Q&A site answer recommendation via question boosting. ACM Transactions on Software Engineering and Methodology 30, 1 (2020), 1–34.
[20]
Mark Grechanik, Chen Fu, Qing Xie, Collin McMillan, Denys Poshyvanyk, and Chad Cumby. 2010. A search engine for finding highly relevant applications. In International Conference on Software Engineering. 475–484.
[21]
Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018. Deep code search. In International Conference on Software Engineering. 933–944.
[22]
Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. 2016. Deep API learning. In International Symposium on Foundations of Software Engineering. 631–642.
[23]
Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. 2017. DeepAM: Migrate APIs with multi-modal sequence to sequence learning. In International Joint Conference on Artificial Intelligence. 3675–3681.
[24]
Mark A. Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA data mining software: An update. SIGKDD Explorations 11, 1 (2009), 10–18.
[25]
Junlin Hu, Jiwen Lu, and Yap-Peng Tan. 2014. Discriminative deep metric learning for face verification in the wild. In IEEE Conference on Computer Vision and Pattern Recognition. 1875–1882.
[26]
Junlin Hu, Jiwen Lu, and Yap-Peng Tan. 2015. Deep transfer metric learning. In IEEE Conference on Computer Vision and Pattern Recognition. 325–333.
[27]
Qiao Huang, Xin Xia, Zhenchang Xing, David Lo, and Xinyu Wang. 2018. API method recommendation without worrying about the task-API knowledge gap. In International Conference on Automated Software Engineering. 293–304.
[28]
He Jiang, Jingxuan Zhang, Xiaochen Li, Zhilei Ren, and David Lo. 2016. A more accurate model for finding tutorial segments explaining APIs. In International Conference on Software Analysis, Evolution, and Reengineering. 157–167.
[29]
He Jiang, Jingxuan Zhang, Zhilei Ren, and Tao Zhang. 2017. An unsupervised approach for discovering relevant tutorial fragments for APIs. In International Conference on Software Engineering. 38–48.
[30]
Xiao-Yuan Jing, Fei Wu, Xiwei Dong, and Baowen Xu. 2017. An improved SDA based defect prediction framework for both within-project and cross-project class-imbalance problems. IEEE Transactions on Software Engineering 43, 4 (2017), 321–339.
[31]
J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics (1977), 159–174.
[32]
Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In International Conference on Machine Learning 32, 2 (2014), 1188–1196.
[33]
Hongwei Li, Sirui Li, Jiamou Sun, Zhenchang Xing, Xin Peng, Mingwei Liu, and Xuejiao Zhao. 2018. Improving API caveats accessibility by mining API caveats knowledge graph. In International Conference on Software Maintenance and Evolution. 183–193.
[34]
Jing Li, Zhenchang Xing, and Muhammad Ashad Kabir. 2018. Leveraging official content and social context to recommend software documentation. IEEE Transactions on Services Computing 14, 2 (2018), 472–486.
[35]
Xiaochen Li, He Jiang, Yasutaka Kamei, and Xin Chen. 2020. Bridging semantic gaps between natural languages and APIs with word embedding. IEEE Transactions on Software Engineering 46, 10 (2020), 1081–1097.
[36]
Zhiqiang Li, Xiao-Yuan Jing, Fei Wu, Xiaoke Zhu, Baowen Xu, and Shi Ying. 2018. Cost-sensitive transfer kernel canonical correlation analysis for heterogeneous defect prediction. Automated Software Engineering 25, 2 (2018), 201–245.
[37]
Bin Lin, Fiorella Zampetti, Gabriele Bavota, Massimiliano Di Penta, and Michele Lanza. 2019. Pattern-based mining of opinions in Q&A websites. In International Conference on Software Engineering. 548–559.
[38]
Hailun Lin, Yong Liu, Weiping Wang, Yinliang Yue, and Zheng Lin. 2017. Learning entity and relation embeddings for knowledge resolution. Procedia Computer Science 108 (2017), 345–354.
[39]
Zeqi Lin, Yanzhen Zou, Junfeng Zhao, and Bing Xie. 2017. Improving software text retrieval using conceptual knowledge in source code. In International Conference on Automated Software Engineering. 123–134.
[40]
Jiakun Liu, Xin Xia, David Lo, Haoxiang Zhang, Ying Zou, Ahmed E. Hassan, and Shanping Li. 2022. Broken external links on stack overflow. IEEE Transactions on Software Engineering 48, 9 (2022), 3242–3267.
[41]
Jiwen Lu, Junlin Hu, and Yap-Peng Tan. 2017. Discriminative deep metric learning for face and kinship verification. IEEE Transactions on Image Processing 26, 9 (2017), 4269–4282.
[42]
Jiwen Lu, Gang Wang, Weihong Deng, Pierre Moulin, and Jie Zhou. 2015. Multi-manifold deep metric learning for image set classification. In IEEE Conference on Computer Vision and Pattern Recognition. 1137–1145.
[43]
Fei Lv, Hongyu Zhang, Jian-Guang Lou, Shaowei Wang, Dongmei Zhang, and Jianjun Zhao. 2015. CodeHow: Effective code search based on API understanding and extended Boolean model (E). In International Conference on Automated Software Engineering. 260–270.
[44]
Suyu Ma, Zhenchang Xing, Chunyang Chen, Cheng Chen, Lizhen Qu, and Guoqiang Li. 2021. Easy-to-deploy API extraction by multi-level feature embedding and transfer learning. IEEE Transactions on Software Engineering 47, 10 (2021), 2296–2311.
[45]
Walid Maalej and Martin P. Robillard. 2013. Patterns of knowledge in API reference documentation. IEEE Transactions on Software Engineering 39, 9 (2013), 1264–1282.
[46]
André N. Meyer, Thomas Fritz, Gail C. Murphy, and Thomas Zimmermann. 2014. Software developers’ perceptions of productivity. In Proceedings of the International Symposium on Foundations of Software Engineering. 19–29.
[47]
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Annual Conference on Neural Information Processing Systems. 3111–3119.
[48]
Trong Duc Nguyen, Anh Tuan Nguyen, Hung Dang Phan, and Tien N. Nguyen. 2017. Exploring API embedding for API usages and applications. In International Conference on Software Engineering. 438–449.
[49]
Thanh Van Nguyen, Ngoc M. Tran, Hung Phan, Trong Duc Nguyen, Linh H. Truong, Anh Tuan Nguyen, Hoan Anh Nguyen, and Tien N. Nguyen. 2018. Complementing global and local contexts in representing API descriptions to improve API retrieval tasks. In Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 551–562.
[50]
Sinno Jialin Pan, Ivor W. Tsang, James T. Kwok, and Qiang Yang. 2009. Domain adaptation via transfer component analysis. In International Joint Conference on Artificial Intelligence. 1187–1192.
[51]
Gayane Petrosyan, Martin P. Robillard, and Renato De Mori. 2015. Discovering information explaining API types using text classification. In International Conference on Software Engineering. 869–879.
[52]
L. Ponzanelli, G. Bavota, A. Mocci, R. Oliveto, M. D. Penta, S. Haiduc, B. Russo, and M. Lanza. 2019. Automatic identification and classification of software development video tutorial fragments. IEEE Transactions on Software Engineering 45, 5 (2019), 464–488.
[53]
Luca Ponzanelli, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, and Michele Lanza. 2014. Mining StackOverflow to turn the IDE into a self-confident programming prompter. In Working Conference on Mining Software Repositories. 102–111.
[54]
Mukund Raghothaman, Yi Wei, and Youssef Hamadi. 2016. SWIM: Synthesizing what i mean: Code search and idiomatic snippet synthesis. In International Conference on Software Engineering. 357–367.
[55]
Mohammad Masudur Rahman and Chanchal K. Roy. 2015. An insight into the unresolved questions at stack overflow. In Working Conference on Mining Software Repositories. 426–429.
[56]
Mohammad Masudur Rahman, Chanchal Kumar Roy, and David Lo. 2017. RACK: Code search in the IDE using crowdsourced knowledge. In International Conference on Software Engineering. 51–54.
[57]
Radim Rehurek and Petr Sojka. 2010. Software framework for topic modelling with large corpora. In The LREC 2010 Workshop on New Challenges for NLP Frameworks. 45–50.
[58]
Martin P. Robillard. 2009. What makes APIs hard to learn? Answers from developers. IEEE Software 26, 6 (2009), 27–34.
[59]
Martin P. Robillard and Yam B. Chhetri. 2015. Recommending reference API documentation. Empirical Software Engineering 20, 6 (2015), 1558–1586.
[60]
Martin P. Robillard and Robert DeLine. 2011. A field study of API learning obstacles. Empirical Software Engineering 16, 6 (2011), 703–732.
[61]
Martin P. Robillard, Andrian Marcus, Christoph Treude, Gabriele Bavota, Oscar Chaparro, Neil A. Ernst, Marco Aurélio Gerosa, Michael W. Godfrey, Michele Lanza, Mario Linares Vásquez, Gail C. Murphy, Laura Moreno, David C. Shepherd, and Edmund Wong. 2017. On-demand developer documentation. In International Conference on Software Maintenance and Evolution. 479–483.
[62]
Riccardo Rubei, Claudio Di Sipio, Phuong T. Nguyen, Juri Di Rocco, and Davide Di Ruscio. 2020. PostFinder: Mining stack overflow posts to support software developers. Information and Software Technology 127 (2020), 106367.
[63]
Siddharth Subramanian, Laura Inozemtseva, and Reid Holmes. 2014. Live API documentation. In International Conference on Software Engineering. 643–652.
[64]
Ferdian Thung, Shaowei Wang, David Lo, and Julia Lawall. 2013. Automatic recommendation of API methods from feature requests. In International Conference on Automated Software Engineering. 290–300.
[65]
Yuan Tian, David Lo, and Julia L. Lawall. 2014. Automated construction of a software-specific word similarity database. In Conference on Software Maintenance, Reengineering, and Reverse Engineering. 44–53.
[66]
Christoph Treude, Ohad Barzilay, and Margaret-Anne D. Storey. 2011. How do programmers ask and answer questions on the web? In International Conference on Software Engineering. 804–807.
[67]
Christoph Treude and Martin P. Robillard. 2016. Augmenting API documentation with insights from stack overflow. In International Conference on Software Engineering. 392–403.
[68]
Christoph Treude and Martin P. Robillard. 2017. Understanding stack overflow code fragments. In International Conference on Software Maintenance and Evolution. 509–513.
[69]
Christoph Treude, Martin P. Robillard, and Barthélémy Dagenais. 2015. Extracting development tasks to navigate software documentation. IEEE Transactions on Software Engineering 41, 6 (2015), 565–581.
[70]
Gias Uddin, Foutse Khomh, and Chanchal K. Roy. 2020. Mining API usage scenarios from stack overflow. Information and Software Technology 122 (2020), 106277.
[71]
Gias Uddin, Foutse Khomh, and Chanchal K. Roy. 2021. Automatic API usage scenario documentation from technical Q&A sites. ACM Transactions on Software Engineering and Methodology 30, 3 (2021), 1–45.
[72]
Ellen M. Voorhees. 1999. The TREC-8 question answering track report. In TREC 99 (1999), 77–82.
[73]
Haoye Wang, Xin Xia, David Lo, John C. Grundy, and Xinyu Wang. 2021. Automatic solution summarization for crash bugs. In International Conference on Software Engineering. 1286–1297.
[74]
Haoye Wang, Xin Xia, David Lo, Qiang He, Xinyu Wang, and John Grundy. 2021. Context-aware retrieval-based deep commit message generation. ACM Transactions on Software Engineering and Methodology 30, 4 (2021), 1–30.
[75]
Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics Bulletin 1, 6 (1945), 80–83.
[76]
Di Wu, Xiao-Yuan Jing, Hongyu Zhang, Bing Li, Yu Xie, and Baowen Xu. 2021. Generating API tags for tutorial fragments from stack overflow. Empirical Software Engineering 26, 4 (2021), 66.
[77]
Di Wu, Xiao-Yuan Jing, Hongyu Zhang, Yuming Zhou, and Baowen Xu. 2021. Leveraging stack overflow to detect relevant tutorial fragments of APIs. In International Conference on Software Analysis, Evolution and Reengineering. 119–130.
[78]
Di Wu, Xiao-Yuan Jing, Haowen Chen, Xiaoke Zhu, Hongyu Zhang, Mei Zuo, Lu Zi, and Chen Zhu. 2018. Automatically answering API-related questions. In International Conference on Software Engineering: Companion Proceedings. 270–271.
[79]
Di Wu, Xiao-Yuan Jing, Hongyu Zhang, Xiaohui Kong, Yu Xie, and Zhiguo Huang. 2020. Data-driven approach to application programming interface documentation mining: A review. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 10, 5 (2020), e1369.
[80]
Bowen Xu, Zhenchang Xing, Xin Xia, and David Lo. 2017. AnswerBot: Automated generation of answer summary to developers’ technical questions. In International Conference on Automated Software Engineering. 706–716.
[81]
Bowen Xu, Deheng Ye, Zhenchang Xing, Xin Xia, Guibin Chen, and Shanping Li. 2016. Predicting semantically linkable knowledge in developer online forums via convolutional neural network. In International Conference on Automated Software Engineering. 51–62.
[82]
Xin Ye, Hui Shen, Xiao Ma, Razvan C. Bunescu, and Chang Liu. 2016. From word embeddings to document similarities for improved information retrieval in software engineering. In International Conference on Software Engineering. 404–415.
[83]
Feng Zhang, Haoran Niu, Iman Keivanloo, and Ying Zou. 2018. Expanding queries for code search using semantically related API class-names. IEEE Transactions on Software Engineering 44, 11 (2018), 1070–1082.
[84]
Hongyu Zhang, Anuj Jain, Gaurav Khandelwal, Chandrashekhar Kaushik, Scott Ge, and Wenxiang Hu. 2016. Bing developer assistant: Improving developer productivity by recommending sample code. In International Symposium on Foundations of Software Engineering. 956–961.
[85]
Jingxuan Zhang, He Jiang, Zhilei Ren, Tao Zhang, and Zhiqiu Huang. 2021. Enriching API documentation with code samples and usage scenarios from crowd knowledge. IEEE Transactions on Software Engineering 47, 6 (2021), 1299–1314.
[86]
Neng Zhang, Qiao Huang, Xin Xia, Ying Zou, David Lo, and Zhenchang Xing. 2022. Chatbot4QR: Interactive query refinement for technical question retrieval. IEEE Transactions on Software Engineering 48, 4 (2022), 1185–1211.
[87]
Hao Zhong and Hong Mei. 2019. An empirical study on API usages. IEEE Transactions on Software Engineering 45, 4 (2019), 319–334.
[88]
Hao Zhong, Tao Xie, Lu Zhang, Jian Pei, and Hong Mei. 2009. MAPO: Mining and recommending API usage patterns. In Object-oriented Programming. 318–343.

Cited By

View all
  • (2024)Semantic Web Approaches in Stack OverflowInternational Journal on Semantic Web & Information Systems10.4018/IJSWIS.35861720:1(1-61)Online publication date: 13-Dec-2024
  • (2024) MR 2 -KG: A multi-relation multi-rationale knowledge graph for modeling software engineering knowledge on Stack Overflow IEEE Transactions on Software Engineering10.1109/TSE.2024.3403108(1-20)Online publication date: 2024
  • (2024)An empirical study of code reuse between GitHub and stack overflow during software developmentJournal of Systems and Software10.1016/j.jss.2024.111964210:COnline publication date: 25-Jun-2024
  • Show More Cited By

Index Terms

  1. Retrieving API Knowledge from Tutorials and Stack Overflow Based on Natural Language Queries

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Software Engineering and Methodology
    ACM Transactions on Software Engineering and Methodology  Volume 32, Issue 5
    September 2023
    905 pages
    ISSN:1049-331X
    EISSN:1557-7392
    DOI:10.1145/3610417
    • Editor:
    • Mauro Pezzè
    Issue’s Table of Contents

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 22 July 2023
    Online AM: 07 October 2022
    Accepted: 13 September 2022
    Revised: 05 August 2022
    Received: 25 November 2021
    Published in TOSEM Volume 32, Issue 5

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. API tutorial
    2. Stack Overflow
    3. deep transfer metric learning
    4. natural language queries

    Qualifiers

    • Research-article

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)267
    • Downloads (Last 6 weeks)18
    Reflects downloads up to 14 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Semantic Web Approaches in Stack OverflowInternational Journal on Semantic Web & Information Systems10.4018/IJSWIS.35861720:1(1-61)Online publication date: 13-Dec-2024
    • (2024) MR 2 -KG: A multi-relation multi-rationale knowledge graph for modeling software engineering knowledge on Stack Overflow IEEE Transactions on Software Engineering10.1109/TSE.2024.3403108(1-20)Online publication date: 2024
    • (2024)An empirical study of code reuse between GitHub and stack overflow during software developmentJournal of Systems and Software10.1016/j.jss.2024.111964210:COnline publication date: 25-Jun-2024
    • (2024)The future of API analyticsAutomated Software Engineering10.1007/s10515-024-00442-z31:2Online publication date: 9-Jun-2024
    • (2024)Learning beyond books: A hybrid model to learn real‐world problemsComputer Applications in Engineering Education10.1002/cae.2279232:6Online publication date: 21-Aug-2024
    • (2023)Automatic recognizing relevant fragments of APIs using API referencesAutomated Software Engineering10.1007/s10515-023-00401-031:1Online publication date: 19-Nov-2023

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    Full Text

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media