Article

Towards standardized evaluation of developer-assistance tools

Authors:

Sebastian Proksch,

Mira MeziniAuthors Info & Claims

RSSE 2014: Proceedings of the 4th International Workshop on Recommendation Systems for Software Engineering

Pages 14 - 18

https://doi.org/10.1145/2593822.2593827

Published: 03 June 2014 Publication History

Abstract

Over the last years, researchers proposed a variety of assistance tools to support developers in their development environments. Many of the respective publications introduce new evaluation strategies or use custom datasets. Size and quality of the performed evaluations differ. Additionally, the strategies often use metrics that are tailored to the respective tools. As a result, comparing different assistance tools is very difficult.

In this work, we present a framework for the standardized evaluation of assistance tools, on the example of code recommenders. The framework combines different ideas and demands from previous work. Furthermore, we discuss how the community could jointly realize the framework.

References

[1]

S. Amann. Code Completion Based on Implicit User Feedback. Master’s thesis, Technische Universität Darmstadt, 2013.

[2]

S. Bajracharya, T. Ngo, E. Linstead, P. Rigor, Y. Dou, P. Baldi, and C. Lopes. Sourcerer: Search Engine for Open Source Code Supporting Structure-Based Search. In Proceedings of Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), 2006.

Digital Library

[3]

M. Bruch, M. Monperrus, and M. Mezini. Learning from Examples to Improve Code Completion Systems. In Proceedings of European Software Engineering Conference (ESEC) and Symposium on Foundations of Software Engineering (FSE), 2009.

Digital Library

[4]

B. E. Cossette and R. J. Walker. Seeking the ground truth: A retroactive study on the evolution and migration of software libraries. In Proceedings of the International Symposium on the Foundations of Software Engineering (FSE), 2012.

Digital Library

[5]

T. Gvero, V. Kuncak, I. Kuraj, and R. Piskac. Complete Completion Using Types and Weights. In Proceedings of Conference on Programming Language Design and Implementation (PLDI), 2013.

Digital Library

[6]

L. Heinemann, V. Bauer, M. Herrmannsdoerfer, and B. Hummel. Identifier-Based Context-Dependent API Method Recommendation. In Proceedings of Software Maintenance and Reengineering (CSMR), 2012.

Digital Library

[7]

R. Holmes and G. C. Murphy. Using Structural Context to Recommend Source Code Examples. In Proceedings of International Conference on Software Engineering (ICSE), 2005.

Digital Library

[8]

D. Hristov, O. Hummel, M. Huq, and W. Janjic. Structuring Software Reusability Metrics for Component-Based Software Development. In Proceedings of Int. Conference on Software Engineering Advances (ICSEA), 2012.

[9]

O. Hummel. Facilitating the comparison of software retrieval systems through a reference reuse collection. In Proceedings of Workshop on Search-driven Development: Users, Infrastructure, Tools and Evaluation, 2010.

Digital Library

[10]

M. Kutschke. Ensemble Learning for Method-Call Recommendation. Master’s thesis, Technische Universität Darmstadt, 2013.

[11]

S. Lessmann, B. Baesens, C. Mues, and S. Pietsch. Benchmarking classification models for software defect prediction: A proposed framework and novel findings. Transaction on Software Engineering, 2008.

Digital Library

[12]

Z. Li and Y. Zhou. PR-Miner: Automatically Extracting Implicit Programming Rules and Detecting Violations in Large Software Code. In Proceedings of European Software Engineering Conference (ESEC) and Symposium on Foundations of Software Engineering (FSE), 2005.

Digital Library

[13]

B. Livshits and T. Zimmermann. DynaMine: Finding Common Error Patterns by Mining Software Revision Histories. In Proceedings of European Software Engineering Conference (ESEC) and Symposium on Foundations of Software Engineering (FSE), 2005.

Digital Library

[14]

D. Mandelin, L. Xu, R. Bod´ık, and D. Kimelman. Jungloid Mining: Helping to Navigate the API Jungle. In Proceedings of Conference on Programming Language Design and Implementation (PLDI), 2005.

Digital Library

[15]

A. Michail. Data Mining Library Reuse Patterns in User-Selected Applications. In Proceedings of Conference on Automated Software Engineering (ASE), 1999.

Digital Library

[16]

M. Nagappan, T. Zimmermann, and C. Bird. Representativeness in software engineering research. Technical report, Microsoft Research, 2012.

[17]

A. T. Nguyen, T. T. Nguyen, H. A. Nguyen, A. Tamrawi, H. V. Nguyen, J. Al-Kofahi, and T. N. Nguyen. Graph-based Pattern-oriented, Context-sensitive Source Code Completion. In Proceedings of International Conference on Software Engineering (ICSE), 2012.

Digital Library

[18]

R. Robbes and M. Lanza. How Program History Can Improve Code Completion. In Proceedings of Conference on Automated Software Engineering (ASE), 2008.

Digital Library

[19]

G. Robles. Replicating MSR: A study of the potential replicability of papers published in the Mining Software Repositories Proceedings. In Proceedings of Mining Software Repositories (MSR), 2010.

[20]

N. Sahavechaphan and K. Claypool. XSnippet: Mining For Sample Code. In Proceedings of Conference on Object-oriented Programming Systems, Languages, and Applications (OOPSLA), 2006.

Digital Library

[21]

S. Thummalapenta and T. Xie. Parseweb: A Programmer Assistant for Reusing Open Source Code on the Web. In Proceedings of Conference on Automated Software Engineering (ASE), 2007.

Digital Library

[22]

J. Whaley, M. C. Martin, and M. S. Lam. Automatic Extraction of Object-Oriented Component Interfaces. In Proceedings of International Symposium on Software Testing and Analysis (ISSTA), 2002.

Digital Library

[23]

D. Wightman, Z. Ye, J. Brandt, and R. Vertegaal. SnipMatch: Using Source Code Context to Enhance Snippet Retrieval and Parameterization. In Proceedings of Symposium on User Interface Software and Technology (UIST), 2012.

Digital Library

[24]

C. Zhang, J. Yang, Y. Zhang, J. Fan, X. Zhang, J. Zhao, and P. Ou. Automatic Parameter Recommendation for Practical API Usage. In Proceedings of International Conference on Software Engineering (ICSE), 2012.

Digital Library

Cited By

Hellendoorn VProksch SGall HBacchelli AAtlee JBultan TWhittle J(2019)When code completion failsProceedings of the 41st International Conference on Software Engineering10.1109/ICSE.2019.00101(960-970)Online publication date: 25-May-2019
https://dl.acm.org/doi/10.1109/ICSE.2019.00101
Cuadrado JGuerra ELara J(2018)Quick fixing ATL transformations with speculative analysisSoftware and Systems Modeling (SoSyM)10.1007/s10270-016-0541-117:3(779-813)Online publication date: 1-Jul-2018
https://dl.acm.org/doi/10.1007/s10270-016-0541-1
dos Reis Je Abreu Fde F. Carneiro G(2017)Code smells detection 2.0: Crowdsmelling and visualization2017 12th Iberian Conference on Information Systems and Technologies (CISTI)10.23919/CISTI.2017.7975961(1-4)Online publication date: Jun-2017
https://doi.org/10.23919/CISTI.2017.7975961
Show More Cited By

Index Terms

Towards standardized evaluation of developer-assistance tools
1. Software and its engineering

Recommendations

Tools for Novice and Expert Accessibility Professionals: Requirements for the Next Generation Web Accessibility Evaluation Tools List
Computers Helping People with Special Needs
Abstract
The W3C Web Accessibility Evaluation Tools list offers one of the most comprehensive and most referenced lists for accessibility evaluation tools today. The tools in the list can help check or monitor various accessibility issues on websites and ...
Towards Reliable E2R Texts: A Proposal for Standardized Evaluation Practices
Computers Helping People with Special Needs
Abstract
Easy-to-Read (E2R) is a method of enhancing the accessibility of written text by using clear, direct, and simple language. E2R texts are designed to improve readability and accessibility, especially for individuals with cognitive disabilities. ...
Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests
W4A '13: Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility

The use of web accessibility evaluation tools is a widespread practice. Evaluation tools are heavily employed as they help in reducing the burden of identifying accessibility barriers. However, an over-reliance on automated tests often leads to setting ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

RSSE 2014: Proceedings of the 4th International Workshop on Recommendation Systems for Software Engineering

June 2014

31 pages

ISBN:9781450328456

DOI:10.1145/2593822

Program Chairs:
Reid Holmes,
Werner Janjic,
Walid Maalej

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 June 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICSE '14

Sponsor:

SIGSOFT

ICSE '14: 36th International Conference on Software Engineering

June 3, 2014

Hyderabad, India

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

25
Total Citations
View Citations
184
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)1

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hellendoorn VProksch SGall HBacchelli AAtlee JBultan TWhittle J(2019)When code completion failsProceedings of the 41st International Conference on Software Engineering10.1109/ICSE.2019.00101(960-970)Online publication date: 25-May-2019
https://dl.acm.org/doi/10.1109/ICSE.2019.00101
Cuadrado JGuerra ELara J(2018)Quick fixing ATL transformations with speculative analysisSoftware and Systems Modeling (SoSyM)10.1007/s10270-016-0541-117:3(779-813)Online publication date: 1-Jul-2018
https://dl.acm.org/doi/10.1007/s10270-016-0541-1
dos Reis Je Abreu Fde F. Carneiro G(2017)Code smells detection 2.0: Crowdsmelling and visualization2017 12th Iberian Conference on Information Systems and Technologies (CISTI)10.23919/CISTI.2017.7975961(1-4)Online publication date: Jun-2017
https://doi.org/10.23919/CISTI.2017.7975961
Pavai GGeetha T(2016)A Survey on Crossover OperatorsACM Computing Surveys10.1145/300996649:4(1-43)Online publication date: 20-Dec-2016
https://dl.acm.org/doi/10.1145/3009966
Proksch SAmann SNadi SMezini MKim MRobbes RBird C(2016)A dataset of simplified syntax trees for C#Proceedings of the 13th International Conference on Mining Software Repositories10.1145/2901739.2903507(476-479)Online publication date: 14-May-2016
https://dl.acm.org/doi/10.1145/2901739.2903507
D'Souza AYang DLopes C(2016)Collective Intelligence for Smarter API Recommendations in Python2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation (SCAM)10.1109/SCAM.2016.22(51-60)Online publication date: Oct-2016
https://doi.org/10.1109/SCAM.2016.22
Amann SBeyer SKevic KGall H(2016)Software Mining Studies: Goals, Approaches, Artifacts, and ReplicabilitySoftware Engineering10.1007/978-3-319-28406-4_5(121-158)Online publication date: 13-Jan-2016
https://doi.org/10.1007/978-3-319-28406-4_5
Proksch SBauer VMurphy G(2016)How to Build a Recommendation System for Software EngineeringSoftware Engineering10.1007/978-3-319-28406-4_1(1-42)Online publication date: 13-Jan-2016
https://doi.org/10.1007/978-3-319-28406-4_1
Cuadrado JGuerra Ede Lara JLethbridge T(2015)Quick fixing ATL model transformationsProceedings of the 18th International Conference on Model Driven Engineering Languages and Systems10.5555/3351736.3351757(146-155)Online publication date: 30-Sep-2015
https://dl.acm.org/doi/10.5555/3351736.3351757
Alesio SBriand LNejati SGotlieb A(2015)Combining Genetic Algorithms and Constraint Programming to Support Stress Testing of Task DeadlinesACM Transactions on Software Engineering and Methodology10.1145/281864025:1(1-37)Online publication date: 2-Dec-2015
https://dl.acm.org/doi/10.1145/2818640
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten