Impact and utility of smell-driven performance tuning for end-user programmers

https://doi.org/10.1016/j.jvlc.2015.01.002Get rights and content

Highlights

  • Smell-driven performance tuning (SDPT) helps programmers improve code performance.

  • SDPT is targeted at end-user programmers working with dataflow languages.

  • SDPT restructures programs’ code based on the programmer’s guidance.

  • Empirical tests confirmed that SDPT substantially improves performance.

  • In a user study, end-user programmers could more easily fix performance problems.

Abstract

This paper proposes a technique, called Smell-driven performance tuning (SDPT), which semi-automatically assists end-user programmers with fixing performance problems in visual dataflow programming languages. A within-subjects laboratory experiment showed SDPT increased end-user programmers’ success rate and decreased the time they required. Another study, based on using SDPT to analyze a corpus of example end-user programs, demonstrated that applying all available SDPT transformations achieved an execution time improvement of 42% and a memory usage improvement of 20%, comparable to improvements that expert programmers historically had manually achieved on the same programs. These results indicate that SDPT is an effective method for helping end-user programmers to fix performance problems.

Introduction

Performance problems are among the trickiest types of bugs to debug. Studies have shown performance bugs take 32% longer to debug than non-performance bugs [22], resulting in substantial programmer frustration [23]. Worsening this problem, many programmers postpone fixing performance problems for as long as possible [20] even though the cost of fixing performance problems increases over time [1], [24]. Consequently, programmers need a fast and easy way to find and fix performance problems early on.

Although most related research has focused on helping professional programmers with code performance, these problems also afflict end-user programmers, who are people who write code for their own use [11]. For example, scientists are concerned with the performance of code for data acquisition and analysis [9]. Other end-user programmers face similar issues, including engineers who for example encounter performance problems when creating prototype systems with the LabVIEW dataflow programming language [4]. For instance, a LabVIEW program (called a “VI”) might need to read a signal every 100 ms to control a scientific instrument or a robot. While the code might be correct, it could be written inefficiently and execute every 250 ms, missing the desired read time, reducing the usefulness of the data. Researchers have noted the need for helping end-user programmers with performance problems in a variety of dataflow languages in addition to LabVIEW [8], [10], [12], [19]. Much of the research on end-user programmers’ code quality has focused on creating tools that aid in testing and debugging, particularly in spreadsheets (e.g., [2], [3], [5], [16], [17]), leaving end-user programmers little help with finding and fixing performance problems.

The chief prior work in this area is Smell-driven performance analysis (SDPA), which automatically provides situated explanations within a visual dataflow language IDE to help end-user programmers overcome performance problems without leaving the visual dataflow paradigm [4]. This technique analyzes end-user programmers’ dataflow code to spot potential performance problems, based on inefficient structures called “smells” that appear in the code. A study showed that an SDPA-based tool for the LabVIEW programming language enabled end-user programmers to diagnose performance problems more quickly.

However, a limitation of that work so far is that SDPA only finds errors: it does not actually help people to fix those errors. Given a problematic code structure, end-user programmers are still left to determine a more efficient, replacement code structure.

We address this limitation through a new extension of SDPA called Smell-driven performance tuning (SDPT), which semi-automatically transforms end-user dataflow code to fix inefficient code structures. To test this approach, we implemented a tool for LabVIEW to investigate how well SDPT might work in practice. Specifically, for each smell, the tool offers the programmer one or more option for removing the smell. In some cases, the removal can be accomplished automatically at the programmer’s command; in many other cases, due to the need for human input and judgment, the tool walks the programmer through the process of fixing the smell. A study using a test corpus of LabVIEW programs showed the tool-based code transformations improved performance almost as well as expert LabVIEW programmers do. An experiment showed the tool was useful for helping end-user programmers to more efficiently and successfully fix performance problems. Together, these studies show that SDPT is an effective technique for helping end-user programmers fix performance problems in dataflow code.

The remainder of this paper is organized as follows. Section 2 provides background on SDPA and summarizes other related work. Section 3 presents the SDPT technique in general, as well as the prototype that we implemented for helping LabVIEW programmers to apply this technique. Section 4 describes the study investigating the impact on performance of applying SDPT on test programs, and Section 5 presents the second study examining the usefulness of the technique for helping end-user programmers. Section 6 summarizes threats to validity, while Section 7 presents conclusions and opportunities for future work.

Section snippets

Background: Finding smells

When the idea of code smells [6] was proposed, the smells were designed to aid programmers in the refactoring of their code to ease understanding and maintenance. The most well-developed tools in this area, which have already made it into industrial practice, incorporate heuristics for finding potential maintainability problems in object-oriented code (e.g., [15], [18], [21]) Such heuristics analyze the code to find code structures that usually (though not always) signal the presence of a

Smell-driven performance tuning

While the programmer is working with a dataflow program in an IDE, SDPA identifies a set of locations in the program containing structures that could potentially impact performance [4]. If the programmer runs the program, SDPA can track whether those portions of the code consume a large proportion of the CPU time or memory, to help filter out false positives. The memory is tracked because excessive memory use has been observed to cause thrashing to disk, depending on how much memory happens to

Evaluation of transformations’ impact on performance

To assess how well the approach would accurately identify performance problems in general, a corpus of real world LabVIEW programs was gathered that the tool could be run on. This is the same corpus that we previously used to evaluate the effectiveness of SDPA for finding performance problems [4]. The corpus consisted of 30 pairs of programs, where the first in each pair was an uploaded program from a programmer that contained a performance problem before fixing, and the second was a version

Evaluation of SDPT in the hands of end-user programmers

We performed a laboratory study to evaluate the utility of SDPT, as well as to uncover opportunities for future prototype enhancements. During this within-subjects study, 32 people performed tasks with our tool, and the same 32 people performed comparable tasks without the tool.

Conclusions and future work

In this paper, we have presented a technique to help guide end-user programmers through the process of debugging dataflow code for performance problems. Our technique has proven useful for improving the success and speed of end-user programmers, and it has achieved performance improvements almost as great as those achieved by expert programmers in a given language.

During the user study, it was clear that most of the participants took the time to read the information the tool provided on a given

Acknowledgment

We thank National Instruments for funding this research, helping to recruit study participants, and providing access to the latest version of the LabVIEW development environment. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of National Instruments.

References (24)

  • C. Chambers et al.

    Reasoning about spreadsheets with labels and dimensions

    J. Visual Lang. Comput.

    (2010)
  • C. Chambers et al.

    Utility and accuracy of smell-driven performance analysis for end-user programmers

    J. Visual Lang. Comput.

    (2015)
  • B. Boehm, 2001. Software engineering economics. Pioneers and Their Contributions to Software Engineering....
  • M. Burnett et al.

    Testing homogeneous spreadsheet grids with the “what you see is what you test” methodology

    IEEE Trans. Softw. Eng.

    (2002)
  • M. Erwig et al.

    Gencel: a program generator for correct spreadsheets

    J. Funct. Program.

    (2006)
  • M. Fowler et al.

    Refactoring: Improving the Design of Existing Code

    (1999)
  • S. Grant, J. Cordy, 2003. An interactive interface for refactoring using source transformation, in: Proceedings of the...
  • O. Hassan et al.

    Enhancing scalability and performance of mashups through merging and operator reordering

    in: 2010 IEEE International Conference on Web Services (ICWS)

    (2010)
  • M. Jones et al.

    Obstacles and opportunities with using visual and domain-specific languages in scientific programming

    IEEE Symp. Visual Lang. Hum. Centric Comput.

    (2011)
  • H. Katagiri et al.

    RF monitoring system in the Injector Linac

    Intl. Conf. Accel. Large Exp. Phys. Control Syst.

    (1999)
  • A. Ko et al.

    The state of the art in end-user software engineering

    ACM Comput. Surv.

    (2011)
  • J. Liu, J. Wei, D. Ye, T. Huang, 2010. A new approach to performance optimization of mashups via data flow refactoring,...
  • Cited by (0)

    View full text