skip to main content
10.1145/1159733.1159775acmconferencesArticle/Chapter ViewAbstractPublication PagesesemConference Proceedingsconference-collections
Article

An empirical evaluation of a testing and debugging methodology for Excel

Published: 21 September 2006 Publication History

Abstract

Spreadsheets are one of the most commonly used types of programs in the world, and it is important that they be sufficiently dependable. To help end users who create spreadsheets do so more reliably, we have created a testing and debugging methodology and environment for use in spreadsheets, known as the WYSIWYT methodology. Our prior experiments with WYSIWYT show that users can utilize it to ensure that their spreadsheets are more dependable, but these experiments to date have considered only an unfamiliar prototype spreadsheet environment, and have not involved spreadsheet creation tasks. In this work we conducted a controlled experiment that addresses these limitations. The results of this study indicate that the use of WYSIWYT did not affect the correctness of spreadsheets created by users, but it did significantly reduce the amount of effort required to create them. Further, the subjects' evaluation of the help provided by WYSIWYT was very positive. Our results provide several insights into the use of the WYSIWYT methodology by end users.

References

[1]
Abraham, R. and Erwig, M. "Header and Unit Inference for Spreadsheets through Spatial Analyses". In Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing. Rome, Italy. Sep., 2004 p. 165--172
[2]
Ahmad, Y., Antoniu, T., Goldwater, S., and Krishnamurthi, S. "A Type System for Statically Detecting Spreadsheet Errors". In Proceedings of International Conference on Automated Software Engineering. Oct., 2003 p. 174--183
[3]
Antoniu, T., Steckler, P., Krishnamurthi, S., Neuwirth, E., and Felleisen, M. "Validating the Unit Correctness of Spreadsheet Programs". In Proceedings of 26th International Conference on Software Engineering. Edinburgh, Scotland, UK. May, 2004 p. 439--448
[4]
Ayalew, Y. and Mittermeir, R. "Interval-Based Testing for Spreadsheets". In Proceedings of International Arab Conference on Information Technology. University of Qatar, Qatar. Dec., 2002 p. 414--422
[5]
Beckwith, L., Burnett, M., and Cook, C. "Reasoning About Many-to-Many Requirement Relationships in Spreadsheets". In Proceedings of IEEE Symposium on Human Centric Computing Languages and Environments. Arlington, VA USA. Sept., 2002 p. 149--157
[6]
Beckwith, L., Burnett, M., Wiedenbeck, S., Cook, C., Sorte, S., and Hastings, M. "Effectiveness of End-User Debugging Software Features: Are There Gender Issues?" In Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems. Portland, OR USA. Apr., 2005 p. 869--878
[7]
Burnett, M., Atwood, J., Djang, R., Gottfried, H., Reichwein, J., and Yang, S., Forms/3: A First-Order Visual Language to Explore the Boundaries of the Spreadsheet Paradigm. Journal of Functional Programming, 2001. 11(2): p. 155--206.
[8]
Burnett, M., Sheretov, A., Ren, B., and Rothermel, G., Testing Homogeneous Spreadsheet Grids with the "What You See Is What You Test" Methodology. IEEE Transactions on Software Engineering, 2002: p. 576--594.
[9]
Burnett, M., Cook, C., Pendse, O., Rothermel, G., Summet, J., and Wallace, C. "End-User Software Engineering with Assertions in the Spreadsheet Paradigm". In Proceedings of 25th International Conference on Software Engineering. Portland, OR USA: IEEE-CS. May, 2003 p. 93--103
[10]
Clermont, M. "Analyzing Large Spreadsheet Programs". In Proceedings of 10th Working Conference on Reverse Engineering. Victoria, BC, Canada. Nov., 2003 p. 306--315
[11]
Clermont, M. and Mittermeir, R. "Auditing Large Spreadsheet Programs". In Proceedings of International Conference on Information Systems Implementation and Modelling. Apr., 2003 p. 87--97
[12]
Cullen, D., Excel Snafu Costs Firm $24 Million, in The Register. 2003.
[13]
Erwig, M., Abraham, R., Cooperstein, I., and Kollmansberger, S. "Automatic Generation and Maintenance of Correct Spreadsheets". In Proceedings of 27th International Conference on Software Engineering. St. Louis, MO USA. May, 2005 p. 136--145
[14]
Fisher II, M., Cao, M., Rothermel, G., Cook, C., and Burnett, M. "Automated Test Case Generation for Spreadsheets". In Proceedings of 24th International Conference on Software Engineering. May, 2002 p. 241--251
[15]
Fisher II, M., Jin, D., Rothermel, G., and Burnett, M. "Test Reuse in the Spreadsheet Paradigm ". In Proceedings of International Symposium on Software Reliability Engineering. 2002 p. 257--268
[16]
Fisher II, M., Rothermel, G., Creelan, T., and Burnett, M. Scaling a Dataflow Testing Methodology to the Multiparadigm World of Commercial Spreadsheets. TR-UNL-CSE-2005-0003. University of Nebraska -- Lincoln: Lincoln, NE USA.2005.
[17]
Ko, A.J. and Myers, B.A. "Designing the Whyline: A Debugging Interface for Asking Questions About Program Failures ". In Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems. Vienna, Austria. Apr., 2004 p. 151--158
[18]
Krishna, V., Cook, C., Keller, D., Cantrell, J., Wallace, C., Burnett, M., and Rothermel, G. "Incorporating Incremental Validation and Impact Analysis into Spreadsheet Maintenance: An Empirical Study". In Proceedings of International Conference on Software Maintenance. Florence, Italy: IEEECS. Nov., 2001 p. 72--81
[19]
Panko, R. and Halverson, R. "Spreadsheets on Trial: A Survey of Research on Spreadsheet Risks". In Proceedings of Hawaii International Conference on System Sciences. Jan., 1996 p. 326--335
[20]
Panko, R., What We Know About Spreadsheet Errors. Journal of End User Computing, 1998: p. 15--21.
[21]
Reichwein, J., Rothermel, G., and Burnett, M. "SlicingSpreadsheets: An Integrated Methodology for Spreadsheet Testing and Debugging". In Proceedings of 2nd Conference on Domain Specific Languages. Austin, TX USA. Oct., 1999 p. 25--38
[22]
Rothermel, G., Li, L., and Burnett, M. "Testing Strategies for Form-Based Visual Programs". In Proceedings of 8th International Symposium on Software Reliability Engineering. Albuquerque, NM USA: IEEE-CS. Nov., 1997 p. 96--107
[23]
Rothermel, G., Burnett, M., Li, L., DuPuis, C., and Sheretov, A., A Methodology for Testing Spreadsheets. ACM Transactions on Software Engineering and Methodology, 2001: p. 110--147.
[24]
Rothermel, K., Cook, C., Burnett, M., Schonfeld, J., Green, T., and Rothermel, G. "Wysiwyt Testing in the Spreadsheet Paradigm: An Empirical Evaluation ". In Proceedings of 22nd International Conference on Software Engineering. Limerick, Ireland: ACM. June, 2000 p. 230--239
[25]
Ruthruff, J., Creswick, E., Burnett, M., Cook, C., Prabhakararao, S., Fisher II, M., and Main, M. "End-User Software Visualizations for Fault Localization". In Proceedings of ACM Symposium on Software Visualization. San Diego, CA USA: ACM. June, 2003 p. 123--132
[26]
Ruthruff, J., Burnett, M., and Rothermel, G. "An Empirical Study of Fault Localization for End-User Programmers". In Proceedings of 27th International Conference on Software Engineering. St. Louis, MO USA. May, 2005 p. 352--361
[27]
Sajaniemi, J., Modeling Spreadsheet Audit: A Rigorous Approach to Automatic Visualization. Journal of Visual Languages and Computing, 2000. 11(1): p. 49--82.
[28]
Scott, A., Shurgard Stock Dives after Auditor Quits over Company's Accounting, in The Seattle Times. 2003.
[29]
Wiedenbeck, S., Zila, P.L., and McConnell, D.S. "End-User Training: An Empirical Study Comparing on-Line Practice Methods". In Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems. Denver, CO USA. May, 1995 p. 74--81
[30]
Wilson, A., Burnett, M., Beckwith, L., Granatir, O., Casburn, L., Cook, C., Durham, M., and Rothermel, G. "Harnessing Curiosity to Increase Correctness in End-User Programming". In Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems Ft. Lauderdale, FL USA: ACM. Apr., 2003 p. 305--312

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ISESE '06: Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering
September 2006
388 pages
ISBN:1595932186
DOI:10.1145/1159733
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 September 2006

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. empirical study
  2. end-user software engineering
  3. human subjects

Qualifiers

  • Article

Conference

ISESE06
Sponsor:

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)9
  • Downloads (Last 6 weeks)3
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2018)ExceLint: automatically finding spreadsheet formula errorsProceedings of the ACM on Programming Languages10.1145/32765182:OOPSLA(1-26)Online publication date: 24-Oct-2018
  • (2015)Embedding, Evolution, and Validation of Model-Driven SpreadsheetsIEEE Transactions on Software Engineering10.1109/TSE.2014.236114141:3(241-263)Online publication date: 1-Mar-2015
  • (2014)CheckCellACM SIGPLAN Notices10.1145/2714064.266020749:10(507-523)Online publication date: 15-Oct-2014
  • (2014)CheckCellProceedings of the 2014 ACM International Conference on Object Oriented Programming Systems Languages & Applications10.1145/2660193.2660207(507-523)Online publication date: 15-Oct-2014
  • (2010)Probabilistic and analytical estimation of software development team sizeInternational Journal of Hybrid Intelligent Systems10.5555/1839206.18392117:2(137-153)Online publication date: 1-Apr-2010

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media