Hridesh Rajan
Kevin Sullivan
ABSTRACT
In program profiling to assess test set adequacy, a challenge is to select code to be included in the assessment. Current mechanisms are coarse-grained; biased to dominant modularizations; require tedious, error-prone manual selection; and leave tester intent implicit in inputs to testing tools. Aspect-oriented constructs promise to improving testing in two ways: by improving our ability to select the code to include in adequacy criteria, and by documenting selection intentions in declarative form in the code itself. One problem is that current join point models do not reveal program behavior in enough detail to support white-box coverage analysis. Our contribution is the formulation, prototyping, and evaluation of a language-and-tool-based approach to white-box coverage adequacy analysis that we call concern coverage. We develop and evaluate one instance of the general idea in which branches, in particular, are exposed as join points to support branch coverage analysis of crosscutting concerns. Our results are consistent with the claim that the idea has the potential to improve test coverage analysis.
- Agrawal, H., "Dominators, Super Blocks and Program Coverage", POPL 94, Portland, Oregon, pp. 25--34. Google Scholar
Digital Library
- AspectC++ Homepage: http://www.aspectc.org.Google Scholar
- AspectJ Homepage: http://www.eclipse.org/aspectj.Google Scholar
- Ball, T., and Larus, J., "Optimally Profilling and Tracing Programs", ACM Transactions on Programming Languages and Systems, Vol 16, no. 4, July 1994, pp. 1319--1360. Google ScholarDigital Library
- Ball, T., and Larus, J., "Efficient Path Profiling", 29th Annual IEEE/ACM International Symposium on Microarchitecture, Paris, France, pp. 46--57. Google ScholarDigital Library
- Boehm, B., "Software and its impact: a quantitative assessment", Datamation 19(May 1973), pp. 48--59.Google Scholar
- Burke B., "Aspect-Oriented Annotations", http://onjava.com article dated Aug 25, 2004.Google Scholar
- Byte Code Engineering Library (BCEL): http://jakarta.apache.org/bcel/Google Scholar
- Chiba, S., "A Metaobject Protocol for C++", In Proceedings of the ACM Conference on Object-Oriented Programming Systems, Languages, and Applications (OOPSLA), page 285--299, October 1995. Google ScholarDigital Library
- CVS client library for C#: http://sharpcvslib.sourceforge.net/Google Scholar
- Cordy, J. R., Halpern-Hamu, C D., and Promislow, E. "TXL: a rapid prototyping system for programming language dialects", Computer Languages 16(1) (1991): pp. 97--107. Google ScholarDigital Library
- Cordy, J.R., and Shukla, M., "Practical Metaprogramming", Proc. CASCON '92, IBM Centre for Advanced Studies 1992 Conference, Toronto, November 1992, pp. 215--224. Google ScholarDigital Library
- Dahl, O. J., Dijkstra, E. W., and Hoare, C. A. R. "Structured Programming", Academic Press, New York, 1972. Google ScholarDigital Library
- Dean, J., Hicks, J.E., Waldspurger, C.A., Weihl, W. E., and Chrysos, G., "ProfileMe: Hardware Support for Instruction-Level Profiling on Out-of-Order Processors", Proceedings of the Thirtieth Annual IEEE/ACM International Symposium on Microarchitecture, December 1-3, 1997, Research Triangle Park, North Carolina, USA, pp. 292--302. Google ScholarDigital Library
- Dean, J., Waldspurger, C. A., and Weihl, W.E., "Transparent, Low-Overhead Profiling on Modern Processors," Workshop on Profile and Feedback-Directed Compilation held in conjunction with PACT 98. Oct. 1998, Paris, France.Google Scholar
- Elbaum, S., Gable, D., Rothermel, G., "The impact of software evolution on code coverage information", IEEE International conference on Software Maintenance, 2001, pp. 170--179. Google ScholarDigital Library
- Emden, E. V., and Moonen, L., "Java Quality Assurance by Detecting Code Smells", Proceedings of the 9th Working Conference on Reverse Engineering, IEEE Computer Society Press, Oct 2002, pp. 97--108. Google ScholarDigital Library
- Eos Homepage: http://www.cs.virginia.edu/~eosGoogle Scholar
- Fagan, M. E., "Design and Code Inspections to Reduce Errors in Program Development", IBM Systems Journal, 15, 3, 1976, pp. 182--211.Google ScholarDigital Library
- Fagan, M. E., "Advances in Software Inspections", IEEE Transactions in Software Engineering SE-12, 7 (July) 1986, pp. 744--751. Google ScholarDigital Library
- FEAT: http://www.cs.ubc.ca/labs/spl/projects/feat/Google Scholar
- Fowler, M., "Refactoring: Improving the Design of Existing Code", Addison-Wesley, 1999. Google ScholarDigital Library
- Gilb, T., and Graham, D., "Software Inspection", Addison-Wesley, 1993. Google ScholarDigital Library
- Goodenough, J. B., Gerhart, S. L., "Toward a Theory of Test Data selection", IEEE Transactions on Software Engineering, 1(2), pp. 156--173, 1975.Google ScholarDigital Library
- Graham, S.L., Kessler, P.B., and McKusick, M.K., "An execution profiler for modular programs", Software Practice and Experience, 13: pp. 671--685, 1983.Google ScholarCross Ref
- Grinwald, R., Harel, E., Orgad, M., Ur S., and Ziv, A., "User Defined Coverage - A Tool Supported Methodology for Design Verification", Design Automation Conference, 1998, pp. 158--163. Google ScholarDigital Library
- Harrold, M. J., "Testing: A Roadmap", In Future of Software Engineering, 22nd International Conference on Software Engineering, June 2000, pp. 61--72. Google ScholarDigital Library
- Harrold M. J., and Rothermel, G., "Aristotle: A System for Research On and Development of Program-Analysis-Based Tools", Technical Report OSU-CISRC-3/97-TR17, Department of Computer and Information Science, The Ohio State University, March 1997.Google Scholar
- Hong, Z., Patrick A. V. H., and John H. R. M., "Software unit test coverage and adequacy", ACM Computing Surveys, 29(4): pp. 366--427, 1997. Google ScholarDigital Library
- Java: http://java.sun.comGoogle Scholar
- JBoss AOP: http://www.jboss.org/products/aopGoogle Scholar
- Kiczales, G., "The More the Merrier", Software Development Magazine, Sept 2004.Google Scholar
- Kiczales, G., "The fun has just begun", Key note address of International Conference on Aspect-Oriented Software Development, Boston, MA, 2003.Google Scholar
- Kiczales, G., Lamping, J., Mendhekar, A., Maeda, C., Lopes, C., Loingtier, J. and Irwin, J., "Aspect-oriented programming," in Proceedings of the European Conference on Object-Oriented Programming (ECOOP), Springer-Verlang, Lecture Notes on Computer Science 1241, June 1997, pp. 220--242.Google ScholarCross Ref
- Kniesel, G., Austermann, M., "CC4J - Code Coverage for Java - A Load-Time Adaptation Success Story", In Component Deployment - IFIP/ACM Working Conference, CD 2002, Berlin, Germany, June 20-21, 2002, Proceedings, Springer LNCS 2370, pp. 155--169, 2002. Google ScholarDigital Library
- Knuth, D., E., and Stevenson, F., R., "Optimal measurement points for program frequency counts", BIT, 13: pp. 313--322, 1973.Google ScholarCross Ref
- Kotik, G., Markosian L., "Automating software analysis and testing using a program transformation system", Proceedings of the ACM SIGSOFT '89 third symposium on Software testing, analysis, and verification, Key West, Florida, United States, pp. 75--84, 1989. Google ScholarDigital Library
- Lamping, J., "The role of the base in aspect-oriented programming", First Workshop on Multi-Dimensional Separation of Concerns in Object-oriented Systems (at OOPSLA '99). Google ScholarDigital Library
- Larus, J., "Whole Program Paths," Proceedings of the SIGPLAN '99 Conference on Programming Languages Design and Implementation (PLDI 99), May 1999, Atlanta Georgia. Google ScholarDigital Library
- Mahrenholz, D., Spinczyk, O., Schröder-Preikschat, W., "Program Instrumentation for Debugging and Monitoring with AspectC++", The 5th IEEE International Symposium on Object-oriented Real-time Distributed Computing, Washington DC, USA, April 29 - May 1, 2002, pp. 249--256. Google ScholarDigital Library
- Marrè, M., and Bertolino, A., "Using Spanning Sets for Coverage Testing", IEEE Transactions on Software Engineering, Vol. 29, No. 11, November 2003, pp. 974--984. Google ScholarDigital Library
- Masuhara, H., and Kiczales G., "Modular Crosscutting in Aspect-Oriented Mechanisms", ECOOP 2003, Darmstadt, Germany, July 2003.Google Scholar
- Microsoft. C# Specification Homepage. http://msdn.microsoft.com/net/ecmaGoogle Scholar
- Microsoft .Net: http://www.microsoft.com/netGoogle Scholar
- NAnt homepage: http://nant.sourceforge.net/Google Scholar
- NUnit homepage: http://nunit.sourceforge.net/Google Scholar
- Ossher, H., and Tarr, P., "Operation-Level Composition: A Case in (Join) Point", Workshop on aspect-oriented programming, ECOOP 1998. Google ScholarDigital Library
- Probert, R. L., "Optimal Insertion of Software Probes in Well-Delimited Programs," IEEE Transactions on Software Engineering, January, 1981, pp. 34--42.Google Scholar
- Rational PureCoverage: http://www.ibm.comGoogle Scholar
- Rajan, H., and Sullivan, K., "Eos: Instance-Level Aspects for Integrated System Design", 2003 Joint European Software Engineering Conference and ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE 03), (Helsinki, Finland, Sept 2003). Google ScholarDigital Library
- Rajan, H., and Sullivan, K., "Classpects: Unifying Aspect-and Object-Oriented Language Design", Technical Report CS-2004-21, Department of Computer Science, University of Virginia, Sept 2004.Google Scholar
- Ramamoorthy, C. V., Kim, K. H., and Chen. W. T., "Optimal placement of software monitors aiding systematic testing." IEEE Transactions on Software Engineering, SE-1(4):403--411, Dec. 1975.Google ScholarDigital Library
- Russell, G. W., "Experience with inspection in ultralargescale developments.", IEEE Software, 8(1):25--31, 1991. Google ScholarDigital Library
- Schulte, W., Personal communication with Hridesh Rajan at Microsoft ResearchGoogle Scholar
- Shukla, D., Fell, S., and Sells, C., "Aspect-Oriented Programming Enables Better Code Encapsulation and Reuse", MSDN Magazine, March 2002.Google Scholar
- Souter, A. L., Shepherd, D., and Pollock, L L., "Concern-based Testing," IEEE International Conference on Software Maintenance, September, 2003Google Scholar
- Soot: a Java Optimization Framework: http://www.sable.mcgill.ca/soot/Google Scholar
- Tarr, P., Ossher, H., Harrison, W., and Sutton, Jr. S.M., "N Degrees of Separation: Multi-Dimensional Separation of Concerns." Proceedings of the International Conference on Software Engineering (ICSE'99), May, 1999. Google ScholarDigital Library
- Tikir M. M., Hollingsworth J. K., "Efficient instrumentation for code coverage testing.", ACM. Sigsoft Software Engineering Notes, vol. 27, no. 4, July 2002, pp. 86--96. USA. Google ScholarDigital Library
- Ubayashi, N., Tamai, T., "Aspect-oriented programming with model checking", Proceedings of the 1st international conference on Aspect-oriented software development, Enschede, The Netherlands, 148--154, 2002. Google ScholarDigital Library
- Ur, S., Ziv, A., "Of-The-Shelf Vs. Custom Made Coverage Models, Which is the one for You?", STAR 98 May 1998.Google Scholar
- Vollmer, J., "Experiences with Gentle: efficient compiler construction based on logic programming." Proc. 3rd International Symposium on Programming Language Implementation and Logic Programming (PLILP 91), Springer Verlag Lecture Notes in Computer Science 528(1991): 425--426.Google ScholarCross Ref
- Weihl, W.E., "CPI: Continous Profiling Infrastructure", DIGITAL Forefront Magazine, Winter 1997, pages 27--28.Google Scholar
- Quilt: http://quilt.sourceforge.net/Google Scholar
Index Terms
- Aspect language features for concern coverage profiling
Recommendations
Checked Coverage and Object Branch Coverage: New Alternatives for Assessing Student-Written Tests
SIGCSE '15: Proceedings of the 46th ACM Technical Symposium on Computer Science EducationMany educators currently use code coverage metrics to assess student-written software tests. While test adequacy criteria such as statement or branch coverage can also be used to measure the thoroughness of a test suite, they have limitations. Coverage ...
Bringing white-box testing to Service Oriented Architectures through a Service Oriented Approach
The attractive feature of Service Oriented Architecture (SOA) is that pieces of software conceived and developed by independent organizations can be dynamically composed to provide richer functionality. The same reasons that enable flexible compositions,...
State coverage: a structural test adequacy criterion for behavior checking
ESEC-FSE '07: Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineeringWe propose a new language-independent, structural test adequacy criterion called state coverage. State coverage measures whether unit-level tests check the outputs and sideeffects of a program.
State coverage differs in several respects from existing ...
Comments