skip to main content
10.1145/2693182.2693185acmconferencesArticle/Chapter ViewAbstractPublication PagesicpeConference Proceedingsconference-collections
research-article

High-Volume Performance Test Framework using Big Data

Published: 01 February 2015 Publication History

Abstract

The inherent issues with handling large files and complex scenarios cause the data-driven approach [1] to be rarely used for performance tests. Volume and scalability testing of enterprise solutions typically requires custom-made test frameworks because of the complexity and uniqueness of data flow. The generation, transformation and transmission of large sets of data pose a unique challenge for testing a highly transactional back-end system like the IBM Sterling Order Management (OMS). This paper describes a test framework built on document-oriented NoSQL database, a design that helps validate the functionality and scalability of the solution simultaneously. This paper also describes various phases of planning, development, and testing of the OMS solution that was executed for a large retailer in Europe to test an extremely high online sales scenario. An out-of-the-box configuration of the OMS with the feature support for database sharding was used to drive scalability. The exercise was a success, and it is the world's largest IBM Sterling Order Management benchmark in terms of sales order volume, to date.

References

[1]
Baker, P., Dai, Z. R., Grabowski, J., Haugen, Ø., Schieferdecker, I., & Williams, C. (2008). Data-Driven Testing. In Model-Driven Testing (pp. 87--95). Springer Berlin Heidelberg.
[2]
"Sterling Order Management", DOI= http://www-03.ibm.com/software/products/en/order-management
[3]
IBM: Rational Performance Tester. http://www-01.ibm.com/software/awdtools/tester/performance
[4]
Hewlett Packard: HP LoadRunner. http://www8.hp.com/us/en/software-solutions/loadrunner-load-testing/index.html
[5]
Apache Software Foundation: Apache JMeter. http://jmeter.apache.org/
[6]
Aston, P.: The Grinder, a Java Load Testing Framework. http://grinder.sourceforge.net/
[7]
Chen, S., Moreland, D., Nepal, S., Zic, J.: Yet Another Performance Testing
[8]
Mayer, D. A., Steele, O., Wetzel, S., & Meyer, U. (2012). CaPTIF: Comprehensive Performance TestIng Framework. In Testing Software and Systems (pp. 55--70). Springer Berlin Heidelberg.
[9]
Chodorow, Kristina. "MongoDB: the definitive guide." O'Reilly Media, Inc.", 2013.
[10]
Crockford, D.: The application/json Media Type for JavaScript Object Notation (JSON). RFC 4627 (Informational) (July 2006)
[11]
Boicea, A., Radulescu, F., & Agapin, L. I. (2012, September). MongoDB vs Oracle-Database Comparison. In EIDWT (pp. 330--335).
[12]
Lv, Q., & Xie, W. (2014, August). A Real-Time Log Analyzer Based on MongoDB. In Applied Mechanics and Materials (Vol. 571, pp. 497--501).

Cited By

View all
  • (2024)Quality assurance strategies for machine learning applications in big data analytics: an overviewJournal of Big Data10.1186/s40537-024-01028-y11:1Online publication date: 30-Oct-2024
  • (2022)Correcting Temporal Overlaps in Process Models Discovered from OLTP DatabasesAdvanced Data Mining and Applications10.1007/978-3-031-22137-8_21(281-296)Online publication date: 24-Nov-2022
  • (2017)A Survey on Quality Assurance Techniques for Big Data Applications2017 IEEE Third International Conference on Big Data Computing Service and Applications (BigDataService)10.1109/BigDataService.2017.42(313-319)Online publication date: Apr-2017
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
LT '15: Proceedings of the 4th International Workshop on Large-Scale Testing
February 2015
24 pages
ISBN:9781450333375
DOI:10.1145/2693182
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 February 2015

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. big data
  2. document oriented storage
  3. load testing
  4. order management
  5. rapid prototyping
  6. test automation tool
  7. test harness

Qualifiers

  • Research-article

Conference

ICPE'15
Sponsor:

Acceptance Rates

Overall Acceptance Rate 2 of 2 submissions, 100%

Upcoming Conference

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)3
  • Downloads (Last 6 weeks)0
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Quality assurance strategies for machine learning applications in big data analytics: an overviewJournal of Big Data10.1186/s40537-024-01028-y11:1Online publication date: 30-Oct-2024
  • (2022)Correcting Temporal Overlaps in Process Models Discovered from OLTP DatabasesAdvanced Data Mining and Applications10.1007/978-3-031-22137-8_21(281-296)Online publication date: 24-Nov-2022
  • (2017)A Survey on Quality Assurance Techniques for Big Data Applications2017 IEEE Third International Conference on Big Data Computing Service and Applications (BigDataService)10.1109/BigDataService.2017.42(313-319)Online publication date: Apr-2017
  • (2016)Preliminary big data in a 5G test network2016 IEEE International Conference on Big Data (Big Data)10.1109/BigData.2016.7840918(2722-2727)Online publication date: Dec-2016

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media