skip to main content
research-article

Knowledge Discovery in Simulation Data

Published: 25 November 2020 Publication History

Abstract

This article provides a comprehensive and in-depth overview of our work on knowledge discovery in simulations. Application-wise, we focus on manufacturing simulations. Specifically, we propose and discuss a methodology for designing, executing, and analyzing large-scale simulation experiments with a broad coverage of possible system behavior targeted at generating knowledge about the system. Based on the concept of data farming, we suggest a two-phase process which starts with a data generation phase, in which a smart experiment design is used to set up and efficiently execute a large number of simulation experiments. In the second phase, the knowledge discovery phase, data mining and visually aided analysis methods are applied on the gathered simulation input and output data. This article gives insights into this knowledge discovery phase by discussing different machine learning approaches and their suitability for different manufacturing simulation problems. With this, we provide guidelines on how to conduct knowledge discovery studies within the manufacturing simulation context. We also introduce different case studies, both academic and applied, and use them to validate our methodology.

References

[1]
R. Arias-Hernandez, L. T. Kaastra, T. M. Green, and B. Fisher. 2011. Pair analytics: Capturing reasoning processes in collaborative visual analytics. In Proceedings of the 44th Hawaii International Conference on System Sciences. IEEE, Washington DC, 1–10.
[2]
Michael Behrisch, Dirk Streeb, Florian Stoffel, Daniel Seebacher, Brian Matejek, Stefan H. Weber, Sebastian Mittelstaedt, Hanspeter Pfister, and Daniel Keim. 2018. Commercial visual analytics systems - Advances in the big data analytics field. IEEE Transactions on Visualization and Computer Graphics.
[3]
Michael R. Berthold, Nicolas Cebron, Fabian Dill, Thomas R. Gabriel, Tobias Kötter, Thorsten Meinl, Peter Ohl, Kilian Thiel, and Bernd Wiswedel. 2009. KNIME - The Konstanz information miner. ACM SIGKDD Explorations Newsletter 11, 1 (2009), 26.
[4]
Christopher M. Bishop. 2009. Pattern recognition and machine learning (8th). Information Science and Statistics. Springer, New York.
[5]
Thomas M. Cioppa and Thomas W. Lucas. 2007. Efficient nearly orthogonal and space-filling Latin hypercubes. Technometrics 49, 1 (2007), 45--55.
[6]
Çağatay Demiralp, Peter J. Haas, Srinivasan Parthasarathy, and Tejaswini Pedapati. 2017. Foresight: Rapid data exploration through guideposts. In Proceedings of the DSIA Workshop at IEEE VIS.
[7]
Filip K. Dosilovic, Mario Brcic, and Nikica Hlupic. 2018. Explainable artificial intelligence: A survey. In Proceedings of the 41st International Convention on Information and Communication Technology, Electronics and Microelectronics. IEEE, 210--215.
[8]
Geoffry Ellis and Alan Dix. 2007. A taxonomy of clutter reduction for information visualisation. IEEE Transactions on Visualization and Computer Graphics 13, 6 (2007), 1216--1223.
[9]
Usama M. Fayyad, Gregory Piatetsky-Shapiro, and Padhraic Smyth. 1996. From data mining to knowledge discovery in databases. AI Magazine 17, 37--54.
[10]
Niclas Feldkamp, Soeren Bergmann, and Steffen Strassburger. 2015. Knowledge discovery in manufacturing simulations. In Proceedings of the 3rd ACM SIGSIM Conference on Principles of Advanced Discrete Simulation (SIGSIM PADS’15). ACM, New York, 3--12.
[11]
Niclas Feldkamp, Soeren Bergmann, and Steffen Strassburger. 2015. Visual analytics of manufacturing simulation data. In Proceedings of the 2015 Winter Simulation Conference. IEEE, Piscataway, N.J., 779--790.
[12]
Niclas Feldkamp, Soeren Bergmann, and Steffen Strassburger. 2017. Online analysis of simulation data with stream-based data mining. In Proceedings of the 2017 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation (SIGSIM-PADS'17). ACM, New York, 241--248.
[13]
Niclas Feldkamp, Soeren Bergmann, Steffen Strassburger, Erik Borsch, Magnus Richter, and Rainer Souren. 2018. Combining data farming and data envelopment analysis for measuring productive efficiency in Manufacturing Simulations. In Proceedings of the 2018 Winter Simulation Conference. IEEE, Piscataway, N.J.
[14]
Niclas Feldkamp, Soeren Bergmann, Steffen Strassburger, and Thomas Schulze. 2017. Knowledge discovery and robustness analysis in manufacturing simulations. In Proceedings of the 2017 Winter Simulation Conference. IEEE.
[15]
Niclas Feldkamp, Soeren Bergmann, Steffen Strassburger, Thomas Schulze, Praneeth Akondi, and Marco Lemessi. 2017. Knowledge discovery in simulation data – A case study for a backhoe assembly line. In Proceedings of the 2017 Winter Simulation Conference. IEEE, 4456--4458.
[16]
Niclas Feldkamp, Sören Bergmann, Steffen Strassburger, and Thomas Schulze. 2016. Knowledge discovery in simulation data: A case study of a gold mining facility. In Proceedings of the 2016 Winter Simulation Conference. IEEE, Piscataway, N. J. 1607--1618.
[17]
Leonardo Feltrin. 2015. KNIME an open source solution for predictive analytics in the geosciences [software and data sets]. IEEE Geoscience and Remote Sensing Magazine 3, 4 (2015), 28--38.
[18]
Philippe J. Giabbanelli. 2010. Impact of complex network properties on routing in backbone networks. In Proceedings of the 2010 IEEE Globecom Workshops // 2010 IEEE Globecom Workshops (GC'10). Workshops: Dec. 5, 2010 to Dec. 10, 2010 in Miami, Florida. IEEE, Piscataway, N.J., 389--393.
[19]
R. R. Hill, J. O. Miller, and G. A. McIntyre. 2001. Applications of discrete event simulation modeling to military problems. In Proceedings of the 2001 Winter Simulation Conference. IEEE, Piscataway, N.J., 780--788.
[20]
Gary Horne, Bernt Åkesson, Ted Meyer. 2014. Data farming in support of NATO. Final Report of Task Group MSG-088. STO technical report, TR-MSG-088. North Atlantic Treaty Organisation, Neuilly-sur-Seine Cedex.
[21]
Gary E. Horne and Ted E. Meyer. 2005. Data farming: Discovering surprise. In Proceedings of the 2005 Winter Simulation Conference. IEEE, Piscataway, N.J., 1082--1087.
[22]
D. Kallfass and T. Schlaak. 2012. NATO MSG-088 case study results to demonstrate the benefit of using data farming for military decision support. In Proceedings of the 2012 Winter Simulation Conference (WSC 2012). IEEE, Piscataway, N. J. 1--12.
[23]
Daniel A. Keim. 2002. Information visualization and visual data mining. IEEE Transactions on Visualization and Computer Graphics 8, 1 (2002), 1--8.
[24]
Daniel A. Keim, Florian Mansmann, Jörn Schneidewind, Jim Thomas, and Hartmut Ziegler. 2008. Visual analytics: Scope and challenges. In Visual Data Mining: Theory, Techniques and Tools for Visual Analytics, Simeon Simoff, Michael H. Boehlen and Arturas Mazeika, Eds. Springer, Berlin.
[25]
Jack P. C. Kleijnen, Susan M. Sanchez, Thomas W. Lucas, and Thomas M. Cioppa. 2005. State-of-the-art review: A user's guide to the brave new world of designing simulation experiments. INFORMS Journal on Computing 17, 3 (2005), 263--289.
[26]
Bum C. Kwon, Janu Verma, Peter J. Haas, and Cagatay Demiralp. 2017. Sampling for scalable visual analytics. IEEE Computer Graphics and Applications 37, 1 (2017), 100--108.
[27]
Averill M. Law. 2003. How to conduct a successful simulation study. In Proceedings of the 2003 Winter Simulation Conference. IEEE, Piscataway, N. J. 66--70.
[28]
Martin Luboschik, Stefan Rybacki, Roland Ewald, Benjamin Schwarze, Heidrun Schumann, and Adelinde M. Uhrmacher. 2012. Interactive visual exploration of simulator accuracy: A case study for stochastic simulation algorithms. In Proceedings of the 2012 Winter Simulation Conference. WSC ’12. IEEE., Piscataway, N.J.
[29]
Martin Luboschik, Christian Tominski, Arne Bittig, Adelinde Uhrmacher, and H. Schumann. 2012. Towards interactive visual analysis of microscopic-level simulation data. In Proceedings of SIGRAD 2012. Interactive Visual Analysis of Data, 81. Linköping University Electronic Press, Linköpings, 91--94.
[30]
Kresimir Matkovic, Denis Gračanin, Mario Jelović, and Helwig Hauser. 2015. Interactive visual analysis of large simulation ensembles. In Proceedings of the 2015 Winter Simulation Conference. IEEE., Piscataway, N.J., 517--528.
[31]
Ana Minanovic, Hrvoje Gabelica, and Zivko Krstic. 2014. Big data and sentiment analysis using KNIME: Online reviews vs. Social Media. In Proceedings of the 37th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO). IEEE, 1464--1468.
[32]
Brian L. Morgan, Harrison C. Schramm, Jerry R. Smith, Thomas W. Lucas, Mary L. McDonald, Paul J. Sánchez, Susan M. Sanchez, and Stephen C. Upton. 2018. Improving U.S. Navy campaign analyses with big data. Interfaces 48, 2 (2018), 130--146.
[33]
Christopher J. Nannini, Jeffrey A. Appleget, and Alejandro S. Hernandez. 2013. Game for peace: Progressive education in peace operations. The Journal of Defense Modeling and Simulation 10, 3 (2013), 283--296.
[34]
Susan M. Sanchez. 2014. Simulation experiments: Better data, not just big data. In Proceedings of the 2014 Winter Simulation Conference. IEEE, Piscataway, N. J. 805--816.
[35]
Susan M. Sanchez, Thomas W. Lucas, Paul J. Sanchez, Christopher J. Nannini, and Hong Wan. 2012. Designs for large‐scale simulation experiments, with applications to defense and homeland security. In Design and Analysis of Experiments: Special Designs and Applications. Klaus Hinkelmann, Ed. John Wiley 8 Sons, Inc., Hoboken, N.J., 413--442.
[36]
Susan M. Sanchez and Hong Wan. 2009. Better than a petaflop: The power of efficient experimental design. In Proceedings of the 2009 Winter Simulation Conference (WSC 2009). (Austin, Texas: 13–16 December 2009). IEEE, Piscataway, N.J., 60--74.
[37]
Johan Schubert, Ronnie Johansson, and Pontus Hörling. 2015. Skewed distribution analysis in simulation-based operation planning. In Proceedings of the 9th Operations Research and Analysis Conference.
[38]
Danielle Soban, David Thornhill, Santosh Salunkhe, and Alastair Long. 2016. Visual analytics as an enabler for manufacturing process decision-making. Procedia CIRP 56, 209--214.
[39]
Zhiwu Tang, Qing Xue, Meng Zhao, and Yang Wei. 2009. Decision tree algorithm for tank damage analysis in combat simulation tests. In Proceedings of the 9th International Conference on Electronic Measurement 8 Instruments (ICEMI 2009), 3-830-3–835.
[40]
Simon J. E. Taylor, Tamas Kiss, Anastasia Anagnostou, Gabor Terstyanszky, Peter Kacsuk, Joris Costes, and Nicola Fantini. 2018. The CloudSME simulation platform and its applications. A Generic Multi-cloud Platform for Developing and Executing Commercial Cloud-based Simulations. Future Generation Computer Systems 88, 524--539.
[41]
Hasan Tercan, Toufik Al Khawli, Urs Eppelt, Christian Büscher, Tobias Meisen, and Sabina Jeschke. 2016. Use of classification techniques to design laser cutting processes. Procedia 5CIRP6 52, 292--297.
[42]
J. J. Thomas and Kristin A. Cook. 2005. Illuminating the Path. Research and Development Agenda for Visual Analytics (1st). IEEE Computer Society, Los Alamitos, California.
[43]
Luxmi Verma, S. Srinivasan, and Varun Sapra. 2014. Integration of rule based and case based reasoning system to support decision making. In Proceedings of the International Conference on Issues and Challenges in Intelligent Computing Techniques (ICICT). IEEE, 106--108.
[44]
Sigrid Wenzel, Jochen Bernhard, and Ulrich Jessen. 2003. Visualization for modeling and simulation: A taxonomy of visualization techniques for simulation in production and logistics. In Proceedings of the 2003 Winter Simulation Conference. IEEE., Piscataway, N.J.
[45]
Mike Wu, Michael C. Hughes, Sonali Parbhoo, Maurizio Zazzi, Volker Roth, and Finale Doshi Velez. 2018. Beyond Sparsity: Tree Regularization of Deep Models for Interpretability. AAAI.
[46]
Zheguang Zhao, Lorenzo de Stefani, Emanuel Zgraggen, Carsten Binnig, Eli Upfal, and Tim Kraska. 2017. Controlling false discoveries during interactive data exploration. In Proceedings of the 2017 ACM International Conference on Management of Data (SIGMOD'17). ACM, New York, 527--540.

Cited By

View all
  • (2024)A Function-Based Behavioral Modeling Method for Air Combat SimulationJournal of Systems Engineering and Electronics10.23919/JSEE.2024.00006835:4(945-954)Online publication date: Aug-2024
  • (2024)Context, Composition, Automation, and Communication: The C2AC Roadmap for Modeling and SimulationACM Transactions on Modeling and Computer Simulation10.1145/367322634:4(1-51)Online publication date: 13-Aug-2024
  • (2024)AsaPy: A Python Library for Aerospace Simulation AnalysisProceedings of the 38th ACM SIGSIM Conference on Principles of Advanced Discrete Simulation10.1145/3615979.3656063(15-24)Online publication date: 24-Jun-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Modeling and Computer Simulation
ACM Transactions on Modeling and Computer Simulation  Volume 30, Issue 4
Special Issue on Toward an Ecosystem of Models and Data
October 2020
116 pages
ISSN:1049-3301
EISSN:1558-1195
DOI:10.1145/3439453
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 November 2020
Online AM: 07 May 2020
Accepted: 01 March 2020
Revised: 01 February 2020
Received: 01 September 2018
Published in TOMACS Volume 30, Issue 4

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Simulation
  2. data farming
  3. data mining
  4. knowledge discovery
  5. visual analytics

Qualifiers

  • Research-article
  • Research
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)62
  • Downloads (Last 6 weeks)0
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)A Function-Based Behavioral Modeling Method for Air Combat SimulationJournal of Systems Engineering and Electronics10.23919/JSEE.2024.00006835:4(945-954)Online publication date: Aug-2024
  • (2024)Context, Composition, Automation, and Communication: The C2AC Roadmap for Modeling and SimulationACM Transactions on Modeling and Computer Simulation10.1145/367322634:4(1-51)Online publication date: 13-Aug-2024
  • (2024)AsaPy: A Python Library for Aerospace Simulation AnalysisProceedings of the 38th ACM SIGSIM Conference on Principles of Advanced Discrete Simulation10.1145/3615979.3656063(15-24)Online publication date: 24-Jun-2024
  • (2023)Knowledge-based and data-driven behavioral modeling techniques in engagement simulationSimulation10.1177/0037549723117712399:10(1069-1089)Online publication date: 1-Oct-2023
  • (2023)From Explainable AI to Explainable Simulation: Using Machine Learning and XAI to understand System RobustnessProceedings of the 2023 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation10.1145/3573900.3591114(96-106)Online publication date: 21-Jun-2023
  • (2023)Automatic Reuse, Adaption, and Execution of Simulation Experiments via Provenance PatternsACM Transactions on Modeling and Computer Simulation10.1145/356492833:1-2(1-27)Online publication date: 28-Feb-2023
  • (2022)An Application of Automated Machine Learning within a Data Farming ProcessProceedings of the Winter Simulation Conference10.5555/3586210.3586378(2013-2024)Online publication date: 11-Dec-2022
  • (2022)Explainable AI for Data Farming Output AnalysisProceedings of the Winter Simulation Conference10.5555/3586210.3586306(1152-1163)Online publication date: 11-Dec-2022
  • (2022)Data Farming und Wissensentdeckung in SimulationsdatenZeitschrift für wirtschaftlichen Fabrikbetrieb10.1515/zwf-2022-1032117:3(144-150)Online publication date: 10-Mar-2022
  • (2022)A Method Using Generative Adversarial Networks for Robustness OptimizationACM Transactions on Modeling and Computer Simulation10.1145/350351132:2(1-22)Online publication date: 4-Mar-2022
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media