skip to main content
10.1145/3578244.3583732acmconferencesArticle/Chapter ViewAbstractPublication PagesicpeConference Proceedingsconference-collections
research-article

A Method to Evaluate the Performance of Predictors in Cyber-Physical Systems

Published: 15 April 2023 Publication History

Abstract

Cyber-Physical Systems (CPS) rely on sensing to control and optimize their operation. Nevertheless, sensing itself is prone to errors that can originate at several stages, from sampling to communication. In this context, several systems adopt multivariate predictors to assess the quality of the sensed data, to replace data from faulty sensors, or to derive variables that cannot be directly sensed. These predictors are often evaluated based on their accuracy and computing demands, however, such evaluations often do not consider the system's architecture from a broader perspective, ignoring the way components are interconnected and how they cascade as inputs of other Machine Learning (ML) models. In this work, we introduce a method to evaluate the performance of interdependent predictors based on the stability of the estimation error dynamics in faulty scenarios. The proposed method estimates the ability of a predictor to produce accurate predictions while accounting for the impacts of cascading predicted values as its inputs. The prediction correctness is estimated based solely on information acquired during the training of the multivariate predictors and mathematical properties of the ML activation functions. The proposed method is evaluated with a meaningful dataset in the scope of monitoring and control of a Cyber-Physical System, and the evaluation demonstrates the ability of the proposed method to account for the interdependence of data predictors.

References

[1]
Angelo Alessandri, Marco Baglietto, Giorgio Battistelli, and Victor Zavala. 2010. Advances in moving horizon estimation for nonlinear systems. In 49th IEEE Conf. on Decision and Control (CDC). IEEE. https://doi.org/10.1109/cdc.2010.5718126
[2]
Martin Arlitt, Manish Marwah, Gowtham Bellala, Amip Shah, Jeff Healey, and Ben Vandiver. 2015. IoTAbench. In Proc. of the 6th ACM/SPEC Int'l Conf. on Performance Engr. ACM. https://doi.org/10.1145/2668930.2688055
[3]
Chengyu Cao and Naira Hovakimyan. 2007. Novel ${\cal L}_{1}$ Neural Network Adaptive Control Architecture With Guaranteed Transient Performance. IEEE Trans. on Neural Networks 18, 4 (July 2007), 1160--1171. https://doi.org/10.1109/tnn.2007.899197
[4]
C. L. Philip Chen, Yan-Jun Liu, and Guo-Xing Wen. 2014. Fuzzy Neural Network- Based Adaptive Control for a Class of Uncertain Nonlinear Stochastic Systems. IEEE Trans. on Cybernetics 44, 5 (May 2014), 583--593. https://doi.org/10.1109/tcyb.2013.2262935
[5]
Adel A. Ghobbar and Chris H. Friend. 2003. Evaluation of forecasting methods for intermittent parts demand in the field of aviation: a predictive model. Computers & Operations Research 30, 14 (Dec. 2003), 2097--2114. https://doi.org/10.1016/s0305-0548(02)00125--9
[6]
Nikolai Helwig, Eliseo Pignanelli, and Andreas Schütze. 2015. Condition monitoring of a complex hydraulic system using multivariate statistics. IEEE Int'l Instrumentation and Measurement Technology Conf. (I2MTC) Proc. (2015), 210--215.
[7]
Benjamin Karg and Sergio Lucia. 2021. Approximate moving horizon estimation and robust nonlinear model predictive control via deep learning. Computers & Chemical Engr. 148 (May 2021), 107266. https://doi.org/10.1016/j.compchemeng.2021.107266
[8]
Qinyuan Liu and Zidong Wang. 2021. Moving-Horizon Estimation for Linear Dynamic Networks With Binary Encoding Schemes. IEEE Trans. on Automatic Control 66, 4 (April 2021), 1763--1770. https://doi.org/10.1109/tac.2020.2996579
[9]
Vittoriano Muttillo, Paolo Giammatteo, and Vincenzo Stoico. 2021. Statement- Level Timing Estimation for Embedded System Design Using Machine Learning Techniques. In Proc. of the ACM/SPEC Int'l Conf. on Performance Engr. ACM. https://doi.org/10.1145/3427921.3450258
[10]
C. Nicolis, Rui A. P. Perdigao, and S. Vannitsem. 2009. Dynamics of Prediction Errors under the Combined Effect of Initial Condition and Model Errors. Journal of the Atmospheric Sciences 66, 3 (March 2009), 766--778. https://doi.org/10.1175/2008jas2781.1
[11]
Alan V. Oppenheim, Alan S. Willsky, and S. Hamid Nawab. 1996. Signals & Systems (2nd Ed.). Prentice-Hall, Inc., USA.
[12]
Björn F. Postema, Tobias Van Damme, Claudio De Persis, Pietro Tesi, and Boudewijn R. Haverkort. 2018. Combining Energy Saving Techniques in Data Centres using Model-Based Analysis. In Companion of the 2018 ACM/SPEC Int'l Conf. on Performance Engr. ACM. https://doi.org/10.1145/3185768.3186310
[13]
Michael B. Richman, Theodore B. Trafalis, and Indra Adrianto. 2009. Missing Data Imputation Through Machine Learning Algorithms. Springer Netherlands, Dordrecht, 153--169. https://doi.org/10.1007/978--1--4020--9119--3_7
[14]
Roberto M. Scheffel and Antnio A. Frhlich. 2018. WSN Data Confidence Attribution Using Predictors. In 2018 Eighth Latin-American Symp. on Dependable Computing (LADC). IEEE. https://doi.org/10.1109/ladc.2018.00025
[15]
Roberto M Scheffel and Antônio A Fröhlich. 2019. Increasing sensor reliability through confidence attribution. Journal of the Brazilian Computer Soc. 25, 1 (2019), 1--20.
[16]
Guido Schillaci, Alejandra Ciria, and Bruno Lara. 2020. Tracking Emotions: Intrinsic Motivation Grounded on Multi - Level Prediction Error Dynamics. In 2020 Joint IEEE 10th Int'l Conf. on Development and Learning and Epigenetic Robotics (ICDL-EpiRob). IEEE. https://doi.org/10.1109/icdl-epirob48136.2020.9278106
[17]
Connie U. Smith. 2020. Software Performance Antipatterns in Cyber-Physical Systems. In Proc. of the ACM/SPEC Int'l Conf. on Performance Engr. ACM. https://doi.org/10.1145/3358960.3379138
[18]
Chunsheng Yang, Yanni Zou, Jie Liu, and Kyle R Mulligan. 2020. Predictive Model Evaluation for PHM. Int'l Journal of Prognostics and Health Management 5, 2 (Nov. 2020). https://doi.org/10.36001/ijphm.2014.v5i2.2238

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICPE '23: Proceedings of the 2023 ACM/SPEC International Conference on Performance Engineering
April 2023
244 pages
ISBN:9798400700682
DOI:10.1145/3578244
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 April 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. estimation error dynamics
  2. machine learning
  3. predictors
  4. stability analysis

Qualifiers

  • Research-article

Funding Sources

  • Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES)

Conference

ICPE '23

Acceptance Rates

ICPE '23 Paper Acceptance Rate 15 of 46 submissions, 33%;
Overall Acceptance Rate 252 of 851 submissions, 30%

Upcoming Conference

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 68
    Total Downloads
  • Downloads (Last 12 months)5
  • Downloads (Last 6 weeks)0
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media