Abstract
Smart embedded systems often run sophisticated pattern recognition algorithms and are found in many areas like automotive, sports and medicine. The developer of such a system is often confronted with the accuracy–cost conflict as the resulting system should be as accurate as possible while being able to run on resource constraint hardware. This article introduces a method to support the solution of this design conflict with accuracy–cost reports. These reports compare classification systems regarding their classification rate (accuracy) and the mathematical operations and parameters of the working phase (cost). Our method is used to deduce the specific cost of various popular pattern recognition algorithms and to derive the overall cost of a classification system. We also show how our analysis can be used to estimate the computational cost for specific hardware architectures. A software toolbox to create accuracy–cost reports was implemented to facilitate the automatic classification system comparison with the presented methodology. The software is available for download and as supplementary material. We performed different experiments on synthetic and real-world data to underline the value of this analysis. Accurate and computationally cheap classification systems were easily identified. We were even able to find a better implementation candidate in an existing embedded classification problem. This work is the first step towards a comprehensive support tool for the design of embedded classification systems.
Similar content being viewed by others
References
Council NR (2001) Embedded, Everywhere. National Academies Press, Washington
Marwedel P (2006) Embedded System Design. Springer, New York
Marwedel P (2010) Embedded and Cyber-Physical Systems in a Nutshell. In: Proc. of the Design Automation Conference
Navet N, Simonot-Lion F (2009) Automotive Embedded Systems Handbook. CRC Press, Boca Raton
Dounis AI, Caraiscos C (2009) Advanced Control Systems Engineering for Energy and Comfort Management in a Building Environment - A Review. Renew Sust Everg Rev 13(6):1246–1261
Ferrarini L, Martinez Lastra JL, Martel A, Valentini A, Vyatkin V (2008) Embedded Systems Design in Intelligent Industrial Automation. EURASIP J Embed Syst 2008:194697
Arshak K, Lyons GM, Cunniffe C, Harris J, Clifford S (2003) A Review of Digital Data Acquisition Hardware and Software for a Portable Electronic Nose. Sens Rev 23(4):332–344
Eskofier BM, Oleson M, DiBenedetto C, Hornegger J (2009) Embedded Surface Classification in Digital Sports. Pattern Recogn Lett 30(16):1448–1456
Ghasemzadeh H, Ostadabbas S, Guenterberg E, Pantelopoulos A (2013) Wireless Medical-Embedded Systems: A Review of Signal-Processing Techniques for Classification. IEEE Sensors J 13(2):423–437
Niemann H (1990) Pattern Analysis and Understanding. Springer, New York
Wolf W, Ozer B, Lv T (2002) Smart Cameras as Embedded Systems. Computer 35(9):48–53
Stewart J, Wang CJ (2003) Waveform Recognition of Back-electro Motive Force for Real-time Embedded System Applications Using Support Vector Machines. In: Proc. of the Artificial Neural Networks in Engineering Conference. pp 951–956
Benbasat AY, Paradiso JA (2002) An Inertial Measurement Framework for Gesture Recognition and Applications. In: Revised Papers from the International Gesture Workshop on Gesture and Sign Languages in Human-Computer Interaction. pp 9–20
Englehart K, Hudgins B (2003) A Robust, Real-time Control Scheme for Multifunction Myoelectric Control. IEEE Trans Biomed Eng 50(7):848–854
Zhang X, Liu Y, Zhang F, Ren J, Sun YL, Yang Q, Huang H (2012) On Design and Implementation of Neural-Machine Interface for Artificial Legs. IEEE Trans Ind Informat 8(2):418–429
Benocci M, Farella E, Benini L (2011) A Context-aware Smart Seat. In: Proc. of the 4th IEEE International Workshop on Advances in Sensors and Interfaces (IWASI 2011). pp 104–109
Attenberger A, Buchenrieder K (2014) An Arduino-Simulink-Control System for Modern Hand Protheses. In: Proc. of the 13th International Conference on Artificial Intelligence and Soft Computing (ICAISC 2014). pp 433–444
Rahman AS, Yap MM, Shakaff AY, Ahmad MN, Dahari Z, Ismail Z, Hitam MS (2004) A Microcontroller-based Taste Sensing System for the Verification of Eurycoma Longifolia. Sensor Actuat B Chem 101(1–2):191–198
Garcia-Breijo E, Atkinson J, Gil-Sanchez L, Masot R, Ibanez J, Garrigues J, Glanc M, Laguarda-Miro N, Olguin C (2011) A Comparison Study of Pattern Recognition Algorithms Implemented on a Microcontroller for Use in an Electronic Tongue for Monitoring Drinking Waters. Sensor Actuat A Phys 172(2):570–582
Breijo EG, Pinatti CO, Peris RM, Fillol M, Martínez-Máñez R, Camino JS (2013) TNT Detection Using a Voltammetric Electronic Tongue Based on Neural Networks. Sensor Actuat A Phys 192:1–8
Hacker C, Batliner A, Nöth E (2006) Are you looking at me, are you talking with me: Multimodal Classification of the Focus of Attention. In: Proc. of the 9th International Conference on Text, Speech and Dialogue (TSD 2006), vol. 4188. LNCS, pp 581–588
Lin CT, Chen YC, Huang TY, Chiu TT, Ko LW, Liang SF, Hsieh HY, Hsu SH, Duann JR (2008) Development of Wireless Brain Computer Interface With Embedded Multitask Scheduling and its Application on Real-Time Driverś Drowsiness Detection and Warning. IEEE Trans Biomed Eng 55(5):1582–1591
Braojos R, Ansaloni G, Atienza D (2013) A Methodology for Embedded Classification of Heartbeats Using Random Projections. In: Proc. of the Design, Automation Test in Europe Conference Exhibition (DATE 2013). pp 899–904
Kao T, Lin C, Wang J (2009) Development of a Portable Activity Detector for Daily Activity Recognition. In: IEEE International Symposium on Industrial Electronics (ISIE 2009). pp 115–120
Boni A, Pianegiani F, Petri D (2007) Low-Power and Low-Cost Implementation of SVMs for Smart Sensors. IEEE Trans Instrum Meas 56(1):39–44
Anguita D, Carlino L, Ghio A, Ridella S (2011) A FPGA Core Generator for Embedded Classification Systems. J Circuit Syst Comp 20(2):263–282
Pedersen R, Schoeberl M (2006) An Embedded Support Vector Machine. In: Proc. of the International Workshop on Intelligent Solutions in Embedded Systems. pp 1–11
Bajaj N, Chiu GT, Allebach JP (2014) Reduction of Memory Footprint and Computation Time for Embedded Support Vector Machine (SVM) by Kernel Expansion and Consolidation. In: Proc. of the IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2014). pp 1–6
Charniya NN (2013) Design of near-optimal classifier using multi-layer perceptron neural networks for intelligent sensors. Int J Model Optim 3(1):56–60
Ortega-Zamorano F, Subirats JL, Jerez JM, Molina I, Franco L (2013) Implementation of the C-Mantec Neural Network Constructive Algorithm in an Arduino UNO Microcontroller. In: Advances in Computational Intelligence. Springer, pp 80–87
Övünç P, Tülay Y (2010) FPGA implementation of a general regression neural network: an embedded pattern classification system. Digit Signal Process 20(3):881–886
Zwartjes A, Havinga P, Smit G, Hurink J (2012) Using Unsupervised Learning to Improve the Naive Bayes Classifier for Wireless Sensor Networks. In: Proc. of the International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies (UBICOMM 2012). pp 71–76
Sohn S (1999) Meta analysis of classification algorithms for pattern recognition. IEEE Trans Pattern Anal Mach Intell 21(11):1137–1144
Reif M, Shafait F, Goldstein M, Breuel T, Dengel A (2014) Automatic classifier selection for non-experts. Pattern Anal Appl 17(1):83–96
Aha DW, Kibler D, Albert MK (1991) Instance-based learning algorithms. Mach Learn 6(1):37–66
Jain A, Nikovski D (2008) Incremental exemplar learning schemes for classification on embedded devices. Mach Learn 72(3):189–203
García-Durán R, Fernández F, Borrajo D (2012) A prototype-based method for classification with time constraints: a case study on automated planning. Pattern Anal Appl 15(3):261–277
Buciluǎ C, Caruana R, Niculescu-Mizil A (2006) Model Compression. In: Proc. of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Philadelphia, pp 535–541
Tan M (1993) Cost-sensitive learning of classification knowledge and its applications in robotics. Mach Learn 13(1):7–33
Iswandy K, Koenig A (2006) Feature selection with acquisition cost for optimizing sensor system design. Adv Radio Sci 4(7):135–141
Weiss Y, Elovici Y, Rokach LL (2013) The CASH algorithm-cost-sensitive attribute selection using histograms. Inform Sci 222:247–268
Ji S, Carin L (2007) Cost-sensitive feature acquisition and classification. Pattern Recogn 40(5):1474–1485
Cebe M, Gunduz-Demir C (2010) Qualitative test-cost sensitive classification. Pattern Recogn Lett 31(13):2043–2051
Paclík P, Duin RP, van Kempen GM, Kohlus R (2002) On Feature Selection with Measurement Cost and Grouped Features. In: Structural, Syntactic, and Statistical Pattern Recognition. Springer, pp 461–469
Ahmadi A, Dehzangi O, Jafari R (2012) Brain-Computer Interface Signal Processing Algorithms: A Computational Cost vs. Accuracy Analysis for Wearable Computers. In: Proc. of the Ninth International Conference on Wearable and Implantable Body Sensor Networks. pp 40–45
Christ P, Sievers G, Einhaus J, Jungeblut T, Porrmann M, Ruckert U (2013) Pareto-optimal Signal Processing on Low-power Microprocessors. In: Proc. of IEEE Sensors Conference. pp 1–4
Witten IH, Frank E, Hall MA (2011) Data Mining: Practical Machine Learning Tools and Techniques, 3rd edn. Morgan Kaufmann, Burlington
Cormen TH, Leiserson CE, Rivest RL, Stein C (2001) Introduction to Algorithms. MIT press, Cambridge
Puschner P, Burns A (2000) A review of worst-case execution-time analysis. Int J Time Critical Comput Syst 18(2):115–128
Wilhelm R, Engblom J, Ermedahl A, Holsti N, Thesing S, Whalley D, Bernat G, Ferdinand C, Heckmann R, Mitra T, Mueller F, Puaut I, Puschner P, Staschulat J, Stenstr-Üm P (2008) The worst-case execution-time problem—overview of methods and survey of tools. ACM T Embed Comput S 7(3):36
Duda RO, Hart PE, Stork DG (2000) Pattern Classification, vol 2. Wiley-Interscience, Hoboken
Kaporin I (1999) A practical algorithm for faster matrix multiplication. Numer Linear Algebr 6(8):687–700
Vetterli M, Nussbaumer HJ (1984) Simple FFT and DCT algorithms with reduced number of operations. Signal Process 6(4):267–278
Chau L, Siu W (1995) Direct formulation for the realization of discrete cosine transform using recursive structure. IEEE Trans Circuits Syst II Analog Digit Signal Process 42(1):50–52
Gabutti B (1985) An algorithm for computing generalized Euler’s transformations of series. Computing 34(2):107–116
Hwang S, Kim W (2006) Fast and efficient method for computing ART. IEEE Trans Image Process 15(1):112–117
Knuth D (1998) The art of computer programming. Addison-Wesley, Boston
Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten I (2009) The WEKA data mining software: An update. SIGKDD Explor 11(1):10–18
Chang C, Lin C (2011) LIBSVM: A library for support vector machines. ACM TIST 2(3):1–27
Polikar R (2007) Bootstrap-inspired techniques in computational intelligence: ensemble of classifiers, incremental learning, data fusion and missing features. IEEE Signal Process Mag 24(4):59–72
Quinlan R (1993) C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, Burlington
Frank E, Wang Y, Inglis S, Holmes G, Witten I (1998) Using model trees for classification. Mach Learn 32(1):63–76
Theodoridis S, Koutroumbas K (2008) Pattern Recognition. Academic Press, Waltham
Frank E, Witten I (1998) Generating Accurate Rule Sets Without Global Optimization. In: Proc. of the Fifteenth International Conference on Machine Learning. pp 144–151
Vapnik VN (1995) The nature of statistical learning theory. Springer, New York
Schölkopf B, Smola AJ (2001) Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge
Kressel U (1998) Pairwise classification and support vector machines. In: Schölkopf B, Burges C, Smola A (eds) Advances in Kernel Methods. MIT Press, pp 255–268
Microchip Technology Inc (1997) AN660 floating point math functions. http://ww1.microchip.com/downloads/en/AppNotes/00660.pdf. Accessed 09 July 2015
Texas Instruments Inc (1999) The MSP430 hardware multiplier. http://www.ti.com/lit/an/slaa042/slaa042.pdf. Accessed 09 July 2015
ARM Ltd (2010) Cortex-M3 technical reference manual. http://infocenter.arm.com/help/topic/com.arm.doc.ddi0337e/DDI0337E_cortex_m3_r1p1_trm.pdf. Accessed 09 July 2015
Texas Instruments Inc (2000) MSP430 family mixed-signal microcontroller application reports. http://www.ti.com/lit/an/slaa024/slaa024.pdf. Accessed 09 July 2015
Iordache C, Tang PT (2003) An Overview of Floating-Point Support and Math Library on the Intel XScale Architecture. In: Proc. of the 16th IEEE Symposium on Computer Arithmetic (ARITH-16 2003). pp 122–128
Texas Instruments Inc (2006) Efficient multiplication and division using MSP430. http://www.ti.com.cn/cn/lit/an/slaa329/slaa329.pdf. Accessed 09 July 2015
Cypress Semiconductor Corp (2008) ImageCraft assembly language guide. http://www.cypress.com/?docID=29030. Accessed 09 July 2015
Cypress Semiconductor Corp (2014) AN2032 PSoC 1 fast unsigned multiplication algorithms. http://www.cypress.com/?docID=50218. Accessed 09 July 2015
Micro Digital Inc (2014) GoFast for Thumb-2 and IAR EWARM. http://www.smxrtos.com/ussw/gofast/gofast_arm_iar.html. Accessed 01 Sept 2014
Goldberger AL, Amaral LAN, Glass L, Hausdorff JM, Ivanov PC, Mark RG, Mietus JE, Moody GB, Peng C, Stanley HE (2000) PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation 101(23):e215–e220
Hausdorff JM, Ladin Z, Wei JY (1995) Footswitch system for measurement of the temporal parameters of gait. J Biomech 28(3):347–351
Hausdorff JM, Mitchell SL, Firtion R, Peng CK, Cudkowicz ME, Wei JY, Goldberger AL (1997) Altered fractal dynamics of gait: reduced stride-interval correlations with aging and Huntington’s Disease. J Appl Physiol 82:262–269
Bache K, Lichman M (2013) UCI machine learning repository. http://archive.ics.uci.edu/ml. Accessed 3 July 2013
Mangasarian OL, Street WN, Wolberg WH (1995) Breast cancer diagnosis and prognosis via linear programming. Oper Res 43(4):570–577
Chen X (2003) An improved branch and bound algorithm for feature selection. Pattern Recogn Lett 24(12):1925–1933
Acknowledgments
Financial support was provided by the adidas AG Herzogenaurach, the Embedded Systems Institute (ESI) Erlangen, the Deutsche Telekom Stiftung Bonn, the Bavarian Ministry for Economic Affairs, Infrastructure, Transport and Technology and the European Fund for Re-gional Development.
Author information
Authors and Affiliations
Corresponding author
Additional information
U. Jensen and P. Kugler equally contributed to this work.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Jensen, U., Kugler, P., Ring, M. et al. Approaching the accuracy–cost conflict in embedded classification system design. Pattern Anal Applic 19, 839–855 (2016). https://doi.org/10.1007/s10044-015-0503-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10044-015-0503-1