Skip to main content
SpringerLink
Log in

Approaching the accuracy–cost conflict in embedded classification system design

  • Industrial and Commercial Application
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

Smart embedded systems often run sophisticated pattern recognition algorithms and are found in many areas like automotive, sports and medicine. The developer of such a system is often confronted with the accuracy–cost conflict as the resulting system should be as accurate as possible while being able to run on resource constraint hardware. This article introduces a method to support the solution of this design conflict with accuracy–cost reports. These reports compare classification systems regarding their classification rate (accuracy) and the mathematical operations and parameters of the working phase (cost). Our method is used to deduce the specific cost of various popular pattern recognition algorithms and to derive the overall cost of a classification system. We also show how our analysis can be used to estimate the computational cost for specific hardware architectures. A software toolbox to create accuracy–cost reports was implemented to facilitate the automatic classification system comparison with the presented methodology. The software is available for download and as supplementary material. We performed different experiments on synthetic and real-world data to underline the value of this analysis. Accurate and computationally cheap classification systems were easily identified. We were even able to find a better implementation candidate in an existing embedded classification problem. This work is the first step towards a comprehensive support tool for the design of embedded classification systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. http://www.tinyurl.com/ecstproject.

References

  1. Council NR (2001) Embedded, Everywhere. National Academies Press, Washington

  2. Marwedel P (2006) Embedded System Design. Springer, New York

    MATH  Google Scholar 

  3. Marwedel P (2010) Embedded and Cyber-Physical Systems in a Nutshell. In: Proc. of the Design Automation Conference

  4. Navet N, Simonot-Lion F (2009) Automotive Embedded Systems Handbook. CRC Press, Boca Raton

    Google Scholar 

  5. Dounis AI, Caraiscos C (2009) Advanced Control Systems Engineering for Energy and Comfort Management in a Building Environment - A Review. Renew Sust Everg Rev 13(6):1246–1261

    Article  Google Scholar 

  6. Ferrarini L, Martinez Lastra JL, Martel A, Valentini A, Vyatkin V (2008) Embedded Systems Design in Intelligent Industrial Automation. EURASIP J Embed Syst 2008:194697

  7. Arshak K, Lyons GM, Cunniffe C, Harris J, Clifford S (2003) A Review of Digital Data Acquisition Hardware and Software for a Portable Electronic Nose. Sens Rev 23(4):332–344

    Article  Google Scholar 

  8. Eskofier BM, Oleson M, DiBenedetto C, Hornegger J (2009) Embedded Surface Classification in Digital Sports. Pattern Recogn Lett 30(16):1448–1456

    Article  Google Scholar 

  9. Ghasemzadeh H, Ostadabbas S, Guenterberg E, Pantelopoulos A (2013) Wireless Medical-Embedded Systems: A Review of Signal-Processing Techniques for Classification. IEEE Sensors J 13(2):423–437

    Article  Google Scholar 

  10. Niemann H (1990) Pattern Analysis and Understanding. Springer, New York

    Book  MATH  Google Scholar 

  11. Wolf W, Ozer B, Lv T (2002) Smart Cameras as Embedded Systems. Computer 35(9):48–53

    Article  Google Scholar 

  12. Stewart J, Wang CJ (2003) Waveform Recognition of Back-electro Motive Force for Real-time Embedded System Applications Using Support Vector Machines. In: Proc. of the Artificial Neural Networks in Engineering Conference. pp 951–956

  13. Benbasat AY, Paradiso JA (2002) An Inertial Measurement Framework for Gesture Recognition and Applications. In: Revised Papers from the International Gesture Workshop on Gesture and Sign Languages in Human-Computer Interaction. pp 9–20

  14. Englehart K, Hudgins B (2003) A Robust, Real-time Control Scheme for Multifunction Myoelectric Control. IEEE Trans Biomed Eng 50(7):848–854

    Article  Google Scholar 

  15. Zhang X, Liu Y, Zhang F, Ren J, Sun YL, Yang Q, Huang H (2012) On Design and Implementation of Neural-Machine Interface for Artificial Legs. IEEE Trans Ind Informat 8(2):418–429

    Article  Google Scholar 

  16. Benocci M, Farella E, Benini L (2011) A Context-aware Smart Seat. In: Proc. of the 4th IEEE International Workshop on Advances in Sensors and Interfaces (IWASI 2011). pp 104–109

  17. Attenberger A, Buchenrieder K (2014) An Arduino-Simulink-Control System for Modern Hand Protheses. In: Proc. of the 13th International Conference on Artificial Intelligence and Soft Computing (ICAISC 2014). pp 433–444

  18. Rahman AS, Yap MM, Shakaff AY, Ahmad MN, Dahari Z, Ismail Z, Hitam MS (2004) A Microcontroller-based Taste Sensing System for the Verification of Eurycoma Longifolia. Sensor Actuat B Chem 101(1–2):191–198

    Article  Google Scholar 

  19. Garcia-Breijo E, Atkinson J, Gil-Sanchez L, Masot R, Ibanez J, Garrigues J, Glanc M, Laguarda-Miro N, Olguin C (2011) A Comparison Study of Pattern Recognition Algorithms Implemented on a Microcontroller for Use in an Electronic Tongue for Monitoring Drinking Waters. Sensor Actuat A Phys 172(2):570–582

    Article  Google Scholar 

  20. Breijo EG, Pinatti CO, Peris RM, Fillol M, Martínez-Máñez R, Camino JS (2013) TNT Detection Using a Voltammetric Electronic Tongue Based on Neural Networks. Sensor Actuat A Phys 192:1–8

    Article  Google Scholar 

  21. Hacker C, Batliner A, Nöth E (2006) Are you looking at me, are you talking with me: Multimodal Classification of the Focus of Attention. In: Proc. of the 9th International Conference on Text, Speech and Dialogue (TSD 2006), vol. 4188. LNCS, pp 581–588

  22. Lin CT, Chen YC, Huang TY, Chiu TT, Ko LW, Liang SF, Hsieh HY, Hsu SH, Duann JR (2008) Development of Wireless Brain Computer Interface With Embedded Multitask Scheduling and its Application on Real-Time Driverś Drowsiness Detection and Warning. IEEE Trans Biomed Eng 55(5):1582–1591

    Article  Google Scholar 

  23. Braojos R, Ansaloni G, Atienza D (2013) A Methodology for Embedded Classification of Heartbeats Using Random Projections. In: Proc. of the Design, Automation Test in Europe Conference Exhibition (DATE 2013). pp 899–904

  24. Kao T, Lin C, Wang J (2009) Development of a Portable Activity Detector for Daily Activity Recognition. In: IEEE International Symposium on Industrial Electronics (ISIE 2009). pp 115–120

  25. Boni A, Pianegiani F, Petri D (2007) Low-Power and Low-Cost Implementation of SVMs for Smart Sensors. IEEE Trans Instrum Meas 56(1):39–44

    Article  Google Scholar 

  26. Anguita D, Carlino L, Ghio A, Ridella S (2011) A FPGA Core Generator for Embedded Classification Systems. J Circuit Syst Comp 20(2):263–282

    Article  Google Scholar 

  27. Pedersen R, Schoeberl M (2006) An Embedded Support Vector Machine. In: Proc. of the International Workshop on Intelligent Solutions in Embedded Systems. pp 1–11

  28. Bajaj N, Chiu GT, Allebach JP (2014) Reduction of Memory Footprint and Computation Time for Embedded Support Vector Machine (SVM) by Kernel Expansion and Consolidation. In: Proc. of the IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2014). pp 1–6

  29. Charniya NN (2013) Design of near-optimal classifier using multi-layer perceptron neural networks for intelligent sensors. Int J Model Optim 3(1):56–60

    Article  Google Scholar 

  30. Ortega-Zamorano F, Subirats JL, Jerez JM, Molina I, Franco L (2013) Implementation of the C-Mantec Neural Network Constructive Algorithm in an Arduino UNO Microcontroller. In: Advances in Computational Intelligence. Springer, pp 80–87

  31. Övünç P, Tülay Y (2010) FPGA implementation of a general regression neural network: an embedded pattern classification system. Digit Signal Process 20(3):881–886

    Article  Google Scholar 

  32. Zwartjes A, Havinga P, Smit G, Hurink J (2012) Using Unsupervised Learning to Improve the Naive Bayes Classifier for Wireless Sensor Networks. In: Proc. of the International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies (UBICOMM 2012). pp 71–76

  33. Sohn S (1999) Meta analysis of classification algorithms for pattern recognition. IEEE Trans Pattern Anal Mach Intell 21(11):1137–1144

    Article  Google Scholar 

  34. Reif M, Shafait F, Goldstein M, Breuel T, Dengel A (2014) Automatic classifier selection for non-experts. Pattern Anal Appl 17(1):83–96

    Article  MathSciNet  Google Scholar 

  35. Aha DW, Kibler D, Albert MK (1991) Instance-based learning algorithms. Mach Learn 6(1):37–66

    Google Scholar 

  36. Jain A, Nikovski D (2008) Incremental exemplar learning schemes for classification on embedded devices. Mach Learn 72(3):189–203

    Article  MathSciNet  Google Scholar 

  37. García-Durán R, Fernández F, Borrajo D (2012) A prototype-based method for classification with time constraints: a case study on automated planning. Pattern Anal Appl 15(3):261–277

    Article  MathSciNet  Google Scholar 

  38. Buciluǎ C, Caruana R, Niculescu-Mizil A (2006) Model Compression. In: Proc. of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Philadelphia, pp 535–541

  39. Tan M (1993) Cost-sensitive learning of classification knowledge and its applications in robotics. Mach Learn 13(1):7–33

    Google Scholar 

  40. Iswandy K, Koenig A (2006) Feature selection with acquisition cost for optimizing sensor system design. Adv Radio Sci 4(7):135–141

    Article  Google Scholar 

  41. Weiss Y, Elovici Y, Rokach LL (2013) The CASH algorithm-cost-sensitive attribute selection using histograms. Inform Sci 222:247–268

    Article  MathSciNet  Google Scholar 

  42. Ji S, Carin L (2007) Cost-sensitive feature acquisition and classification. Pattern Recogn 40(5):1474–1485

    Article  MATH  Google Scholar 

  43. Cebe M, Gunduz-Demir C (2010) Qualitative test-cost sensitive classification. Pattern Recogn Lett 31(13):2043–2051

    Article  Google Scholar 

  44. Paclík P, Duin RP, van Kempen GM, Kohlus R (2002) On Feature Selection with Measurement Cost and Grouped Features. In: Structural, Syntactic, and Statistical Pattern Recognition. Springer, pp 461–469

  45. Ahmadi A, Dehzangi O, Jafari R (2012) Brain-Computer Interface Signal Processing Algorithms: A Computational Cost vs. Accuracy Analysis for Wearable Computers. In: Proc. of the Ninth International Conference on Wearable and Implantable Body Sensor Networks. pp 40–45

  46. Christ P, Sievers G, Einhaus J, Jungeblut T, Porrmann M, Ruckert U (2013) Pareto-optimal Signal Processing on Low-power Microprocessors. In: Proc. of IEEE Sensors Conference. pp 1–4

  47. Witten IH, Frank E, Hall MA (2011) Data Mining: Practical Machine Learning Tools and Techniques, 3rd edn. Morgan Kaufmann, Burlington

  48. Cormen TH, Leiserson CE, Rivest RL, Stein C (2001) Introduction to Algorithms. MIT press, Cambridge

    MATH  Google Scholar 

  49. Puschner P, Burns A (2000) A review of worst-case execution-time analysis. Int J Time Critical Comput Syst 18(2):115–128

    Google Scholar 

  50. Wilhelm R, Engblom J, Ermedahl A, Holsti N, Thesing S, Whalley D, Bernat G, Ferdinand C, Heckmann R, Mitra T, Mueller F, Puaut I, Puschner P, Staschulat J, Stenstr-Üm P (2008) The worst-case execution-time problem—overview of methods and survey of tools. ACM T Embed Comput S 7(3):36

    Google Scholar 

  51. Duda RO, Hart PE, Stork DG (2000) Pattern Classification, vol 2. Wiley-Interscience, Hoboken

    MATH  Google Scholar 

  52. Kaporin I (1999) A practical algorithm for faster matrix multiplication. Numer Linear Algebr 6(8):687–700

    Article  MathSciNet  MATH  Google Scholar 

  53. Vetterli M, Nussbaumer HJ (1984) Simple FFT and DCT algorithms with reduced number of operations. Signal Process 6(4):267–278

    Article  MathSciNet  MATH  Google Scholar 

  54. Chau L, Siu W (1995) Direct formulation for the realization of discrete cosine transform using recursive structure. IEEE Trans Circuits Syst II Analog Digit Signal Process 42(1):50–52

    Article  Google Scholar 

  55. Gabutti B (1985) An algorithm for computing generalized Euler’s transformations of series. Computing 34(2):107–116

    Article  MathSciNet  MATH  Google Scholar 

  56. Hwang S, Kim W (2006) Fast and efficient method for computing ART. IEEE Trans Image Process 15(1):112–117

    Article  Google Scholar 

  57. Knuth D (1998) The art of computer programming. Addison-Wesley, Boston

    MATH  Google Scholar 

  58. Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten I (2009) The WEKA data mining software: An update. SIGKDD Explor 11(1):10–18

    Article  Google Scholar 

  59. Chang C, Lin C (2011) LIBSVM: A library for support vector machines. ACM TIST 2(3):1–27

    Article  Google Scholar 

  60. Polikar R (2007) Bootstrap-inspired techniques in computational intelligence: ensemble of classifiers, incremental learning, data fusion and missing features. IEEE Signal Process Mag 24(4):59–72

    Article  Google Scholar 

  61. Quinlan R (1993) C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, Burlington

  62. Frank E, Wang Y, Inglis S, Holmes G, Witten I (1998) Using model trees for classification. Mach Learn 32(1):63–76

    Article  MATH  Google Scholar 

  63. Theodoridis S, Koutroumbas K (2008) Pattern Recognition. Academic Press, Waltham

    MATH  Google Scholar 

  64. Frank E, Witten I (1998) Generating Accurate Rule Sets Without Global Optimization. In: Proc. of the Fifteenth International Conference on Machine Learning. pp 144–151

  65. Vapnik VN (1995) The nature of statistical learning theory. Springer, New York

    Book  MATH  Google Scholar 

  66. Schölkopf B, Smola AJ (2001) Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge

    Google Scholar 

  67. Kressel U (1998) Pairwise classification and support vector machines. In: Schölkopf B, Burges C, Smola A (eds) Advances in Kernel Methods. MIT Press, pp 255–268

  68. Microchip Technology Inc (1997) AN660 floating point math functions. http://ww1.microchip.com/downloads/en/AppNotes/00660.pdf. Accessed 09 July 2015

  69. Texas Instruments Inc (1999) The MSP430 hardware multiplier. http://www.ti.com/lit/an/slaa042/slaa042.pdf. Accessed 09 July 2015

  70. ARM Ltd (2010) Cortex-M3 technical reference manual. http://infocenter.arm.com/help/topic/com.arm.doc.ddi0337e/DDI0337E_cortex_m3_r1p1_trm.pdf. Accessed 09 July 2015

  71. Texas Instruments Inc (2000) MSP430 family mixed-signal microcontroller application reports. http://www.ti.com/lit/an/slaa024/slaa024.pdf. Accessed 09 July 2015

  72. Iordache C, Tang PT (2003) An Overview of Floating-Point Support and Math Library on the Intel XScale Architecture. In: Proc. of the 16th IEEE Symposium on Computer Arithmetic (ARITH-16 2003). pp 122–128

  73. Texas Instruments Inc (2006) Efficient multiplication and division using MSP430. http://www.ti.com.cn/cn/lit/an/slaa329/slaa329.pdf. Accessed 09 July 2015

  74. Cypress Semiconductor Corp (2008) ImageCraft assembly language guide. http://www.cypress.com/?docID=29030. Accessed 09 July 2015

  75. Cypress Semiconductor Corp (2014) AN2032 PSoC 1 fast unsigned multiplication algorithms. http://www.cypress.com/?docID=50218. Accessed 09 July 2015

  76. Micro Digital Inc (2014) GoFast for Thumb-2 and IAR EWARM. http://www.smxrtos.com/ussw/gofast/gofast_arm_iar.html. Accessed 01 Sept 2014

  77. Goldberger AL, Amaral LAN, Glass L, Hausdorff JM, Ivanov PC, Mark RG, Mietus JE, Moody GB, Peng C, Stanley HE (2000) PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation 101(23):e215–e220

    Article  Google Scholar 

  78. Hausdorff JM, Ladin Z, Wei JY (1995) Footswitch system for measurement of the temporal parameters of gait. J Biomech 28(3):347–351

    Article  Google Scholar 

  79. Hausdorff JM, Mitchell SL, Firtion R, Peng CK, Cudkowicz ME, Wei JY, Goldberger AL (1997) Altered fractal dynamics of gait: reduced stride-interval correlations with aging and Huntington’s Disease. J Appl Physiol 82:262–269

    Google Scholar 

  80. Bache K, Lichman M (2013) UCI machine learning repository. http://archive.ics.uci.edu/ml. Accessed 3 July 2013

  81. Mangasarian OL, Street WN, Wolberg WH (1995) Breast cancer diagnosis and prognosis via linear programming. Oper Res 43(4):570–577

    Article  MathSciNet  MATH  Google Scholar 

  82. Chen X (2003) An improved branch and bound algorithm for feature selection. Pattern Recogn Lett 24(12):1925–1933

    Article  Google Scholar 

Download references

Acknowledgments

Financial support was provided by the adidas AG Herzogenaurach, the Embedded Systems Institute (ESI) Erlangen, the Deutsche Telekom Stiftung Bonn, the Bavarian Ministry for Economic Affairs, Infrastructure, Transport and Technology and the European Fund for Re-gional Development.

Author information

Author notes

    Authors and Affiliations

    1. Digital Sports Group, Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Haberstr. 2, 91058, Erlangen, Germany

      Ulf Jensen, Patrick Kugler, Matthias Ring & Bjoern M. Eskofier

    Authors
    1. Ulf Jensen
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Patrick Kugler
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Matthias Ring
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Bjoern M. Eskofier
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Ulf Jensen.

    Additional information

    U. Jensen and P. Kugler equally contributed to this work.

    Electronic supplementary material

    Below is the link to the electronic supplementary material.

    Supplementary material 1 (pdf 641 KB)

    Rights and permissions

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Jensen, U., Kugler, P., Ring, M. et al. Approaching the accuracy–cost conflict in embedded classification system design. Pattern Anal Applic 19, 839–855 (2016). https://doi.org/10.1007/s10044-015-0503-1

    Download citation

    • Received:

    • Accepted:

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s10044-015-0503-1

    Keywords

    Search

    Navigation