Skip to main content
SpringerLink
Log in

Development of a combined time-frequency technique for accurate extraction of pNN50 metric from noisy heart rate measurements

  • Regular Paper
  • Published:
International Journal of Intelligent Robotics and Applications Aims and scope Submit manuscript

Abstract

In the design of human-in-the-loop robotic systems , it is of importance to evaluate human mental workload changes as this could inform the robots the degree to which a human is overwhelmed and error-prone in decision making. With an accurate assessment of mental workload, one can modulate the robots accordingly with the aim to assist the human in a task toward reduced mental workload. One key metric reliably utilized in the literature to assess human mental workload changes is called the pNN50—a metric associated with variations in the instantaneous frequency of human heart rate (HR). pNN50 can be extracted, for instance, from photoplethysmography (PPG) sensor measurements by implementing a well-known time-domain technique that is based on inter-beat intervals (IBI) of HR. When PPG measurements are contaminated with noise, the traditional time-domain approach may lead to inaccurate estimations of the pNN50 metric, as it heavily depends on precise calculation of the IBI from time series. In this manuscript, we present a combined time-frequency technique to remedy this problem. This new approach does not rely on time-domain based IBI data; and, hence, it can effectively avoid problems associated with noisy signals, rendering reliable computation of the pNN50. Examples using noisy synthetic signals and experimentally measured PPG data with and without noise contamination confirm the benefits of the proposed technique over the traditional time-domain based approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. One precaution here is that Hilbert transform cannot be applied to broadband signals, such as real data (Huang et al. 1998). See further remarks in Sect. 3.2.

  2. Notice that with added noise, the arising signal is no longer narrowband hence Hilbert transform can no longer be utilized. Indeed, with this transform the prediction of instantaneous frequency is erroneous and hence suppressed from the figures.

References

  • Allen, J.: Photoplethysmography and its application in clinical physiological measurement. Physiol. Meas. 28(3), R1–R39 (2007)

    Article  Google Scholar 

  • Aoun, J.E.: Robot-Proof: Higher Education in the Age of Artificial Intelligence. MIT Press, Cambridge (2017)

    Google Scholar 

  • Azevedo, C.R., Raizer, K., Souza, R.: A vision for human-machine mutual understanding, trust establishment, and collaboration. In: Cognitive and Computational Aspects of Situation Management (CogSIMA), 2017 IEEE Conference on, pp. 1–3. IEEE (2017)

  • Bailón, R., Laouini, G., Grao, C., Orini, M., Laguna, P., Meste, O.: The integral pulse frequency modulation model with time-varying threshold: application to heart rate variability analysis during exercise stress testing. IEEE Trans. Biomed. Eng. 58(3), 642–652 (2011)

    Article  Google Scholar 

  • Berntson, G.G., Quigley, K.S., Jang, J.F., Boysen, S.T.: An approach to artifact identification: application to heart period data. Psychophysiology 27(5), 586–598 (1990)

    Article  Google Scholar 

  • Berntson, G.G., Stowell, J.R.: Ecg artifacts and heart period variability: don’t miss a beat!. Psychophysiology 35(1), 127–132 (1998)

    Article  Google Scholar 

  • Bianchi, M., Valenza, G., Serio, A., Lanata, A., Greco, A., Nardelli, M., Scilingo, E.P., Bicchi, A.: Design and preliminary affective characterization of a novel fabric-based tactile display. In: Haptics Symposium (HAPTICS), 2014 IEEE, pp. 591–596. IEEE (2014)

  • Breazeal, C., Aryananda, L.: Recognition of affective communicative intent in robot-directed speech. Auton. Rob. 12(1), 83–104 (2002)

    Article  MATH  Google Scholar 

  • Cai, H., Lin, Y.: A roadside its data bus prototype for intelligent highways. IEEE Trans. Intell. Transp. Syst. 9(2), 344–348 (2008)

    Article  Google Scholar 

  • Camm, A.J., Malik, M., Bigger, J., Breithardt, G., Cerutti, S., Cohen, R., Coumel, P., Fallen, E., Kennedy, H., Kleiger, R., et al.: Heart rate variability: standards of measurement, physiological interpretation and clinical use. Task force of the european society of cardiology and the north American Society of pacing and electrophysiology. Circulation 93(5), 1043–1065 (1996)

    Article  Google Scholar 

  • Chan, H.L., Huang, H.H., Lin, J.L.: Time-frequency analysis of heart rate variability during transient segments. Ann. Biomed. Eng. 29(11), 983–996 (2001)

    Article  Google Scholar 

  • Chen JY, Haas EC, Barnes MJ (2007) Human performance issues and user interface design for teleoperated robots. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 37(6): 1231–1245.

  • Chen, V., Ling, H.: Time-Frequency Transforms for Radar Imaging and Signal Analysis. Technology and Engineering, Artech House (2001)

  • Cinaz, B., Arnrich, B., LaMarca, R., Tröster, G.: Monitoring of mental workload levels during an everyday life office-work scenario. Pers. Ubiquit. Comput. 17(2), 229–239 (2013)

    Article  Google Scholar 

  • Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., Taylor, J.G.: Emotion recognition in human–computer interaction. IEEE Signal Process. Mag. 18(1), 32–80 (2001)

    Article  Google Scholar 

  • Dehais, F., Causse, M., Vachon, F., Tremblay, S.: Cognitive conflict in human-automation interactions: a psychophysiological study. Appl. Ergon. 43(3), 588–595 (2012)

    Article  Google Scholar 

  • Dementyev, A., Hernandez, J., Follmer, S., Choi, I., Paradiso, J.: Skinbot: A wearable skin climbing robot. In: Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 5–6. ACM (2017)

  • Edsinger, A., Kemp, C.C.: Human-robot interaction for cooperative manipulation: Handing objects to one another. In: Robot and Human interactive Communication, 2007. RO-MAN 2007. The 16th IEEE International Symposium on, pp. 1167–1172. IEEE (2007)

  • El-Nasr, M.S., Drachen, A., Canossa, A.: Game analytics. Springer, Berlin (2016)

    Google Scholar 

  • Farha, F.: Hovakimyan and team aim to design coexisting, friendly robots (2015). https://mechanical.illinois.edu/news/hovakimyan-and-team-aim-design-coexisting-friendly-robots. [Online; Visited 26-January-2018]

  • Fong, T., Nourbakhsh, I., Dautenhahn, K.: A survey of socially interactive robots. Robot. Auton. Syst. 42(3), 143–166 (2003)

    Article  MATH  Google Scholar 

  • Friesen, G.M., Jannett, T.C., Jadallah, M.A., Yates, S.L., Quint, S.R., Nagle, H.T.: A comparison of the noise sensitivity of nine qrs detection algorithms. IEEE Trans. Biomed. Eng. 37(1), 85–98 (1990)

    Article  Google Scholar 

  • Gaillard, A.: Comparing the concepts of mental load and stress. Ergonomics 36(9), 991–1005 (1993)

    Article  Google Scholar 

  • Gao, Y.: A Digital Signal Processing Approach for Affective Sensing of a Computer User through Pupil Diameter Monitoring. Ph.D. thesis, FIU Electronic Thesis and Dissertations (2009)

  • Glaser, A.: Robots will start delivering food to doorsteps in silicon valley and Washington, D.C., today (2017). https://www.recode.net/2017/1/18/14306674/starship-robot-food-delivery-washington-dc-silicon-valley. [Online; Visited 26-January-2018]

  • Guyton, A., Hall, J.: Textbook of Medical Physiology. Elsevier Saunders, Amsterdam (2006)

    Google Scholar 

  • Haynes, J.D., Rees, G.: Decoding mental states from brain activity in humans. Nat. Rev. Neurosci. 7(7), 523–534 (2006)

    Article  Google Scholar 

  • Healey, J.A., Picard, R.W.: Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans. Intell. Transp. Syst. 6(2), 156–166 (2005)

    Article  Google Scholar 

  • Helander, M.G.: Handbook of Human-Computer Interaction. Elsevier, Amsterdam (2014)

    Google Scholar 

  • Hoover, A., Singh, A., Fishel-Brown, S., Muth, E.: Real-time detection of workload changes using heart rate variability. Biomed. Signal Process. Control 7(4), 333–341 (2012)

    Article  Google Scholar 

  • Huang, N.E., Shen, Z., Long, S.R., Wu, M.C., Shih, H.H., Zheng, Q., Yen, N.C., Tung, C.C., Liu, H.H.: The empirical mode decomposition and the hilbert spectrum for nonlinear and non-stationary time series analysis. In: Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences 454(1971), 903–995 (1998)

  • Hudlicka, E.: To feel or not to feel: the role of affect in human–computer interaction. Int. J. Hum. Comput. Stud. 59(1), 1–32 (2003)

    Article  Google Scholar 

  • Hudlicka, E.: Affective game engines: motivation and requirements. In: Proceedings of the 4th International Conference on Foundations of Digital Games, pp. 299–306. ACM (2009)

  • Islam, M.T., Zabir, I., Ahamed, S.T., Yasar, M.T., Shahnaz, C., Fattah, S.A.: A time-frequency domain approach of heart rate estimation from photoplethysmographic (PPG) signal. Biomed. Signal Process. Control 36, 146–154 (2017)

    Article  Google Scholar 

  • Heredia-Juesas, J., Thatcher, Y., Lu, J., Squiers, D., King, W., Fan, J., DiMaio, Martinez-Lorenzo, J.A.: Non-invasive optical imaging techniques for burn-injured tissue detection for debridement surgery. In: IEEE EMBC, Orlando, Florida (2016)

  • Kenney, J., Keeping, E.: Mathematics of statistics. Van Nostrand, Mathematics of Statistics (1947)

  • Khan, N.A., Jönsson, P., Sandsten, M.: Performance comparison of time-frequency distributions for estimation of instantaneous frequency of heart rate variability signals. Appl. Sci. 7(3), 221 (2017)

    Article  Google Scholar 

  • Klein, M.I., Warm, J.S., Riley, M.A., Matthews, G., Doarn, C., Donovan, J.F., Gaitonde, K.: Mental workload and stress perceived by novice operators in the laparoscopic and robotic minimally invasive surgical interfaces. J. Endourol. 26(8), 1089–1094 (2012)

    Article  Google Scholar 

  • Kohler, B.U., Hennig, C., Orglmeister, R.: The principles of software QRS detection. IEEE Eng. Med. Biol. Mag. 21(1), 42–57 (2002)

    Article  Google Scholar 

  • Kosuge, K., Hirata, Y.: Human-robot interaction. In: IEEE International Conference on Robotics and Biomimetics, 2004. ROBIO 2004, pp. 8–11. IEEE (2004)

  • Liu, C., Rani, P., Sarkar, N.: Human-robot interaction using affective cues. In: The 15th IEEE International Symposium on Robot and Human Interactive Communication, 2006. ROMAN 2006, pp. 285–290. IEEE (2006)

  • Malik, M.: Heart rate variability. Ann. Noninvasive Electrocardiol. 1(2), 151–181 (1996)

    Article  Google Scholar 

  • Malik, M., Cripps, T., Farrell, T., Camm, A.: Prognostic value of heart rate variability after myocardial infarction. A comparison of different data-processing methods. Med. Biol. Eng. Compu. 27(6), 603–611 (1989)

    Article  Google Scholar 

  • Mallat, S.: A Wavelet Tour of Signal Processing. The Sparse Way. Academic Press, Cambridge (2009)

    MATH  Google Scholar 

  • Mandryk, R.L.: Physiological measures for game evaluation. In: K.Isbister, N.Schaffer (eds.) Game Usability: Advice from the Experts for Advancing the Player Experience, pp. 207–235. Morgan Kaufmann (2008)

  • McSweeney, K.: Industrial robots are good for the economy, study suggests (2017). http://www.zdnet.com/article/industrial-robots-are-good-for-the-economy/. [Online; Visited 26-January-2018]

  • Mehler, B., Reimer, B., Coughlin, J.F.: Sensitivity of physiological measures for detecting systematic variations in cognitive demand from a working memory task an on-road study across three age groups. Hum. Factors J. Hum. Factors Ergon. Soc. 54(3), 396–412 (2012)

    Article  Google Scholar 

  • Mehler, B., Reimer, B., Coughlin, J.F., Dusek, J.A.: Impact of incremental increases in cognitive workload on physiological arousal and performance in young adult drivers. Transp. Res. Rec. J. Trans. Res. Board 2138(1), 6–12 (2009)

    Article  Google Scholar 

  • Mitchell, J.P.: Inferences about mental states. Philos. Trans. R. Soc. Lond. B Biol. Sci. 364(1521), 1309–1316 (2009)

    Article  Google Scholar 

  • Murray, I.R., Arnott, J.L.: Toward the simulation of emotion in synthetic speech: a review of the literature on human vocal emotion. J. Acoust. Soc. Am. 93(2), 1097–1108 (1993)

    Article  Google Scholar 

  • Orsila, R., Virtanen, M., Luukkaala, T., Tarvainen, M., Karjalainen, P., Viik, J., Savinainen, M., Nygard, C.H.: Perceived mental stress and reactions in heart rate variability-a pilot study among employees of an electronics company. Int. J. Occup. Saf. Ergon. (JOSE) 14(3), 275–283 (2008)

    Article  Google Scholar 

  • Pantic, M., Rothkrantz, L.J.M.: Automatic analysis of facial expressions: the state of the art. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1424–1445 (2000)

    Article  Google Scholar 

  • Parker, L.E.: Distributed intelligence: overview of the field and its application in multi-robot systems. J. Phys. Agents 2(1), 5–14 (2008)

    Google Scholar 

  • Parsinejad, P.: Inferring mental workload changes of subjects unfamiliar with a touch screen game through physiological and behavioral measurements. Ph.D. thesis, Northeastern University (2016)

  • Parsinejad, P., Rodriguez-Vaqueiro, Y., Martinez-Lorenzo, J.A., Sipahi, R.: Combined time-frequency calculation of pnn50 metric from noisy heart rate measurements. In: ASME 2014 Dynamic Systems and Control Conference, pp. V001T06A004–V001T06A004. American Society of Mechanical Engineers (2014)

  • Parsinejad, P., Sipahi, R.: A touchscreen game to induce mental workload on human subjects. In: Bioengineering Conference (NEBEC), 2014 40th Annual Northeast, pp. 1–2. IEEE (2014)

  • Parsinejad, P., Sipahi, R.: Assessment of human vulnerability in a touch-screen game; metrics and analysis. In: ASME 2015 Dynamic Systems and Control Conference, pp. V001T09A004–V001T09A004. American Society of Mechanical Engineers (2015)

  • Parsinejad, P., Sipahi, R.: Analysis of subjects’ vulnerability in a touch screen game using behavioral metrics. Applied Psychophysiology and Biofeedback pp. 1–14 (2017)

  • Penttilä, J., Helminen, A., Jartti, T., Kuusela, T., Huikuri, H.V., Tulppo, M.P., Coffeng, R., Scheinin, H.: Time domain, geometrical and frequency domain analysis of cardiac vagal outflow: effects of various respiratory patterns. Clin. Physiol. Funct. Imaging 21(3), 365–376 (2001)

    Article  Google Scholar 

  • Picard, R.W.: Affective Computing. MIT Press, Cambridge (1997)

    Book  Google Scholar 

  • Picard, R.W., Picard, R.: Affective Computing, vol. 252. MIT Press, Cambridge (1997)

    Google Scholar 

  • Rand, J., Hoover, A., Fishel, S., Moss, J., Pappas, J., Muth, E.: Real-time correction of heart interbeat intervals. IEEE Trans. Biomed. Eng. 54(5), 946–950 (2007)

    Article  Google Scholar 

  • Rani, P., Sarkar, N., Smith, C.A., Kirby, L.D.: Anxiety detecting robotic system-towards implicit human–robot collaboration. Robotica 22(1), 85–95 (2004)

    Article  Google Scholar 

  • Rowe, D.W., Sibert, J., Irwin, D.: Heart rate variability: indicator of user state as an aid to human–computer interaction. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 480–487. ACM Press/Addison-Wesley Publishing Co. (1998)

  • Ryu, K., Myung, R.: Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic. Int. J. Ind. Ergon. 35(11), 991–1009 (2005)

    Article  Google Scholar 

  • Shaffer, F., Ginsberg, J.: An overview of heart rate variability metrics and norms. Front. Public Health 5 (2017)

  • Speier, C.: The influence of information presentation formats on complex task decision-making performance. Int. J. Hum. Comput. Stud. 64(11), 1115–1131 (2006)

    Article  Google Scholar 

  • Steinfeld, A., Fong, T., Kaber, D., Lewis, M., Scholtz, J., Schultz, A., Goodrich, M.: Common metrics for human-robot interaction. In: Proceedings of the 1st ACM SIGCHI/SIGART Conference on Human–Robot Interaction, pp. 33–40. ACM (2006)

  • Sun, F.T., Kuo, C., Cheng, H.T., Buthpitiya, S., Collins, P., Griss, M.: Activity-aware mental stress detection using physiological sensors. In: International Conference on Mobile Computing, Applications, and Services, pp. 282–301. Springer (2010)

  • Taelman, J., Vandeput, S., Spaepen, A., VanHuffel, S.: Influence of mental stress on heart rate and heart rate variability. In: 4th European Conference of the International Federation for Medical and Biological Engineering, pp. 1366–1369. Springer (2009)

  • Tao, J., Tan, T.: Affective computing: a review. In: International Conference on Affective Computing and Intelligent Interaction, pp. 981–995. Springer (2005)

  • Vicente, K.J., Thornton, D.C., Moray, N.: Spectral analysis of sinus arrhythmia: a measure of mental effort. Hum. Factors J. Hum. Factors Ergon. Soc. 29(2), 171–182 (1987)

    Article  Google Scholar 

  • Willemse, C.J., Toet, A., VanErp, J.B.: Affective and behavioral responses to robot-initiated social touch: toward understanding the opportunities and limitations of physical contact in human–robot interaction. Front. in ICT 4, 12 (2017)

    Article  Google Scholar 

  • Yao, X., Lin, Y.: Emerging manufacturing paradigm shifts for the incoming industrial revolution. Int. J. Adv. Manuf. Technol. 85(5–8), 1665–1676 (2016)

    Article  Google Scholar 

Download references

Acknowledgements

Experimental data utilized in this study is protected under IRB protocol 11-19-11 at Northeastern University. R.S. acknowledges support of DARPA Young Faculty Award N66001-11-1-4161. The content of this research does not necessarily reflect the viewpoints of the funding agency, and no official endorsement of the US Government should be inferred. RS acknowledges fruitful discussions on the topic with Paul de la Houssaye (SPAWAR) and Gill Pratt (formerly Program Manager at DARPA, currently at Toyota Research Institute). Authors acknowledge the many valuable comments of the reviewers, which helped improve the quality of the manuscript. Authors Dr. Vaqueiro and Dr. Parsinejad equally contributed to this manuscript.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Northeastern University, Boston, MA, USA

    Yolanda R. Vaqueiro & Jose A. Martinez Lorenzo

  2. University of Vigo, Vigo, Spain

    Yolanda R. Vaqueiro

  3. Department of Mechanical and Industrial Engineering, Northeastern University, Boston, MA, USA

    Payam Parsinejad, Rifat Sipahi & Jose A. Martinez Lorenzo

Authors
  1. Yolanda R. Vaqueiro
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Payam Parsinejad
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rifat Sipahi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jose A. Martinez Lorenzo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rifat Sipahi.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vaqueiro, Y.R., Parsinejad, P., Sipahi, R. et al. Development of a combined time-frequency technique for accurate extraction of pNN50 metric from noisy heart rate measurements. Int J Intell Robot Appl 2, 193–208 (2018). https://doi.org/10.1007/s41315-018-0052-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41315-018-0052-z

Keywords

Search

Navigation