Skip to main content
Springer Nature Link
Log in

A Multi-modal Framework with Contrastive Learning and Sequential Encoding for Enhanced Sleep Stage Detection

  • Conference paper
  • First Online:
Pattern Recognition and Computer Vision (PRCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15035))

Included in the following conference series:

  • 239 Accesses

Abstract

Sleep stage detection is an important aspect of assessing sleep quality. Conventional research has primarily focused on automatic sleep scoring using single-channel electroencephalogram (EEG) data. However, accurately detecting sleep stages based solely on EEG signals is challenging due to artifacts and similarities in EEG patterns introduced by eye movements during certain sleep stages. To address these issues, we propose a multi-modal framework that combines contrastive learning and sequential encoding to enhance sleep stage detection by leveraging both EEG and electrooculogram (EOG) signals. Our framework consists of several key components, including contrastive learning to optimize convolutional encoders for epoch-level feature representation, two distinct sequence encoders to capture temporal dependencies at the sequence level, and an ensemble prediction strategy to improve sleep stage detection performance further. Through extensive comparisons with state-of-the-art techniques on three publicly available datasets, our proposed algorithm consistently demonstrated superior performance across various evaluation metrics, thereby validating its effectiveness and potential to outperform existing methods in automatic sleep stage detection.

Supported by the Special Fund for Science and Technology of Guangdong Province under Grant 2020182, in part by Projects for International Scientific and Technological Cooperation of Guangdong Province under Grant 2023A0505050144.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Šušmáková, K.: Human sleep and sleep eeg. Meas. Sci. Rev. 4(2), 59–74 (2004)

    MATH  Google Scholar 

  2. Finan, P.H., Quartana, P.J., Remeniuk, B., Garland, E.L., Rhudy, J.L., Hand, M., Irwin, M.R., Smith, M.T.: Partial sleep deprivation attenuates the positive affective system: effects across multiple measurement modalities. Sleep 40(1), zsw017 (2017)

    Article  Google Scholar 

  3. Wang, H., Liu, X., Hongying, H., Wan, F., Li, T., Gao, L., Bezerianos, A., Sun, Y., Jung, T.-P.: Dynamic reorganization of functional connectivity unmasks fatigue related performance declines in simulated driving. IEEE Trans. Neural Syst. Rehabil. Eng. 28(8), 1790–1799 (2020)

    Article  Google Scholar 

  4. Wang, H., Linfeng, X., Bezerianos, A., Chen, C., Zhang, Z.: Linking attention-based multiscale cnn with dynamical gcn for driving fatigue detection. IEEE Trans. Instrum. Meas. 70, 1–11 (2020)

    Article  MATH  Google Scholar 

  5. Wang, H., Liu, X., Li, J., Tao, X., Bezerianos, A., Sun, Y., Wan, F.: Driving fatigue recognition with functional connectivity based on phase synchronization. IEEE Trans. Cognit. Dev. Syst. 13(3), 668–678 (2020)

    Article  MATH  Google Scholar 

  6. Tao, X., Linfeng, X., Zhang, H., Ji, Z., Li, J., Bezerianos, A., Wang, H.: Effects of rest-break on mental fatigue recovery based on eeg dynamic functional connectivity. Biomed. Signal Process. Control 77, 103806 (2022)

    Article  Google Scholar 

  7. Xu, T., Wang, H., Lu, G., Wan, F., Deng, M., Qi, P., Bezerianos, A., Guan, C., Sun, Y.: E-key: an eeg-based biometric authentication and driving fatigue detection system. IEEE Trans. Affect. Comput. 14(2), 864–877 (2021)

    Article  Google Scholar 

  8. Suzuki, K., Miyamoto, M., Hirata, K.: Sleep disorders in the elderly: diagnosis and management. J. Gen. Family Med. 18(2), 61–71 (2017)

    Article  MATH  Google Scholar 

  9. Sleep Computing Committee of the Japanese Society of Sleep Research Society (JSSR), Hori, T., Sugita, Y., Koga, E., Shirakawa, S., Inoue, K., Uchida, S., Kuwahara, H., Kousaka, M., Kobayashi, T., et al.: Proposed supplements and amendments to ‘a manual of standardized terminology, techniques and scoring system for sleep stages of human subjects’, the rechtschaffen & kales (1968) standard. Psychiatry Clin. Neurosci. 55(3), 305–310 (2001)

    Google Scholar 

  10. Engstrøm, M., Rugland, E., Heier, M.S.: Polysomnography (psg) for studying sleep disorders. Tidsskrift for den Norske laegeforening: tidsskrift for praktisk medicin, ny raekke 133(1), 58–62 (2013)

    Article  Google Scholar 

  11. Seo, H., Back, S., Lee, S., Park, D., Kim, T., Lee, K.: Intra-and inter-epoch temporal context network (iitnet) using sub-epoch features for automatic sleep scoring on raw single-channel eeg. Biomed. Signal Process. Control 61, 102037 (2020)

    Article  MATH  Google Scholar 

  12. Stepnowsky, C., Levendowski, D., Popovic, D., Ayappa, I., Rapoport, D.M.: Scoring accuracy of automated sleep staging from a bipolar electroocular recording compared to manual scoring by multiple raters. Sleep Med. 14(11), 1199–1207 (2013)

    Article  Google Scholar 

  13. Zhu, G., Li, Y., Wen, P.: Analysis and classification of sleep stages based on difference visibility graphs from a single-channel eeg signal. IEEE J. Biomed. Health Inform. 18(6), 1813–1821 (2014)

    Article  MATH  Google Scholar 

  14. Wang, Q., Lu, G., Pei, Z., Tang, C., Xu, L., Wang, Z., Wang, H.: P300 recognition based on ensemble of svms:-bci controlled robot contest of 2019 world robot conference. In: 2020 39th Chinese Control Conference (CCC), pp. 3035–3039. IEEE (2020)

    Google Scholar 

  15. Memar, P., Faradji, F.: A novel multi-class eeg-based sleep stage classification system. IEEE Trans. Neural Syst. Rehabil. Eng. 26(1), 84–95 (2017)

    Article  Google Scholar 

  16. Wang, H., Pei, Z., Linfeng, X., Tao, X., Bezerianos, A., Sun, Y., Li, J.: Performance enhancement of p300 detection by multiscale-cnn. IEEE Trans. Instrum. Meas. 70, 1–12 (2021)

    Article  MATH  Google Scholar 

  17. Wang, H., Huang, H., Liu, Y., Xu, H., Li, T.: An event related potential electroencephalogram signal analysis method based on denoising auto-encoder neural network. Control Theory Appl. 36(4), 589–595 (2019)

    MATH  Google Scholar 

  18. Chen, C., Li, Z., Wan, F., Leicai, X., Bezerianos, A., Wang, H.: Fusing frequency-domain features and brain connectivity features for cross-subject emotion recognition. IEEE Trans. Instrum. Meas. 71, 1–15 (2022)

    Google Scholar 

  19. Wang, H., Cong, W., Li, T., He, Y., Chen, P., Bezerianos, A.: Driving fatigue classification based on fusion entropy analysis combining eog and eeg. IEEE Access 7, 61975–61986 (2019)

    Article  Google Scholar 

  20. Wang, H., Li, Y., Long, J., Tianyou, Yu., Zhenghui, G.: An asynchronous wheelchair control by hybrid eeg-eog brain-computer interface. Cogn. Neurodyn. 8, 399–409 (2014)

    Article  MATH  Google Scholar 

  21. Chen, C., Ji, Z., Sun, Y., Bezerianos, A., Thakor, N., Wang, H.: Self-attentive channel-connectivity capsule network for eeg-based driving fatigue detection. IEEE Trans. Neural Syst. Rehabil. Eng. (2023)

    Google Scholar 

  22. Wang, Z., Zhang, H., Ji, Z., Yang, Y., Wang, H.: A review of deep learning methods for cross-subject rapid serial visual presentation detection in world robot contest 2022. Brain Sci. Adv. 9(3), 195–209 (2023)

    Article  MATH  Google Scholar 

  23. Supratak, A., Dong, H., Chao, W., Guo, Y.: Deepsleepnet: a model for automatic sleep stage scoring based on raw single-channel eeg. IEEE Trans. Neural Syst. Rehabil. Eng. 25(11), 1998–2008 (2017)

    Article  MATH  Google Scholar 

  24. Yong, Yu., Si, X., Changhua, H., Zhang, J.: A review of recurrent neural networks: Lstm cells and network architectures. Neural Comput. 31(7), 1235–1270 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  25. Supratak, A., Guo, Y.: Tinysleepnet: an efficient deep learning model for sleep stage scoring based on raw single-channel eeg. In: 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 641–644. IEEE (2020)

    Google Scholar 

  26. Mousavi, S., Afghah, F., Acharya, U.R.: Sleepeegnet: automated sleep stage scoring with sequence to sequence deep learning approach. PLoS ONE 14(5), e0216456 (2019)

    Article  Google Scholar 

  27. Eldele, E., Chen, Z., Liu, C., Min, W., Kwoh, C.-K., Li, X., Guan, C.: An attention-based deep learning approach for sleep stage classification with single-channel eeg. IEEE Trans. Neural Syst. Rehabil. Eng. 29, 809–818 (2021)

    Article  Google Scholar 

  28. Anderer, P., Roberts, S., Schlögl, A., Gruber, G., Klösch, G., Herrmann, W., Rappelsberger, P., Filz, O., Barbanoj, M.J., Dorffner, G., et al.: Artifact processing in computerized analysis of sleep eeg-a review. Neuropsychobiology 40(3), 150–157 (1999)

    Article  Google Scholar 

  29. Smelser, N.J., Baltes, P.B., et al.: International Encyclopedia of the Social & Behavioral Sciences, vol. 11. Elsevier Amsterdam (2001)

    Google Scholar 

  30. Goldberger, A.L., Amaral, L.A.N., Glass, L., Hausdorff, J.M., Ivanov, P.C., Mark, R.G., Mietus, J.E., Moody, G.B., Peng, C.-K., Stanley, H.E.: Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. Circulation 101(23), e215–e220 (2000)

    Article  Google Scholar 

  31. Ghassemi, M.M., Moody, B.E., Lehman, L.-W.H., Song, C., Li, Q., Sun, H., Mark, R.G., Westover, M.B., Clifford, G.D.: You snooze, you win: the physionet/computing in cardiology challenge 2018. In: 2018 Computing in Cardiology Conference (CinC), vol. 45, pp. 1–4. IEEE (2018)

    Google Scholar 

  32. Perslev, M., Jensen, M., Darkner, S., Jennum, P.J., Igel, C.: U-time: a fully convolutional network for time series segmentation applied to sleep staging. In: Advances in Neural Information Processing Systems, 32 (2019)

    Google Scholar 

  33. You, Y., Guo, X., Yang, Z., Shan, W.: A siamese network-based method for improving the performance of sleep staging with single-channel eeg. Biomedicines 11(2), 327 (2023)

    Article  MATH  Google Scholar 

  34. Efe, E., Özsen, S.: A new approach for automatic sleep staging: siamese neural networks. Traitement du Signal (2021)

    Google Scholar 

  35. You, Y., Chang, S., Yang, Z., Sun, Q.: Psnsleep: a self-supervised learning method for sleep staging based on siamese networks with only positive sample pairs. Front. Neurosci. 17, 1167723 (2023)

    Article  Google Scholar 

  36. Lee, S., Yeonguk, Yu., Back, S., Seo, H., Lee, K.: Sleepyco: automatic sleep scoring with feature pyramid and contrastive learning. Expert Syst. Appl. 240, 122551 (2024)

    Article  MATH  Google Scholar 

  37. Crnjanski, J., Krstić, M., Totović, A., Pleros, N., Gvozdić, D.: Adaptive sigmoid-like and prelu activation functions for all-optical perceptron. Opt. Lett. 46(9), 2003–2006 (2021)

    Article  MATH  Google Scholar 

  38. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)

    Google Scholar 

  39. Wenjian, W., Qian, X., Jun, X., Zhikun, H.: Dynamicsleepnet: a multi-exit neural network with adaptive inference time for sleep stage classification. Front. Physiol. 14, 1171467 (2023)

    Article  Google Scholar 

  40. Fan, J., Sun, C., Long, M., Chen, C., Chen, W.: Eognet: A novel deep learning model for sleep stage classification based on single-channel eog signal. Front. Neurosci. 15, 573194 (2021)

    Article  MATH  Google Scholar 

  41. Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607. PMLR (2020)

    Google Scholar 

  42. Yeh, C.-H., Hong, C.-Y., Hsu, Y.-C., Liu, T.-L., Chen, Y., LeCun, Y.: Decoupled contrastive learning. In: European Conference on Computer Vision, pp. 668–684. Springer (2022)

    Google Scholar 

  43. Berry, R.B., Budhiraja, R., Gottlieb, D.J., Gozal, D., Iber, C., Kapur, V.K., Marcus, C.L., Mehra, R., Parthasarathy, S., Quan, S.F., et al.: Rules for scoring respiratory events in sleep: update of the 2007 aasm manual for the scoring of sleep and associated events: deliberations of the sleep apnea definitions task force of the American academy of sleep medicine. J. Clin. Sleep Med. 8(5), 597–619 (2012)

    Article  Google Scholar 

  44. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems, 30 (2017)

    Google Scholar 

  45. Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., Liu, T.: On layer normalization in the transformer architecture. In: International Conference on Machine Learning, pp. 10524–10533. PMLR (2020)

    Google Scholar 

  46. Tsinalis, O., Matthews, P.M., Guo, Y., Zafeiriou, S.: Automatic sleep stage scoring with single-channel eeg using convolutional neural networks (2016). arXiv:1610.01683

  47. Sokolovsky, M., Guerrero, F., Paisarnsrisomsuk, S., Ruiz, C., Alvarez, S.A.: Deep learning for automated feature discovery and classification of sleep stages. IEEE/ACM Trans. Comput. Biol. Bioinf. 17(6), 1835–1845 (2019)

    Article  MATH  Google Scholar 

  48. Phan, H., Andreotti, F., Cooray, N., Chén, O.Y., De Vos, M.: Joint classification and prediction cnn framework for automatic sleep stage classification. IEEE Trans. Biomed. Eng. 66(5), 1285–1296 (2018)

    Article  MATH  Google Scholar 

  49. Li, F., Yan, R., Mahini, R., Wei, L., Wang, Z., Mathiak, K., Liu, R., Cong, F.: End-to-end sleep staging using convolutional neural network in raw single-channel eeg. Biomed. Signal Process. Control 63, 102203 (2021)

    Article  Google Scholar 

  50. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization (2017). arXiv:1711.05101

  51. Loshchilov, I., Hutter, F.: Stochastic gradient descent with warm restarts. In: Proceedings of the 5th International Conference Learning Representations, pp. 1–16 (2016)

    Google Scholar 

  52. LaValley, M.P.: Logistic regression. Circulation 117(18), 2395–2399 (2008)

    Article  MATH  Google Scholar 

  53. Kimura, M.: Understanding test-time augmentation. In: International Conference on Neural Information Processing, pp. 558–569. Springer (2021)

    Google Scholar 

  54. Ganaie, M.A., Hu, M., Malik, A.K., Tanveer, M., Suganthan, P.N.: Ensemble deep learning: a review. Eng. Appl. Artif. Intell. 115, 105151 (2022)

    Article  Google Scholar 

  55. Phan, H., Andreotti, F., Cooray, N., Chén, O.Y., De Vos, M.: Seqsleepnet: end-to-end hierarchical recurrent neural network for sequence-to-sequence automatic sleep staging. IEEE Trans. Neural Syst. Rehabil. Eng. 27(3), 400–410 (2019)

    Article  Google Scholar 

  56. Phan, H., Mikkelsen, K., Chén, O.Y., Koch, P., Mertins, A., De Vos, M.: Sleeptransformer: automatic sleep staging with interpretability and uncertainty quantification. IEEE Trans. Biomed. Eng. 69(8), 2456–2467 (2022)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Electronics and Information Engineering, Wuyi University, Jiangmen, China

    Zehui Wang & Hongtao Wang

  2. School of Life Sciences, Ludong University, Yantai, China

    Zhihan Zhang

Authors
  1. Zehui Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhihan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hongtao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hongtao Wang .

Editor information

Editors and Affiliations

  1. Peking University, Beijing, China

    Zhouchen Lin

  2. Nankai University, Tianjin, China

    Ming-Ming Cheng

  3. Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Ran He

  4. Xinjiang University, Ürümqi, Xinjiang, China

    Kurban Ubul

  5. Xinjiang University, Ürümqi, China

    Wushouer Silamu

  6. Peking University, Beijing, China

    Hongbin Zha

  7. Tsinghua University, Beijing, China

    Jie Zhou

  8. Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Cheng-Lin Liu

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, Z., Zhang, Z., Wang, H. (2025). A Multi-modal Framework with Contrastive Learning and Sequential Encoding for Enhanced Sleep Stage Detection. In: Lin, Z., et al. Pattern Recognition and Computer Vision. PRCV 2024. Lecture Notes in Computer Science, vol 15035. Springer, Singapore. https://doi.org/10.1007/978-981-97-8620-6_1

Download citation

  • DOI: https://doi.org/10.1007/978-981-97-8620-6_1

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-97-8619-0

  • Online ISBN: 978-981-97-8620-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics