Skip to main content

Advertisement

Springer Nature Link
Log in

Quadro-W learning for human behavior prediction in an evolving environment: a case study of the intelligent butler technology

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

In recent years, with advances in hardware devices (e.g., sensors and microprocessors), increases in the maturity of software technology, increases in popularity of the Internet, and decreases in the costs of technologies, embedded systems have been widely used in various applications, including in crop growth monitoring, commodity defect detection, transportation system management, and vital signs monitoring. However, to obtain sufficient information for analysis, various sensors must usually be installed in environments. This requirement can cause many problems, such as changes in the original environment because of the installation of hardware devices, a time-consuming process for setting up the system, high costs given the use of many hardware devices, and difficulty in system maintenance. Therefore, attempts should be made to collect sufficient information by using limited hardware devices. In addition to the hardware equipment used to collect environmental information, software models must be developed for each application scene for analyzing the collected information. Such models are usually designed according to certain environmental conditions and cannot be updated automatically with changes in the environment, which decreases the flexibility and life cycle of the system on which these models are installed. Therefore, in this study, we developed a Quadro-W (QW) learning method to predict human behavior. QW encompasses humans (who), objects (what), locations (where), and time (when). This system obtained QW information only from the data collected by cameras and did not use additional sensors. This study constructed a behavior prediction model on the basis of the obtained QW information. The developed model can not only make predictions based on the initial environment but also update itself with changes in the environment to increase the system flexibility and life cycle.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29
Fig. 30
Fig. 31
Fig. 32
Fig. 33
Fig. 34
Fig. 35
Fig. 36
Fig. 37
Fig. 38
Fig. 39
Fig. 40
Fig. 41

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

The data presented in this study are available upon request from the corresponding author.

References

  1. Wang H, Schmid C (2013) Action recognition with improved trajectories. In: 2013 IEEE international conference on computer vision, pp 3551–3558. https://doi.org/10.1109/ICCV.2013.441

  2. Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. In: NIPS'14: proceedings of the 27th international conference on neural information processing systems, vol 1, pp 568–576

  3. Yue-Hei Ng J, Hausknecht M, Vijayanarasimhan S, Vinyals O, Monga R, Toderici G (2015) Beyond short snippets: deep networks for video classification. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 4694–4702

  4. Qiu Z, Yao T, Mei T (2017) Learning spatio-temporal representation with pseudo-3d residual networks. In: Proceedings of the IEEE international conference on computer vision, pp 5533–5541

  5. Shou Z, Wang D, Chang S-F (2016) Temporal action localization in untrimmed videos via multi-stage CNNs. In: IEEE conference on computer vision and pattern recognition, pp 1049–1058

  6. Lin T, Zhao X, Su H, Wang C, Yang M (2018) BSN: boundary sensitive network for temporal action proposal generation. In: Proceedings of the European conference on computer vision (ECCV), pp 3–19

  7. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  8. Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84–90

    Article  Google Scholar 

  9. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: The 3rd international conference on learning representations (ICLR2015). https://arxiv.org/abs/1409.1556

  10. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  11. Watkins CJ, Dayan P (1992) Q-learning. Mach Learn 8(3–4):279–292

    Article  MATH  Google Scholar 

  12. Reinhard E, Adhikhmin M, Gooch B, Shirley P (2001) Color transfer between images. IEEE Comput Gr Appl 21(5):34–41

    Article  Google Scholar 

  13. Li Z, Jing Z, Yang X, Sun S (2005) Color transfer based remote sensing image fusion using non-separable wavelet frame transform. Pattern Recogn Lett 26(13):2006–2014. https://doi.org/10.1016/j.patrec.2005.02.010

    Article  Google Scholar 

  14. Vincent L, Soille P (1991) Watersheds in digital spaces: an efficient algorithm based on immersion simulations. IEEE Trans Pattern Anal Mach Intell 13(6):583–598

    Article  Google Scholar 

  15. Lin T-Y, Dollár P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. In: IEEE conference on computer vision and pattern recognition, pp 2117–2125

  16. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 580–587

  17. Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: Towards real-time object detection with region proposal networks. In: NIPS’15: proceedings of the 28th international conference on neural information processing systems, vol 1, pp 91–99

  18. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 779–788

  19. Liu W et al (2016) SSD: Single shot multibox detector. In: European conference on computer vision, vol 9905. Springer, pp 21–37

  20. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention, Springer, pp 234–241

  21. Google, Speech-to-Text: Automatic Speech Recognition, Cloud Speech-to-Text. https://cloud.google.com/speech-to-text. erests include mobile computing, intelligent systems, and multimedia

  22. Mnih V, Kavukcuoglu K, Silver D et al (2015) Human-level control through deep reinforcement learning. Nature 518:529–533. https://doi.org/10.1038/nature14236

    Article  Google Scholar 

  23. Xiao Y, Li J, Zhu Y, Li Q (2020) User behavior prediction of social hotspots based on multimessage interaction and neural network. IEEE Trans Comput Soc Syst 7(2):536–545. https://doi.org/10.1109/TCSS.2020.2969484

    Article  Google Scholar 

  24. Li D, Shen D, Kou Y, Nie T (2019) Integrating sign prediction with behavior prediction for signed heterogeneous information networks. IEEE Access 7:171357–171371. https://doi.org/10.1109/ACCESS.2019.2937508

    Article  Google Scholar 

  25. Jiang W, Lv S, Wang Y, Chen J, Liu X, Sun Y (2021) Computational experimental study on social organization behavior prediction problems. IEEE Trans Comput Soc Syst 8(1):148–160. https://doi.org/10.1109/TCSS.2020.3017818

    Article  Google Scholar 

  26. Woźniak M, Połap D (2020) Intelligent home systems for ubiquitous user support by using neural networks and rule-based approach. IEEE Trans Ind Inf 16(4):2651–2658. https://doi.org/10.1109/TII.2019.2951089

    Article  Google Scholar 

  27. Liu Y, Xie DY, Gao Q, Han J, Wang S, Gao X (2019) Graph and autoencoder based feature extraction for zero-shot learning. In: IJCAI, vol 1, no 2, p 6

  28. Library of Congress (2020) Who is credited with inventing the telephone? https://www.loc.gov/everyday-mysteries/item/who-is-credited-with-inventing-the-telephone/. Accessed 07 Jan 2020

  29. He K, Zhang X, Ren S, Sun J (2015) Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Analysis Mach Intell 37(9):1904–1916

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the Ministry of Science and Technology (MOST) of Taiwan under Grant MOST 111-2221-E-218-011 and in part by the Allied Advanced Intelligent Biomedical Research Center, STUST from Higher Education Sprout Project, Ministry of Education, Taiwan.

Author information

Authors and Affiliations

  1. Department of Computer Science and Information Engineering, National Cheng Kung University, Tainan City, Taiwan

    Sheng-Tzong Cheng, Chih-Wei Hsu & Kuan-Ting Tsai

  2. Department of Computer Science and Information Engineering, Southern Taiwan University of Science and Technology, Tainan City, Taiwan

    Gwo-Jiun Horng

Authors
  1. Sheng-Tzong Cheng
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Chih-Wei Hsu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Gwo-Jiun Horng
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Kuan-Ting Tsai
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Gwo-Jiun Horng.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cheng, ST., Hsu, CW., Horng, GJ. et al. Quadro-W learning for human behavior prediction in an evolving environment: a case study of the intelligent butler technology. J Supercomput 79, 6309–6346 (2023). https://doi.org/10.1007/s11227-022-04899-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-022-04899-1

Keywords