Skip to main content
SpringerLink
Log in

Building a smart lecture-recording system using MK-CPN network for heterogeneous data sources

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Nowadays, lecture-recording systems play a vital role in collecting spoken discourse for e-learning. However, in view of the growing development of e-learning, the lack of content is becoming a problem. This research presents a smart lecture-recording (SLR) system that can record orations at the same level of quality as a human team, but with a reduced degree of human involvement. The proposed SLR system is composed of two subsystems, referred to as virtual cameraman (VC), and virtual director (VD), respectively. All camera man components of VC subsystem are automatic and can take actions that include target and event detection, tracking, and view searching. The videos taken by these three components are forwarded to the VD subsystem, in which the representative shot is chosen for recording or direct broadcasting. We refer to this function of the VD subsystem as shot selection that is based on the content analysis. The capability of shot selection is pre-trained through a machine-learning process characterized by the counter-propagation neural (CPN) network. However, the CPN network yielded poor results when the input data were heterogeneous data. To increases the accuracy of shot selection, we applied multiple kernel learning (MKL) techniques into CPN network, called MK-CPN, to transform all the heterogeneous data from different content analysis methods into unified space. A series of experiments for real lecture has been conducted. The results showed that the proposed SLR system can provide oration records close to some extend to those taken by real human teams. We believe that the proposed system may not be limited to live speeches, if it can be configured with appropriate training materials.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29
Fig. 30
Fig. 31
Fig. 32
Fig. 33

Similar content being viewed by others

References

  1. Rowe LA, Harley D, Pletcher P, Lawrence S (2001) BIBS: a lecture webcasting system. Berkeley Multimedia Research Center Report, pp 1–23

  2. Rui Y, He L, Gupta A, Liu Q (2001) Building an intelligent camera management system. ACM Multimed 9:2–11

    Google Scholar 

  3. Bianchi M (1998) AutoAuditorium: a fully automatic, multi-camera system to televise auditorium presentations. In: Joint DARPA/NIST smart spaces technology workshop

  4. Bianchi M (2004) Automatic video production of lectures using an intelligent and aware environment. In: The 3rd international conference on mobile and ubiquitous multimedia, pp 117–123

  5. Abowd GD (1999) Classroom 2000: an experiment with the instrumentation of a living educational environment. IBM Syst J 38:508–530

    Article  Google Scholar 

  6. Cruz G, Hill R (1994) Capturing and playing multimedia events with STREAMS. In: ACM international conference on multimedia, pp 193–200

  7. Zhang C, Rui Y, Crawford J, He LW (2008) An automated end-to-end lecture capture and broadcasting system. ACM Trans Multimed Comput Commun Appl 4:2–11

    Article  Google Scholar 

  8. Yong R, Anoop G, Jonathan G, He LW (2004) Automating lecture capture and broadcast: technology and videography. Multimed Syst 10:3–15

    Article  Google Scholar 

  9. Onishi M, Fukunaga K (2004) Shooting the lecture scene using computer-controlled cameras based on situation understanding and evaluation of video images. In: The 17th international conference on pattern recognition, pp 781–784

  10. Lu CT, Chen SW (2011) Automatic lecture recording system. In: The 24th IPPR conference on computer vision, graphics, and image processing

  11. Cheng Y (1995) Mean shift, mode seeking, and clustering. IEEE Trans Pattern Anal Mach Intell 17(8):790–799

    Article  Google Scholar 

  12. Gleicher M, Masanz J (2000) Towards virtual videography. In: ACM Multimedia, pp 375–378

  13. Okuni S, Tsuruoka S, Rayat GP, Kawanaka H, Shinogi T (2007) Video scene segmentation using the state recognition of blackboard for blended learning. In: International conference on convergence information technology, pp 2437–2442

  14. Kumano M, Ariki Y, Amano M, Uehara K (2002) Video editing support system based on video grammar and content analysis. In: International conference on pattern recognition, pp 1031–1036

  15. Wang T, Mansfield A, Hu R, Collomosse J (2009) An evolutionary approach to automatic video editing. In: International conference on visual media production (CVMP), pp 127–134

  16. Machnicki E, Rowe LA (2002) Virtual director: automating a webcast. In: SPIE international conference on multimedia computer network. San Jose, California, pp 208–225

  17. Liu Q, Rui Y, Gupta A, Cadiz JJ (2001) Automating camera management for lecture room environments. In: The SIGCHI conference on human factors in computing systems, pp 442–449

  18. Ugalde HMR, Carmona JC, Reyes-Reyes J, Alvarado VM, Corbier C (2015) Balanced simplicity–accuracy neural network model families for system identification. Neural Comput Appl 26(1):171–186

    Article  Google Scholar 

  19. Xu Z, Song Q, Wang D (2014) A robust recurrent simultaneous perturbation stochastic approximation training algorithm for recurrent neural networks. Neural Comput Appl 24(7):1851–1866

    Article  Google Scholar 

  20. LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput 1(4):541–551

    Article  Google Scholar 

  21. Zhang H, Cao X, Ho J, Chow T (2017) Object-level video advertising: an optimization framework. IEEE Trans Ind Inform 13(2):520–531

    Article  Google Scholar 

  22. Hecht-Nielsen R (1987) Counter-propagation networks. Appl Opt 26(23):4979–4983

    Article  Google Scholar 

  23. G¨onen M, Alpaydın E (2011) Multiple kernel learning algorithms. J Mach Learn Res 12:2211–2268

    MathSciNet  MATH  Google Scholar 

  24. Lin YY, Liu TL, Fuh CS (2011) Multiple kernel learning for dimensionality reduction. IEEE Trans Pattern Anal Mach Intell 33(6):1147–1160

    Article  Google Scholar 

  25. Cheng KH, Hsieh CH, Wang CC (2011) Human action recognition using 3D body joints. In: The 24th IPPR conference on computer vision, graphics, and image processing

  26. Lin SY, You ZH, Hung YP (2011) A real-time action recognition approach with 3D tracked body joints and its application. In: The 24th IPPR conference on computer vision, graphics, and image processing

  27. Johann P, Hamböker R (1994) Parametric statistical theory. Walter de Gruyter, Berlin, pp 207–208. ISBN 3-11-013863-8

    Google Scholar 

  28. Rosten E, Drummond T (2005) Fusing points and lines for high performance tracking. In: IEEE international conference on computer vision (ICCV’05), vol 2, pp 1508–1511

  29. Lucas BD, Kanade T (1981) An iterative image registration technique with an application to stereo vision. In: Imaging understanding workshop, pp 121–130

  30. Liu T, Yuan Z, Sun J, Wang J, Zheng N, Tang X, Shum HY (2011) Learning to detect a salient object. IEEE Trans Pattern Anal Mach Intell 33(2):353–367

    Article  Google Scholar 

  31. Fang CJ, Chen SW, Fu CS (2003) Automatic change detection of driving environments in a vision-based driver assistance system. IEEE Trans Neural Netw 14(3):646–657

    Article  Google Scholar 

  32. Abdollahian G, Taskiran CM, Pizlo Z, Delp EJ (2010) Camera motion-based analysis of user generated video. IEEE Trans Multimed 12(1):28–41

    Article  Google Scholar 

Download references

Acknowledgements

The article was written as parts of a research Grant No. NSC-102-2221-E-003-013 financed by the Ministry of Science and Technology (MOST), Taiwan, R.O.C.

Author information

Authors and Affiliations

  1. Department of Computer Science and Information Engineering, National Taiwan Normal University, No.88, Sec. 4, Tingzhou Rd, Wenshan Dist, Taipei City, 11677, Taiwan, ROC

    Chiung-Yao Fang, Chia-Ju Lu & Sei-Wang Chen

  2. Industrial Technology Research Institute, No.195, Sec.4, Chung Hsing Rd, Chutung, Hsinchu, 31040, Taiwan, ROC

    An-Chun Luo & Yu-Shan Deng

Authors
  1. Chiung-Yao Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. An-Chun Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yu-Shan Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chia-Ju Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sei-Wang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to An-Chun Luo.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fang, CY., Luo, AC., Deng, YS. et al. Building a smart lecture-recording system using MK-CPN network for heterogeneous data sources. Neural Comput & Applic 31, 3759–3777 (2019). https://doi.org/10.1007/s00521-017-3328-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-017-3328-6

Keywords

Search

Navigation