Skip to main content

Advertisement

SpringerLink
Log in

Affective social big data generation algorithm for autonomous controls by CRNN-based end-to-end controls

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Affective social multimedia computing provides us the opportunity to improve our daily lives. Various things, such as devices in ubiquitous computing environments and autonomous vehicles in real environments considering human beings, can be controlled by analyzing and learning affective social big data. Deep learning is a core learning algorithm for autonomous control; however, it requires huge amounts of learning data, and the process of collecting various types of learning data is expensive. The collection limit of affective social videos for deep learning is resolved by analyzing affective social videos, such as YouTube and Closed Circuit Television (CCTV) videos collected in advance, and generating new affective social videos more as learning data without human beings autonomously controlling other cameras. The control signals of the cameras are generated by Convolutional Neural Network (CNN)-based end-to-end controls. However, images captured consecutively need to be analyzed to improve the quality of the generated control signals. This paper proposes a system that generates affective social videos for deep learning by Convolutional Recurrent Neural Network (CRNN)-based end-to-end controls. The extracted images in affective social videos are utilized for calculating the control signals based on the CRNN. Additional affective social videos are then generated by the extracted consecutive images and camera control signals. The effectiveness of the proposed method was verified in the experiments by comparing the results obtained using the proposed method with those obtained using the traditional CNN. The results showed that the accuracy of the control signals obtained using the proposed method was 56.30% higher than that of the control signals obtained using the traditional CNN.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Bojarski M, Testa DD, Dworakowski D, Firner B, Flepp B, Goyal P, Jackel LD, Monfort M, Muller U, Zhang J, Zhang X, Zhao J, Zieba K (2016) End to end learning for self-driving cars. https://arxiv.org/abs/1604.07316. Accessed 15 January 2019

  2. Chen Y, Aygün RS (2015) SpriteCam: virtual camera control using sprite. Mutimedia Tools and Applications 74(3):1067–1089. https://doi.org/10.1007/s11042-013-1711-6

    Article  Google Scholar 

  3. Chen C, Seff A, Kornhauser A, Xiao J (2015) DeepDriving: learning affordance for direct perception in autonomous driving. Proceeding the IEEE International Conference on Computer Vision (ICCV). Santiago, Chile:1–9

  4. Codevilla F, Muller M, Lopez A, Koltun V, Dosovitskiy A (2018) End-to-End driving via conditional imitation learning. Proceeding 2018 IEEE International Conference on Robotics and Automation (ICRA2018). Brisbane:1–8

  5. DJI Ground Station https://www.dji.com. Accessed 18 January 2019

  6. Ebeid E, Skriver M, Terkildsen KH, Jensen K, Schultz UP (2018) A survey of open-source UAV flight controllers and flight simulators. Microprocess Microsyst 61:11–20. https://doi.org/10.1016/j.micpro.2018.05.002

    Article  Google Scholar 

  7. GCS Paparazzi https://wiki.paparazziuav.org. Accessed 18 January 2019

  8. Giusti A, Guzzi J, Ciresan DC, He F, Rodríguez JP, Fontana F, Faessler M, Forster C, Schmidhuber J, Caro GD, Scaramuzza D, Gambardella LM (2016) A machine learning approach to visual perception of forest trails for mobile robots. IEEE Robotics and Automation Letters 1(2):661–667. https://doi.org/10.1109/LRA.2015.2509024

    Article  Google Scholar 

  9. Hentati AI, Krichen L, Fourati M, Fourati LC (2018) Simulation tools, environments and frameworks for UAV systems performance analysis. Proceeding 14th International Wireless Communications & Mobile Computing Conference (IWCMC). Limassol, Cyprus:1495–1500

  10. Herzig J, Feigenblat G, Shmueli-Scheuer M, Konopnicki D, Rafaeli A (2016) Predicting customer satisfaction in customer support conversations in social media using affective features. Proceeding 2016 Conference on User Modeling Adaptation and Personalization. Nova Scotia, Canada: 115–119

  11. Hussein A, Elyan E, Gaber MM, Jayne C (2018) Deep imitation learning for 3D navigation tasks. Neural Comput & Applic 29(7):389–404

    Article  Google Scholar 

  12. Kersandt K, Munoz G, Barrado C (2018) Self-training by reinforcement learning for full-autonomous drones of the future. Proceeding IEEE/AIAA 37th Digital Avionics Systems Conference (DASC). London, UK:1–10

  13. Kim J, Chung D, Ko I (2017) A climbing motion recognition method using anatomical information for screen climbing games. Human-centric Computing and Information Sciences 7(25):1–14. https://doi.org/10.1186/s13673-017-0106-5

    Google Scholar 

  14. Lee S, Sung Y, Kim Y, Cha E (2018) Variations of AlexNet and GoogLeNet to improve Korean character recognition performance. Journal of Information Processing Systems 14(1):205–217. https://doi.org/10.3745/JIPS.04.0061

    Google Scholar 

  15. Li Z, Tang J (2016) Weakly supervised deep matrix factorization for social image understanding. IEEE Transaction on Image Processing 26(1):276–288. https://doi.org/10.1109/TIP.2016.2624140

    Article  MathSciNet  MATH  Google Scholar 

  16. Merino L, Caballero F, Martínez-de-Dios JR, Maza I, Ollero A (2012) An unmanned aircraft system for automatic forest fire monitoring and measurement. Journal of Intelligent & Robotic Systems 65:533–548. https://doi.org/10.1007/s10846-011-9560-x

    Article  Google Scholar 

  17. Mission Planner http://ardupilot.org/planner. Accessed 18 January 2019

  18. Polvara R, Patacchiola M, Sharma S, Wan J, Manning A, Sutton R, Cangelosi A (2018) Toward end-to-end control for UAV autonomous landing via deep reinforcement learning. Proceeding 2018 International Conference on Unmanned Aircraft Systems (ICUAS). Dallas, TX, USA:115–123

  19. Sanchez-Escobedo D, Lin X, Casas JR, Pardas M (2018) Hybridnet for depth estimation and semantic segmentation. Proceeding 2018 IEEE International Conference on Acoustics, Speech and Signal Processing. Alberta, Canada:1563–1567

  20. Shah U, Khawad R, Krishna KM (2016) DeepFly-towards complete autonomous navigation of MAVs with monocular camera. Proceeding the Tenth Indian Conference on Computer Vision, Graphics, and Image Processing (ICVGIP 2016). Indian:1–8

  21. Shah S, Dey D, Lovett C, Kapoor A (2017) AirSim: high-fidelity visual and physical simulation for autonomous vehicles. Proceeding 11th Conference on Field and Service Robotics (FSR 2017). Zurich:1–14

  22. Shah U, Khawd R, Krishna KM (2017) Detecting, localizing, and recognizing trees with a monocular MAV: Towards preventing deforestation. Proceeding 2017 IEEE International Conference on Robotics and Automation (ICRA). Singapore, Singapore:1982–1987

  23. Shi B, Bai X, Yao C (2017) An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Transactions on Pattern Analysis & Machine Intelligence 39(11):2298–2304. https://doi.org/10.1109/TPAMI.2016.2646371

    Article  Google Scholar 

  24. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. https://arxiv.org/pdf/1409.1556.pdf. Accessed 15 January 2019

  25. Smolyanskiy N, Kamenev A, Smith J, Birchfield ST (2017) Toward low-flying autonomous MAV trail navigation using deep neural networks for environmental awareness. Proceeding 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2017). Vancouver, Canada:1–7

  26. Song Y, Kim I (2018) DeepAct: a deep neural network model for activity detection in untrimmed videos. Journal of Information Processing Systems 14(1):150–161. https://doi.org/10.3745/JIPS.04.0059

    Google Scholar 

  27. Su Y, Grauman K (2017) Making 360° video watchable in 2D: learning videography for click free viewing. Proceeding 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA:1368–1376

  28. Sung Y, Kwak J, Yang D, Park Y (2015) Ground station design for the control of multi heterogeneous UAVs. Proceeding 2015 Spring Conference of Korea Multimedia Society. Andong, Repulic of Korea: 828–829

  29. Sung Y, Jin Y, Kwak J, Lee S, Cho K (2018) Advanced camera image cropping approach for CNN-based end-to-end controls on sustainable computing. Sustainability 10(3):1–13. https://doi.org/10.3390/su10030816

    Article  Google Scholar 

  30. Truong MTN, Kim S (2017) Parallel implementation of color-based particle filter for object tracking in embedded systems. Human-centric Computing and Information Sciences 7(2):1–13. https://doi.org/10.1186/s13673-016-0082-1

    Google Scholar 

  31. Wang Y, Wang S, Tang J, Liu H, Li B (2015) Unsupervised sentiment analysis for social media images. Proceeding Twenty-Fourth International Joint Conference on Artificial Intelligence. Buenos Aires, Argentina:2378–2379

  32. Xu A, Liu Z, Guo Y, Sinha V, Akkiraju R (2017) A new chatbot for customer service on social media. Proceeding 2017 CHI Conference on Human Factors in Computing Systems. Colorado, USA: 3506–3510

  33. Yang X, Zhang T, Xu C, Hossain MS (2015) Automatic visual concept learning for social event understanding. IEEE Transactions on Multimedia 17(3):346–358. https://doi.org/10.1109/TMM.2015.2393635

    Article  Google Scholar 

  34. YouTube https://www.youtube.com. Accessed 15 January 2019

  35. Zhou P, Zhou Y, Wu D, Jin H (2016) Differentially private online learning for cloud-based video recommendation with multimedia big data in social networks. IEEE Transactions on Multimedia 18(6):1217–1229. https://doi.org/10.1109/TMM.2016.2537216

    Article  Google Scholar 

Download references

Acknowledgements

This research was supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education (2018R1D1A1B07049990).

Author information

Authors and Affiliations

  1. Department of Multimedia Engineering, Dongguk University-Seoul, 30 Pildong-ro, 1-gil, Jung-gu, Seoul, 04620, Republic of Korea

    Jeonghoon Kwak & Yunsick Sung

  2. Department of Computer Science and Engineering, Seoul National University of Science and Technology, 232 Gongneung-ro, 1-gil, Nowon-gu, Seoul, 01811, Republic of Korea

    Jong Hyuk Park

Authors
  1. Jeonghoon Kwak
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jong Hyuk Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yunsick Sung
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jong Hyuk Park or Yunsick Sung.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kwak, J., Park, J.H. & Sung, Y. Affective social big data generation algorithm for autonomous controls by CRNN-based end-to-end controls. Multimed Tools Appl 78, 27175–27192 (2019). https://doi.org/10.1007/s11042-019-7703-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-7703-4

Keywords

Search

Navigation