Skip to main content
SpringerLink
Log in

Media Synchronization in Networked Multisensory Applications with Haptics

  • Chapter
  • First Online:
MediaSync

Abstract

In this chapter, we explain the present status of studies for media synchronization in networked multisensory applications with haptics. We also specify the characteristics of haptic media and different features of haptic media from other media such as olfactory, auditory, and visual media. By using such other media together with haptic media, we can get higher realistic sensation when we use the applications for various purposes such as remote education, entertainment, and networked games. When multisensory media streams are transmitted over a network like the Internet, the temporal relationships among the media streams may be disturbed owing to the network delay, delay jitter, and packet loss. Thus, the quality of experience (QoE) may seriously be degraded. To solve this problem, we need to carry out media synchronization control. To achieve a high quality of media synchronization, a number of media synchronization algorithms have been proposed so far. Especially, in networked multisensory applications, we need to take account of human perception of media synchronization errors in the algorithms. This is because the requirements of media synchronization quality depend on types of media. Some algorithms such as Virtual-Time Rendering (VTR) take human perception of the errors into account. For example, VTR tries to accomplish the synchronization by changing the buffering time of each media stream dynamically according to the network delay jitter with several threshold values about the errors. In the algorithms, instead of synchronizing the output timings of media streams exactly, ranges of human perception such as the allowable range, in which users feel that the synchronization error is allowable, the imperceptible range, in which users cannot perceive the error, and the operation range, which is narrower than the imperceptible range and should be kept usually, are taken into account for the sake of high synchronization quality. In this chapter, we explain the algorithms taking account of human perception of the media synchronization errors and enhance other algorithms such as the group (or inter-destination) synchronization control and the adaptive ∆-causality control algorithms for simultaneous output-timing control among multiple terminals by taking account of the perception. It is indispensable to clarify the ranges by QoE assessment in networked multisensory applications. In this chapter, we further make a survey of studies on the assessment. Finally, we discuss the future directions of media synchronization in networked multisensory applications with haptics.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 44.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 59.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 59.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Srinivasan, M.A., Basdogan, C.: Haptics in virtual environments: taxonomy, research status, and challenges. J. Comput. Graph. 21(4), 393–404 (1997)

    Article  Google Scholar 

  2. Kovacs, P.T., Murray, N., Rozinaj, G., Sulema, Y., Rybarova, R.: Application of immersive technologies for education: state of the art. In: Proceedings of 9th International Conference on Interactive Mobile Communication Technologies and Learning (IMCL), pp. 283–288, Nov 2015

    Google Scholar 

  3. Huang, P., Ishibashi, Y.: QoS control and QoE assessment in multi-sensory communications with haptics. IEICE Trans. Commun., Special Section on Quality of Communication Services Improving Quality of Life, E96-B(2), 392–403 (2013)

    Google Scholar 

  4. Ghinea, G., Ademoye, O.: Olfaction-enhanced multimedia: perspectives and challenges. Multimed. Tools Appl. 55(3), 601–626 (2011)

    Google Scholar 

  5. Murray, N., Lee, B., Qiao, Y., Muntean, G.: Olfaction-enhanced multimedia: a survey application domains, displays, and research challenges. ACM Comput. Surv. 48(4), 1–34 (2016)

    Article  Google Scholar 

  6. Murray, N., Qiao, Y., Lee, B., Muntean, G.: Subjective evaluation of olfactory and visual media synchronization. In: Proceedings of ACM Multimedia Systems, pp. 162–171, Mar 2013

    Google Scholar 

  7. Sithu M., Arima, R., Ishibashi, Y.: Experiment on object recognition with haptic, olfactory, and audio senses in virtual environment. IEICE Technical Report, CQ2015, Mar 2016

    Google Scholar 

  8. Ishibashi, Y., Hoshino, S., Zeng, Q., Fukushima, N., Sugawara, S.: QoE assessment of fairness in networked game with olfaction: Influence of time it takes for smell to reach player. Springer’s Multimed. Syst. J. (MMSJ), Special Issue on Network and Systems Support for Games 20(5), 621–631 (2014)

    Google Scholar 

  9. Huang, P., Ishibashi, Y., Fukushima, N., Sugawara, S.: QoE assessment of olfactory media in remote ikebana with haptic media. In: Proceedings of IEEE 2012 International Communications Quality and Reliability (CQR) Workshop, May 2012

    Google Scholar 

  10. Sithu, M., Ishibashi, Y., Huang, P., Fukushima, N.: Influences of network delay on quality of experience for soft objects in networked real-time game with haptic sense. Int. J. Commun. Netw. Syst. Sci. (IJCNS) 8(11), 440–455 (2015)

    Google Scholar 

  11. Sithu, M., Ishibashi, Y., Fukushima, N.: Effects of dynamic local lag control on sound synchronization and interactivity in joint musical performance. ITE Trans Media Technol. Appl., Special Section on Multimed. Transmission System and Services 2(4), 299–309 (2014)

    Google Scholar 

  12. Gunn, C., Hutchins, M., Steveson, D., Adcock, M., Youngblood, P.: Using collaborative haptics in remote surgical training. In: Proceedings of the First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pp. 1–2, Mar 2005

    Google Scholar 

  13. Mizuno, H., Ishibashi, Y., Sugawara, S.: A comparison between transmission methods of sense of force in a remote haptic instruction system. In: Proceedings of IEEE ICME, pp. 1202–1205, June/July 2009

    Google Scholar 

  14. Zeng, Q., Iwata, K., Ishibashi, Y., Fukushima, N., Sugawara, S.: Influence of network delay in networked ensemble with haptic media, sound, and video. In: National Convention of IPSJ, 3Y-5, Mar 2012

    Google Scholar 

  15. Lun, M., Friedman, D., Sankaranarayanan, G., King, H., Fodero, K., Leuschke, R., Hannaford, B.: The RAVEN: design and validation of a telesurgery system. Int. J. Robot. Res. 1(1), 1–16 (2009)

    Google Scholar 

  16. ITU E. 800, Terms and definitions related to quality of service and network performance including dependability, Aug 1994

    Google Scholar 

  17. ITU-T Rec. P. 10/G. 100 Amendment 1, New appendix I—definition of quality of experience (QoE). Int. Telecommun. Union, Jan 2007

    Google Scholar 

  18. Blakowski, G., Steinmetz, R.: A media synchronization survey: reference model, specification, and case studies. IEEE J. Sel. Areas Commun. 14(1), 5–35 (1996)

    Article  Google Scholar 

  19. Huang, Z., Nahrstedt, K., Steinmetz, R.: Evolution of temporal multimedia synchronization principles: a historical viewpoint. ACM Trans. Multimed. Comput. Commun. Appl. 9(1), 34 (2013)

    Google Scholar 

  20. Sithu, M., Ishibashi, Y.: Media synchronization control in multimedia communication. IGI Global Emerg. Res. Netw. Multimed. Commun. Syst. Chapter 2, 25–61 (2015)

    Google Scholar 

  21. Ishibashi, Y., Tasaka, S., Hasegawa, T.: The Virtual-Time Rendering algorithm for haptic media synchronization in networked virtual environments. In: Proceedings of the 16th International Workshop on Communications Quality and Reliability (CQR), pp. 213–217, May 2002

    Google Scholar 

  22. Ishibashi, Y., Kanbara, T., Tasaka, S.: Inter-stream synchronization between haptic media and voice in collaborative virtual environments. In: Proceedings of ACM Multimedia, pp. 604–611, Oct 2004

    Google Scholar 

  23. Sannomiya, H., Osada, J., Ishibashi, Y., Fukushima, N., Sugawara, S.: Inter-stream synchronization control with group synchronization algorithm. In: Proceedings of the 2nd IEEE Global Conference on Consumer Electronics (GCCE), pp. 520–524, Oct 2013

    Google Scholar 

  24. Ishibashi, Y., Tasaka, S.: A synchronization mechanism for continuous media in multimedia communications. In: Proceedings of IEEE INFOCOM, pp. 1010–1019, Apr 1995

    Google Scholar 

  25. Sithu, M., Ishibashi, Y., Fukushima, N.: Effects of dynamic local lag control on sound synchronization and interactivity in joint musical performance. ITE Trans Media Technol. Appl., Special Section on Multimedia Transmission System and Services 2(4), 299–309 (2014)

    Google Scholar 

  26. Ishibashi, Y., Tasaka, S.: A distributed control scheme for group synchronization in multicast communications. In: Proceedings of International Symposium on Communications (ISCOM), pp. 317–323, Nov 1999

    Google Scholar 

  27. Ishibashi, Y., Hashimoto, Y., Ikedo, T., Sugawara, S.: Adaptive ∆-causality control with adaptive dead-reckoning in networked games. In: Proceedings of the 13th Annual Workshop on Network and Systems Support for Games (NetGames), pp. 75–80, Sep 2007

    Google Scholar 

  28. Marshall, A., Wai, Y.: Haptic virtual environment performance over IP networks: a case study. In: Proceedings of IEEE International Symposium on Distributed Simulation and Real-Time Applications, pp. 181–189, Oct 2003

    Google Scholar 

  29. El Saddik, A.: The potential of haptics technologies. IEEE Instrum. Meas. Mag. 10(1), 10–17 (2007)

    Article  Google Scholar 

  30. Lin, M., Salisbury, K.: Haptic rendering: beyond visual computing. IEEE Comput. Graph. Appl. 24(2), 22–23 (2004)

    Article  Google Scholar 

  31. Salisbury, K., Conti, F., Barbgli, F.: Haptic rendering: Introductory concepts. IEEE Comput. Graph. Appl. 24(2), 24–32 (2004)

    Article  Google Scholar 

  32. Weller, R., Zachmann, G.: User performance in complex bimanual haptic manipulation with 3DoFs vs. 6DoFs. In: Proceedings of IEEE Haptics, Mar 2012

    Google Scholar 

  33. http://geomagic.com/en/products/phantom-omni/overview

  34. http://geomagic.com/en/products/phantom-desktop/overview

  35. http://geomagic.com/en/products/phantom-premium/overview

  36. http://www.forcedimension.com

  37. Sato, M.: Development of string-based force display: SPIDAR. In: Proceedings of the 8th International Conference on Virtual System and Multi Media (VSMM), pp. 1034–1039, Sept 2002

    Google Scholar 

  38. Novint Technologies, Inc., Haptic device abstraction layer programmer’s guide, version 1.1.9 Beta, Sept 2007

    Google Scholar 

  39. Gosselin, F., Riwan, A.: Design of virtuose 3D: a new haptic interface for teleoperation and virtual reality. In: Proceedings of ICAR, pp. 205–212, Aug 2001

    Google Scholar 

  40. Vanderlinde, R.Q., Lammertse, P., Frederiksen, E., Ruiter, B.: The hapticmaster: a new high-performance haptic interface. In: Proceedings of EuroHaptics, pp. 1–5 (2002)

    Google Scholar 

  41. Immersion Corporation: CyberGrasp. http://www.immersion.com/3d/products/cybergrasp.php

  42. http://geomagic.com/en/products/sculpt/touch

  43. Yoo,Y., Sung, M., Kim, N., Jun, K.: An experimental study on the performance of haptic data transmission in networked haptic collaboration. In: Proceedings of International Conference on Advanced Communication Technology, pp. 657–662, Feb 2007

    Google Scholar 

  44. Osada, J., Ishibashi, Y., Fukushima, N., Sugawara, S.: QoE comparison of haptic control schemes in remote instruction system with haptic media, video, and voice. In: Proceedings of International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC), July 2012

    Google Scholar 

  45. Khademian, B., Hashtrudi-Zaad, K.: Performance issues in collaborative haptic training. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA), pp. 3275–3263, Apr 2007

    Google Scholar 

  46. Willaert, B., Corteville, B., Reynaerts, D., Brussel, H., Poorten, E.: Bounded environment passivity of the classical position-force teleoperation controller. In: Proceedings of IEEE/RSJ IROS, Oct 2009

    Google Scholar 

  47. Fujimoto, T., Ishibashi, Y., Sugawara, S.: Influences of inter-stream synchronization error on collaborative work in haptic and visual environments. In: Proceedings of Haptics, pp. 113–119, Mar 2008

    Google Scholar 

  48. Tatematsu, A., Ishibashi, Y., Fukushima, N., Sugawara, S.: QoE assessment in tele-operation with 3D video and haptic media. In: Proceedings of IEEE ICME Workshop on Hot Topics in Multimedia, July 2011

    Google Scholar 

  49. Zeng, Q., Ishibashi, Y., Fukushima, N., Sugawara, S., Psannis, K.E.: Influences of inter-stream synchronization errors among haptic media, sound, and video on quality of experience in networked ensemble. In: Proceedings of IEEE GCCE, pp. 466–470, Oct 2013

    Google Scholar 

  50. Ishibashi, Y., Kaneoka, H.: Faieness among game players in networked haptic environments: influence of network latency. In: Proceedings of IEEE ICME, July 2005

    Google Scholar 

  51. Ishibashi, Y., Tasaka, S.: Causality and media synchronization control for networked multimedia games: centralized versus distributed. In: Proceedings of NetGames, pp. 42–51, May 2003

    Google Scholar 

  52. Hoshino, S., Ishibashi, Y., Fukushima, N., Sugawara, S.: QoE assessment in olfactory and haptic media transmission: Influence of inter-stream synchronization error. In: Proceedings of IEEE CQR, May 2011

    Google Scholar 

  53. Nakano, S., Maeda, Y., Ishibashi, Y., Fukushima, N., Huang, P., Psannis, K.E.: Influence of network delay on fairness between players in networked game with olfactory and haptic senses. IEICE Technical Report, CQ2014-94, Jan 2015

    Google Scholar 

  54. Tasaka, S. Ishibashi, Y.: Mutually compensatory property of multimedia QoS. In: Conference Record IEEE ICC, pp. 1105–1111, Apr/May 2002

    Google Scholar 

  55. Tasaka, S., Ito, Y.: Psychometric analysis of the mutually compensatory property of multimedia QoS. In: Conference Record IEEE ICC, pp. 1880–1886, May 2003

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Seijoh University, Aichi, 476-8588, Japan

    Pingguo Huang

  2. Nagoya Institute of Technology, Nagoya, 466-8555, Japan

    Mya Sithu & Yutaka Ishibashi

Authors
  1. Pingguo Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mya Sithu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yutaka Ishibashi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pingguo Huang .

Editor information

Editors and Affiliations

  1. Departamento de Comunicaciones, Universitat Politècnica de València (UPV) – Campus de Gandia, Valencia, Grao de Gandia, Spain

    Mario Montagud

  2. Centrum Wiskunde & Informatica (CWI), Amsterdam, The Netherlands

    Pablo Cesar

  3. Centrum Wiskunde & Informatica (CWI), Amsterdam, The Netherlands

    Fernando Boronat

  4. Departamento de Comunicaciones, Universitat Politècnica de València (UPV) – Campus de Gandia, Valencia, Grao de Gandia, Spain

    Jack Jansen

Definitions

Quality of Service (QoS)

QoS is the quality of service which is provided from a layer to its upper layer in network layer model and it is defined as how much the service is faithful to the ideal situation.

Quality of Experience (QoE)

QoE is also called the user-level QoS. QoE is the quality which is perceived subjectively and/or experienced objectively by end-users.

Human perception

Human perception is the mental process in which human beings perceive or recognize an object or idea. In this chapter we quantitatively express human perception of synchronization error as ranges in which users can feel/perceive/allow the synchronization error.

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Huang, P., Sithu, M., Ishibashi, Y. (2018). Media Synchronization in Networked Multisensory Applications with Haptics. In: Montagud, M., Cesar, P., Boronat, F., Jansen, J. (eds) MediaSync. Springer, Cham. https://doi.org/10.1007/978-3-319-65840-7_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-65840-7_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-65839-1

  • Online ISBN: 978-3-319-65840-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation