Skip to main content
SpringerLink
Log in

Panoramic Video Construction from Mobile Video Streams

  • Chapter
  • First Online:
Mobile Cloud Visual Media Computing

  • 845 Accesses

Abstract

Constructing a panoramic video out of multiple incoming live mobile video streams is a challenging problem with many applications in consumer, education, and security domains. This problem involves multiple users live streaming the same scene from different points of view, using their mobile phones, with the objective of constructing a panoramic video of the scene. The main challenge in this problem is the lack of coordination between the streaming users, resulting in too much, too little, or no overlap between incoming streams. To add to the challenge, the streaming users are generally free to move, which means that the amounts of overlap between the different streams are dynamically changing. In this chapter, we propose a method for automatically coordinating between streaming users, such that the quality of the resulting panoramic video is enhanced. The method works by analyzing incoming video streams, and automatically providing active feedback to the streaming users. We investigate different methods for generating and presenting the active feedback to the streaming users resulting in an improved panoramic video output compared to the case where no feedback is utilized.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 54.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Temporal information can be easily integrated to avoid frame-by-frame stitching as proposed in [9, 16].

  2. 2.

    we assume that the captured scene is at a large enough distance such that in-plane camera motion would be sufficient. If the assumption is violated, motion parallax problems will arise. Dealing with these issues are left for future work.

  3. 3.

    We provide in the supplementary material with this submission the set of frames that were used in the human evaluation study to aid in understanding what the human judges were asked to evaluate.

References

  1. Live cast. http://www.periscopeapp.co

  2. Meerkat. http://meerkatapp.co/

  3. Kaheel, A., El-Saban, M., Refaat, M., Izz, M.: Mobicast—a system for collaborative event casting using mobile phones ACM Mobile and Ubiquitous Multimedia—MUM (2009)

    Google Scholar 

  4. El-Saban, M., Wang, X.-J., Hasan, N., Bassiouny, M., Refaat, M.: Seamless annotation and enrichment of mobile captured video streams in real-time. In: ICME, IEEE International Conference on Multimedia & Expo (ICME) (2011)

    Google Scholar 

  5. Bassiouny, M., El Saban, M.: Object matching using feature aggregation over a frame sequence. In: WACV. IEEE (2011)

    Google Scholar 

  6. Agarwala, A., Agrawala, M., Cohen, M., Salesin, D., Szeliski, R.: Photographing long scenes with multi-viewpoint panoramas. In Proceeding of the SIGGRAPH, vol. 25, pp. 853–861 (2006)

    Google Scholar 

  7. Sorek, N., Bregman-Amitai, O.: Method for constructing a composite image, Samsung Electronics patent, Jan. 2009

    Google Scholar 

  8. Hannuksela, J., Sangi, P., Heikkila, J., Liu, X., Doermann, D.: Document image mosaicing with mobile phones. In: ICIAP (2007)

    Google Scholar 

  9. El-Saban, M., Refaat, M., Kaheel, A., Hamid, A.: Stitching videos streamed by mobile phones in real-time. In: ACM MM (2009)

    Google Scholar 

  10. Shimizu, T., Yoneyama, A., Takishima, Y.: A fast video stitching method for motion-compensated frames in compressed video streams. In: International Conference on Consumer Electronics (2006)

    Google Scholar 

  11. Kopf, J., Uyttendale, M., Deussen, O., Cohen, M.: Capturing and viewing gigapixel images. In: SIGGRAPH (2007)

    Google Scholar 

  12. Brown, M., Lowe, D.: Automatic panoramic image stitching using invariant features. In: ICCV (2007)

    Google Scholar 

  13. Boutellier, J., Silvn, O., Tico, M., Korhonen, L.: Objective evaluation of image mosaics. In: International Conference VISIGRAPH (2007)

    Google Scholar 

  14. Baudisch, P., et al.: Panoramic viewfinder: providing a realtime preview to help users avoid flaws in panoramic pictures. In: OZCHI (2005)

    Google Scholar 

  15. Agarwala, A., Zheng, C., Pal, C., Agrawala, M., Cohen, M., Curless, B., Salesin, D., Szeliski, R., Panoramic video textures. In: Proceeding of the SIGGRAPH, vol. 24, pp. 821–827 (2005)

    Google Scholar 

  16. El-Saban, M., Izz, M., Kaheel, A: Fast stitching of videos captured from freely moving devices by exploiting temporal redundancy, In: ICIP. IEEE (2010)

    Google Scholar 

  17. Martin, D., Fowlkes, C., Malik, J.: Learning to detect natural image boundaries using local brightness, color, and texture cues. IEEE Trans. Pattern Anal. Mach. Intell. 26(5), 530–549 (2004)

    Article  Google Scholar 

  18. Gevers, T., van de Weijer, J., Stokman, H.: In: Lukac, R., Plataniotis, K.N. (eds.) Color Image Processing: Methods and Applications. Color feature detection. CRC Press, Boca Raton (2006)

    Google Scholar 

  19. Mortensen, E.N.: Vision-assisted image editing. Comput. Gr. 33(4), 55–57 (1999)

    Article  Google Scholar 

  20. Shi, J,. Tomasi, C.: Good features to track. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 94) pp. 593–600 (1994)

    Google Scholar 

  21. Zhang, Z., et al.: A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry. Artif. Intell. 78, 87–119 (1995)

    Article  Google Scholar 

  22. Florack, L.M.J., Haar Romeny, BMt, Koenderink, J.J., Viergever, M.A.: General intensity transformations and differential invariants. JMIV 4, 171–187 (1994)

    Article  MathSciNet  Google Scholar 

  23. Mindru, F., Tuytelaars, T., Van Gool, L., Moons, T.: Moment invariants for recognition under changing viewpoint and illumination. CVIU 94, 3–27 (2004)

    Google Scholar 

  24. Baumberg, A.: Reliable feature matching across widely separated views. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2000), pp. 774–781 (2000)

    Google Scholar 

  25. Matas, J., et al.: Robust wide baseline stereo from maximally stable extremal regions. Image Vis. Comput. 22(10), 761–767 (2004)

    Article  Google Scholar 

  26. Lindeberg, T., Garding, J.: Shape-adapted smoothing in estimation of 3-D shape cues from affine deformations of local 2-D brightness structure. Image Vis. Comput. 15(6), 415–434 (1997)

    Article  Google Scholar 

  27. Mikolajczyk, K., Schmid, C.: Scale & affine invariant interest point detectors. Int. J. Comput. Vis. 60(1), 63–86 (2004)

    Article  Google Scholar 

  28. Mikolajczyk, K., et al.: A comparison of affine region detectors. Int. J. Comput. Vis. 65(1–2), 43–72 (2003)

    Google Scholar 

  29. Tuytelaars, T., Mikolajczyk, K.: Local invariant feature detectors. Found. Trends Comput. Gr. Comput. Vis. 3(1) (2007)

    Google Scholar 

  30. Morel, J.M., Yu, G.S.: ASIFT: a new framework for fully affine invariant image comparison. SIAM J. Imaging Sci. 2, 438–469 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  31. Shi, J., Tomasi, C.: Good features to track. In: Proceeding of the CVPR (1994)

    Google Scholar 

  32. Szeliski, R.: Image alignment and stitching: a tutorial, Microsoft Research. Technical report, MSR-TR-2004-92 (2006)

    Google Scholar 

  33. Zomet, A., Levin, A., Peleg, S.: Seamless image stitching by minimizing false edges. IEEE Trans. Image Process. (2006)

    Google Scholar 

  34. Agarwala, A.: Efficient gradient-domain compositing using quadtrees. ACM Trans. Gr. (2007)

    Google Scholar 

  35. Duan, Z., Gopalan, K., Dong, Y.: Push versus pull: Implications of protocol design on controlling unwanted traffic. In: Proceeding of the USENIX SRUTI (2005)

    Google Scholar 

  36. Bouguet, J.: Pyramidal implementation of the Lucas–Kanade feature tracker: description of the algorithm, Intel Research Labs. Technical report, OpenCV Document (2000)

    Google Scholar 

  37. Aggarwal, J.K., Nandhakumar, N.: On the computation of motion from sequences of images-a review, In: IEEE, vol. 76, pp. 917–935 (1988)

    Google Scholar 

  38. Baker, S., Scharstein, D., Lewis, J.P., Roth, S., Black, M., Szeliski, R.: A database and evaluation methodology for optical flow. In: ICCV (2007)

    Google Scholar 

  39. Lucas. B.D. Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proceedings of Imaging understanding workshop (1981)

    Google Scholar 

  40. Mikolajczyk, K., Schmid, C.: A performance evaluation of local descriptors. In: TPAMI (2005)

    Google Scholar 

  41. Davison, A.J., Murray D.W.: Simultaneous localization and map-building using active vision. In: IEEE Transactions on Pattern Analysis and Machine Intelligence (2002)

    Google Scholar 

  42. Gil, A., Reinoso, O., Burgard, W., Stachniss,C., Martnez Mozos, O.: Improving data association in rao-blackwellized visual SLAM. In: IEEE/RSJ International Conference on Intelligent Robots & Systems (2006)

    Google Scholar 

  43. Little, J., Se, S., Lowe D.G.: Global localization using distinctive visual features. In: IEEE/RSJ International Conference on Intelligent Robots & Systems (2002)

    Google Scholar 

  44. IPTC (1999). IPTC-NAA Information Interchange Model Version 4.1. Retrieved April 4, 2010, from http://www.iptc.org/std/IIM/4.1/specification/IIMV4.1.pdf

  45. Seon H.K., Sakire A.A., Byunggu Y., Roger Z.: Vector model in support of versatile georeferenced video search. In: MMSys ’10 Proceedings of the First Annual ACM SIGMM Conference on Multimedia systems

    Google Scholar 

  46. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to thank Ahmad Abd El Hamid, Mostafa Izz, and Mahmoud Refaat for contributing to this chapter’s content.

Author information

Authors and Affiliations

  1. Microsoft Technology & Research, Advanced Technology Lab, Cairo, Egypt

    Motaz El Saban

  2. Yahoo Inc., Sunnyvale, CA, USA

    Ayman Kaheel

Authors
  1. Motaz El Saban
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ayman Kaheel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Motaz El Saban .

Editor information

Editors and Affiliations

  1. Visual Computing Group, Microsoft Research Asia, Beijing, Beijing, China

    Gang Hua

  2. Alibaba Group, Hangzhou, Zhejiang, China

    Xian-Sheng Hua

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Saban, M.E., Kaheel, A. (2015). Panoramic Video Construction from Mobile Video Streams. In: Hua, G., Hua, XS. (eds) Mobile Cloud Visual Media Computing. Springer, Cham. https://doi.org/10.1007/978-3-319-24702-1_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-24702-1_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-24700-7

  • Online ISBN: 978-3-319-24702-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation