Skip to main content
Springer Nature Link
Log in

Aligning windows of live video from an imprecise pan-tilt-zoom camera into a remote panoramic display for remote nature observation

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

A pan-tilt-zoom (PTZ) robotic camera can provide a detailed live video of selected areas of interest within a large potential viewing field. The selective coverage is ideal for nature observation applications where power and bandwidth are often limited. To provide the spatial context for human observers, it is desirable to insert the live video into a large spherical panoramic display representing the entire viewing field. Accurate alignment of the video stream within the panoramic display is difficult due to imprecise pan-tilt values and rapid changes in camera configurations. Common image alignment algorithms are computationally expensive for real time applications. We are interested in designing algorithms that fit low power computation platform and hence can be implemented inside the PTZ camera in the future. We present a sampling-based constant-time image alignment algorithm based on spherical projection and projection-invariant selective sampling that accurately registers paired images at 100 frames per second on a simulated embedded platform. The alignment accuracy actually is better than existing methods when high rotational difference is involved. Experiments suggest that the new alignment algorithm is faster than existing algorithms by 1,471.6 times when aligning a six-mega-pixel image pair.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Benosman, R., Kang, S.B.: Panoramic vision. Springer, New York (2001)

    MATH  Google Scholar 

  2. Foote, J., Kimber, D.: Flycam: practical panoramic video and automatic camera control. In: IEEE International Conference on Multimedia and Expo, vol. 3. ICME 2000, New York, pp. 1419–1422 (2000)

  3. Foote, J., Kimber, D.: Enhancing distance learning with panoramic video. In: Proceedings of the 34th Hawaii International Conference on System Sciences (2001)

  4. Song D., Goldberg K.: Approximate algorithms for a collaboratively controlled robotic camera. IEEE. Trans. Robotics 23(5), 1061–1070 (2007)

    Article  Google Scholar 

  5. Song, D.: Sharing a Vision: Systems and Algorithms for Collaboratively-Teleoperated Robotic Cameras. Springer, New York (2009)

  6. Song D., Qin N., Goldberg K.: Systems, control models, and codec for collaborative observation of remote environments with an autonomous networked robotic camera. Auton. Robots 24(4), 435–449 (2008)

    Article  Google Scholar 

  7. Song, D., Qin, N., Goldberg, K.: A minimum variance calibration algorithm for pan-tilt robotic cameras in natural environments. In: IEEE International Conference on Robotics and Automation (ICRA), Orlando (2006)

  8. Swaminathan R., Nayar S.K.: Nonmetric calibration of wide-angle lenses and polycameras. IEEE Trans. Pattern Anal. Mach. Intell. 22(10), 1172–1178 (2000)

    Article  Google Scholar 

  9. Tan K.-H., Hua H., Ahuja N.: Multiview panoramic cameras using mirror pyramids. IEEE Trans. Pattern Anal. Mach. Intell. 26(7), 1941–946 (2004)

    Google Scholar 

  10. Baker S., Nayar S.K.: A theory of single-viewpoint catadioptric image formation. Int. J. Comp. Vis. 35(2), 175–196 (1999)

    Article  Google Scholar 

  11. Nayar, S.K.: Catadioptric omnidirectional camera. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Juan, Puerto Rico, pp. 482–488 (1997)

  12. Xiong, Y., Turkowski, K.: Creating image-based vr using a self-calibrating fisheye lens. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Juan, Puerto Rico, pp. 237–243 (1997)

  13. Szeliski, R.: Image alignment and stitching, microsoft research, technical report msr-tr-2004-92 (2004)

  14. Kim, B.Y., Jang, K.H., Jung, S.K.: Adaptive strip compression for panorama video streaming. In Computer Graphics International (CGI’04), Crete, Greece (2004)

  15. Szeliski, R.: Image alignment and stitching: a tutorial. Found. Trends. Comput. Graph. Vis. 2(1), 1–104 (2006)

    Article  Google Scholar 

  16. Shum, H.-Y., Szeliski, R.: Panoramic image mosaics. Microsoft Research Technical Report MSR-TR-97-23, Tech. Rep. (1997)

  17. Hartley R.: Self-calibration of stationary cameras. Int. J. Comp. Vis. 22, 5–23 (1997)

    Article  MathSciNet  Google Scholar 

  18. Coorg S., Teller S.: Spherical mosaics with quaternions and dense correlation. Int. J. Comp. Vis. 37(3), 259–273 (2000)

    Article  MATH  Google Scholar 

  19. Castro E.D., Morandi C.: Registration of translated and rotated images using finite fourier transform. IEEE Trans. Pattern Anal. Mach. Intell. 9(5), 700–703 (1987)

    Article  Google Scholar 

  20. Reddy B.S., Chatterji B.N.: An fft-based technique for translation, rotation, and scale-invariant image registration. IEEE Trans. Image Process 5(8), 1266–1271 (1996)

    Article  Google Scholar 

  21. Hu, B., Brown, C., Choi, A.: Acquiring an environment map through image mosaicking. In University of Rochester. TR-786, November 2001

  22. Cho, S.-H., Chung, Y.-K., Lee, J.Y.: Automatic image mosaic system using image feature detection and taylor series. In Proceedings of VIIth Digital Image Computing: Techniques and Applications. Sydney, pp. 549–556 (2003)

  23. Kanazawa, Y., Kanatani, K.: Image mosaicing by stratified matching. In: Proceedings of Statistical Methods in Video Processing Workshop, Denmark, pp. 31–36 (2002)

  24. Zhang, W., Kosecka, J., Li, F.: Mosaics construction from a sparse set of views. In: Proceedings of First International Symposium on 3D Data Processing Visualization and Transmission, pp. 177–180 (2002)

  25. Harris C.J., Stephens M.: A combined corner and edge detector. In: Proceedings 4th Alvey Vision Conference, Manchester, pp. 147–151 (1988)

  26. Torr, P.H.S., Zisserman, A.: Feature based methods for structure and motion estimation. In: Proceedings of the International Workshop on Vision Algorithm: Theory and Practice, Corfu, pp. 278–294 (1999)

  27. Zoghlami, I., Faugeras, O., Deriche, R.: Using geometric corners to build a 2d mosaic from a set of images. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition, San Juan, Puerto Rico, pp. 420–425 (1997)

  28. Brown M., Lowe D. (2003) Recognising panoramas. In: Proceedings of IEEE International Conference on Computer Vision, Nice, France, vol. 2, pp. 1218–1225

  29. Bay, H., Tuytelaars, T., Gool, L.V.: Surf: Speeded up robust features. In: 9th European Conference on Computer Vision (ECCV), Graz, Austria, pp. 404–417 (2006)

  30. Zhang Z., Deriche R., Faugeras O., Luong Q.T.: A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry. Art. Intell. 78, 87–119 (1995)

    Article  Google Scholar 

  31. Triggs B., McLauchlan P.F., Hartley R.I., Fitzgibbon A.W.: Bundle adjustment—a modern synthesis. In ICCV: Proceedings of the International Workshop on Vision Algorithms. Springer, London, pp. 298–372 (2000)

  32. Steedly, D., Pal, C., Szeliski, R.: Efficiently registering video into panoramic mosaics. In: Proceedings of Tenth IEEE International Conference on Computer Vision, vol. 2, pp. 1300–1307 (2005)

  33. Lee S., Hong M., Wee J. (2005) Low-hardware-cost motion estimation with large search range for vlsi multimedia processors. IEICE Trans. Inf. Syst. E88-D(9), 2177–2182

    Article  Google Scholar 

  34. Babionitakis K., Doumenis G., Georgakarakos G., Lentaris G., Nakos K., Reisis D., Sifnaios I., Vlassopoulos N.: A real-time motion estimation FPGA architecture. J. RealTime Image Proc. 3(1–2), 3–20 (2008)

    Article  Google Scholar 

  35. Mazare, S., Dugelay, J., Pacalet, R.: Using gpu for fast block-matching. In: European Signal Processing Conference, Florence, Italy (2006)

  36. Trucco E., Doull A., Odone F., Fusiello A., Lane D.: Dynamic video mosaicing and augmented reality for subsea inspection and monitoring. Oceanology International Conference, Brighton, UK (2000)

  37. Agarwala, A., Zheng, C., Pal, C., Agrawala, M., Cohen, M., Curless, B., Salesin, D., Szeliski, R.: Panoramic video textures. In: ACM Transactions on Graphics (Proceedings of SIGGRAPH 2005), Los Angeles (2005)

  38. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge Press, London (2000)

  39. Gonzalez, R., Woods, R.: Digital Image Processing. Addison-Wesley Publishing Company, USA (1992)

  40. Jolliffe, I.: Principal Component Analysis. Springer, New York (2002)

  41. Stockman G., Shapiro L.G.: Computer Vision. Prentice Hall, Upper Saddle River (2001)

    Google Scholar 

Download references

Acknowledgments

Thanks to J. Zhang for his help in implementing part of the algorithms. Thanks are given to H. Lee, C. Kim, and Z. Bing for their contributions to NetBot Laboratory, Department of Computer Science and Engineering, Texas A&M University. Thanks to K. Goldberg, J. Yi, D. Volz, R. Gutierrez-Osuna, and V. Taylor for insightful discussions and feedback.

Author information

Authors and Affiliations

  1. Computer Science and Engineering Department, Texas A&M University, College Station, TX, 77843, USA

    Dezhen Song, Yiliang Xu & Ni Qin

Authors
  1. Dezhen Song
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Yiliang Xu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Ni Qin
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Dezhen Song.

Additional information

This work was supported in part by Panasonic Inc., by Microsoft Inc., and by the National Science Foundation under IIS-0534848 and IIS-0643298.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Song, D., Xu, Y. & Qin, N. Aligning windows of live video from an imprecise pan-tilt-zoom camera into a remote panoramic display for remote nature observation. J Real-Time Image Proc 5, 57–70 (2010). https://doi.org/10.1007/s11554-009-0127-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-009-0127-z

Keywords