Abstract
This paper presents a spatiotemporal super-resolution method to enhance both the spatial resolution and the frame rate in a hybrid stereo video system. In this system, a scene is captured by two cameras to form two videos, including a low spatial resolution with high-frame-rate video and a high spatial resolution with low-frame-rate video. For the low-spatial-resolution video, the low-resolution frames are spatially super-resolved by the high-resolution video via the stereo matching, the bilateral overlapped block motion estimation, and the adaptive overlapped block motion compensation algorithms, while for the low-frame-rate video, those missed frames are interpolated using the high-resolution frames obtained by fusing the disparity compensation and the motion compensation frame rate up-conversion. Experimental results demonstrate that the proposed mixed spatiotemporal super-resolution method has a more significant contribution to both the subjective and objective qualities than the pure spatial super-resolution or the frame rate up-conversion.









Similar content being viewed by others
References
Stoykova, E., Alatan, A.A., Benzie, P., Grammalidis, N., Malassiotis, S., Ostermann, J., Piekh, S., Sainov, V., Theobalt, C., Thevar, T., Zabulis, X.: 3-D time-varying scene capture technologies—a survey. IEEE Trans. Circuits Syst. Video Technol. 17(11), 1568–1586 (2007)
Ozkalayci, B., Gedik, S., Alatan, A.A.: Multi-view video coding via dense depth estimation. In: Proceedings of the 3DTV Conference, pp. 1–4. Kos Island (2007)
Aflaki, P., Hannuksela, M.M., Gabbouj, M.: Subjective quality assessment of asymmetric stereoscopic 3D video. Signal Image Video Process. 9(2), 331–345 (2015)
Tian, J., Ma, K.K.: A state-space super-resolution approach for video reconstruction. Signal Image Video Process. 3(3), 217–240 (2009)
Nasrollahi, K., Moeslund, T.B.: Super-resolution: a comprehensive survey. Mach. Vis. Appl. 25(6), 1423–1468 (2014)
Tian, J., Ma, K.K.: A survey on super-resolution imaging. Signal Image Video Proces. 5(3), 329–342 (2011)
Brandi, F., de Queiroz, R., Mukherjee, D.: Super resolution of video using key frames. In: Proceeding of International Symposium on Circuits and Systems, pp. 1608–1611. Seattle (2008)
Brandi, F., de Queiroz, R., Mukherjee, D.: Super-resolution of video using key frames and motion estimation. In: Proceeding of International Conference on Image Processing, pp. 321–324. San Diego (2008)
Song, B.C., Jeong, S.C., Choi, Y.: Key frame-based video super-resolution using bi-directional overlapped block motion compensation and trained dictionary. In: Proceeding of International Conference on Image Processing Theory Tools and Applications, pp. 181–186. Paris (2010)
Song, B.C., Jeong, S.C., Choi, Y.: Video super-resolution algorithm using bi-directional overlapped block motion compensation and on-the-fly dictionary training. IEEE Trans. Circuits Syst. Video Technol. 21(3), 274–285 (2011)
Ge, J., Liu, J., Ge, C., Yang, X.: A robust video super-resolution based on adaptive overlapped block motion compensation. In: Proceeding of International Conference on Signal-Image Technology and Internet-Based Systems, pp. 187–194. Kyoto (2013)
Schierl, T., Narasimhan, S.: Transport and storage systems for 3-D video using MPEG-2 systems, RTP, and ISO file format. Proc. IEEE 99(4), 671–683 (2011)
Vetro, A., Tourapis, A.M., Muller, K., Chen, T.: 3D-tv content storage and transmission. IEEE Trans. Broadcast. 57(2), 384–394 (2011)
Lee, Y.L., Nguyen, T.: Method and architecture design for motion compensated frame interpolation in high-definition video processing. In: Proceeding of IEEE International Symposium on Circuits and Systems, pp. 1633–1636. Taipei (2009)
Zhang, B., Liu, J., Ge, J., Chen, C., Yuan, H., Liu, W.: A super resolution reconstruction scheme for mixed spatio-temporal stereo video. In: Proceeding of International Conference on Audio, Language and Image Processing, pp. 490–496. Shanghai (2012)
Le, H.V., Seetharaman, G.: A super-resolution imaging method based on dense subpixel-accurate motion fields. J. VLSI Signal Proces. Syst. Signal Image Video Technol. 42(1), 79–89 (2006)
Hung, E.M., de Queiroz, R.L., Brandi, F., de Oliveira, K.F., Mukherjee, D.: Video super-resolution using codebooks derived from key-frames. IEEE Trans. Circuits Syst. Video Technol. 22(9), 1321–1331 (2012)
Chen, Y.K., Vetro, A., Sun, H., Kung, S.Y.: Frame-rate up-conversion using transmitted true motion vectors. In: Proceeding of IEEE Second Workshop on Multimedia Signal Processing, pp. 622–627. Redondo Beach (1998)
De Haan, G., Biezen, P.W.A.C., Huijgen, H., Ojo, O.A.: True-motion estimation with 3-d recursive search block matching. IEEE Trans. Circuits Syst. Video Technol. 3(5), 368–379 (1993)
Liu, C.: Beyond pixels: exploring new representations and applications for motion analysis. Doctoral Thesis, Massachusetts Institute of Technology (2009)
Gu, K., Zhai, G., Yang, X., Zhang, W., Liu, M.: Structural similarity weighting for image quality assessment. In: Proceeding of IEEE International Conference on Multimedia and Expo Workshops, pp. 1–6. San Jose (2013)
Gu, K., Zhai, G., Yang, X., Zhang, W.: An efficient color image quality metric with local-tuned-global model. In: Proceeding of IEEE International Conference on Image Processing, pp. 506–510. Paris (2014)
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grants (61201211), in part by the Excellent Youth Scientist Award Foundation of Shandong Province under Grants BS(2012DX021), in part by the China Postdoctoral Science Foundation Funded Project under Grant (2013M530320), in part by the open research fund of Sciences Key Laboratory of Wireless Sensor Network and Communication under Grant (2013002) and in part by the Cultivation Fund the Key Scientific and Technical Innovation Project (708059).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Ge, J., Liu, J., Yuan, H. et al. A spatiotemporal super-resolution algorithm for a hybrid stereo video system. SIViP 10, 559–566 (2016). https://doi.org/10.1007/s11760-015-0774-4
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11760-015-0774-4