Skip to main content
SpringerLink
Log in

Interactive multi-frame reconstruction for mobile devices

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The small size of handheld devices, their video capabilities and multiple cameras are under-exploited assets. Properly combined, the features can be used for creating novel applications that are ideal for pocket-sized devices, but may not be useful in laptop computers, such as interactively capturing and analyzing images on the fly. In this paper we consider building mosaic images of printed documents and natural scenes from low resolution video frames. High interactivity is provided by giving a real-time feedback on the video quality, while simultaneously guiding the user’s actions. In our contribution, we analyze and compare means to reach interactivity and performance with sensor signal processing and GPU assistance. The viability of the concept is demonstrated on a mobile phone. The achieved usability benefits suggest that combining interactive imaging and energy efficient high performance computing could enable new mobile applications and user interactions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Adams A, Gelfand N, Pulli K (2008) Viewfinder alignment. In: Eurographics 2008, pp 597–606

  2. Ahonen T, Hadid A, Pietikäinen M (2006) Face description with local binary patterns: application to face recognition. IEEE Trans Pattern Anal Mach Intell 28(12):2037–2041

    Article  Google Scholar 

  3. Bilcu R, Burian A, Knuutila A, Vehvilainen M (2008) High dynamic range imaging on mobile devices. In: 15th IEEE international conference on electronics, circuits and systems. ICECS 2008, 31 August–3 September 2008, pp 1312–1315

  4. Bordallo M, Hannuksela J, Silvén O, Vehviläinen M (2009) Graphics hardware accelerated panorama builder for mobile phones. In: Proceeding of SPIE electronic imaging 2009, p 7256

  5. Bordallo M, Nykänen H, Hannuksela J, Silvén O, Vehviläinen M (2011) Accelerating image recognition on mobile devices using gpgpu. In: Proceeding of SPIE electronic imaging 2011, p 7872

  6. Boutellier J, Bordallo M, Silvén O, Tico M, Vehviläinen M (2007) Creating panoramas on mobile phones. In: Proceeding of SPIE electronic imaging 2007, p 6498

  7. Capin T, Pulli K, Akenine-Möller T (2008) The state of the art in mobile graphics research. IEEE Comput Graph Appl 1:74–84

    Article  Google Scholar 

  8. Dabrowski J, Munson E (2001) Is 100 milliseconds too fast? In: Conference on human factors in computing systems, pp 317–318

  9. DiVerdi S, Höllerer T (2007) Groundcam: a tracking modality for mobile mixed reality. In: IEEE virtual reality, pp 75–82

  10. Fung J, Mann S (2008) Using graphics devices in reverse: Gpu-based image processing and computer vision. In: 2008 IEEE international conference on multimedia and expo, 23 March–26 April 2008, pp 9–12

  11. Gelfand N, Adams A, Park SH, Pulli K (2010) Multi-exposure imaging on mobile devices. In: Proceedings of the international conference on multimedia, MM ’10. ACM, New York, pp 823–826

    Chapter  Google Scholar 

  12. Ha SJ, Lee SH, Cho NI, Kim SK, Son B (2008) Embedded panoramic mosaic system using auto-shot interface. IEEE Trans Consumer Electron 54(1):16–24

    Article  Google Scholar 

  13. Hachet M, Pouderoux J, Guitton P (2005) A camera-based interface for interaction with mobile handheld computers. In: I3D’05—ACM SIGGRAPH 2005 symposium on interactive 3D graphics and games. ACM Press, pp 65–71

  14. Hannuksela J, Sangi P, Heikkilä J (2007) Vision-based motion estimation for interaction with mobile devices. Comput Vis Image Underst (Special Issue on Vision for Human-Computer Interaction) 108(1–2):188–195

    Article  Google Scholar 

  15. Hannuksela J, Sangi P, Heikkilä J, Liu X, Doermann D (2007) Document image mosaicing with mobile phones. In: 14th international conference on image analysis and processing, pp 575–580

  16. Hannuksela J, Silvén O, Ronkäinen S, Alenius S, Vehviläinen M (2010) Camera assisted multimodal user interaction. In: Proceeding of SPIE electronic imaging 2010, p 754203

  17. Haro A, Mori K, Capin T, Wilkinson S (2005) Mobile camera-based user interaction. In: IEEE international conference on computer vision, workshop on human–computer interaction. Beijing, China, pp 79–89

  18. Hwang J, Jung J, Kim GJ (2006) Hand-held virtual reality: a feasibility study. In: ACM virtual reality software and technology, pp 356–363

  19. Kalva H, Colic A, Garcia A, Furht B (2011) Parallel programming for multimedia applications. Multimed Tools Appl 51(2):801–818

    Article  Google Scholar 

  20. Kim S, Su W-Y (1993) Recursive high-resolution reconstruction of blurred multiframe images. IEEE Trans Image Process 2(4):534–539

    Article  Google Scholar 

  21. Möhring M, Lessig C, Bimber O (2004) Optical tracking and video see-through ar on consumer cell phones. In: Workshop on virtual and augmented reality of the GI-Fachgruppe AR/VR, pp 193–204

  22. Nazhandali L, Zhai B, Olson J, Reeves A, Minuth M, Helfand R, Pant S, Austin T, Blaauw D (2005) Energy optimization of subthreshold-voltage sensor network processors. In: Proceedings of the 32nd annual international symposium on computer architecture, ISCA ’05. IEEE Computer Society, Washington, DC, pp 197–207

    Google Scholar 

  23. Pears N, Olivier P, Jackson D (2008) Display registration for device interaction. In: 3rd international conference on computer vision theory and applications, pp 446–451

  24. Pulli K, Baksheev A, Kornyakov K, Eruhimov V (2012) Real-time computer vision with opencv. Commun ACM 55(6):61–69

    Article  Google Scholar 

  25. Pulli K, Chen W-C, Gelfand N, Grzeszczuk R, Tico M, Vedantham R, Wang X, Xiong Y (2009) Mobile visual computing. In: International symposium on ubiquitous virtual reality. ISUVR ’09, pp 3–6

  26. Rakhmatov D, Vrudhula S (2003) Energy management for battery-powered embedded systems. ACM Trans Embed Comput Syst 2(3):277–324

    Article  Google Scholar 

  27. Ready JM, Taylor CN (2007) Gpu acceleration of real-time feature based algorithms. In: Proceedings of the IEEE workshop on motion and video computing. IEEE Computer Society, Washington, DC, p 8

    Google Scholar 

  28. Rohs M (2004) Real-world interaction with camera-phones. In: 2nd international symposium on ubiquitous computing systems, pp 39–48

  29. Seo B-K, Park J, Park J-I (2011) 3-d visual tracking for mobile augmented reality applications. In: 2011 IEEE international conference on multimedia and expo (ICME), pp 1–4

  30. Silven O, Rintaluoma T (2007) Energy efficiency of video decoder implementations. In: Fitzek F, Reichert F (eds) Mobile phone programming and its applications to wireless networking. Springer, pp 421–439

  31. Singhal N, Park IK, Cho S (2010) Implementation and optimization of image processing algorithms on handheld gpu. In: 2010 17th IEEE international conference on image processing (ICIP), pp 4481–4484

  32. Sinha SN, Frahm JM, Pollefeys M, Genc Y (2006) Gpu-based video feature tracking and matching. In: Workshop on edge computing using new commodity architectures

  33. Tian Y, Yap K-H, He Y (2011) Vehicle license plate super-resolution using soft learning prior. Multimed Tools Appl 1–17. doi:10.1007/s11042-011-0821-2

  34. Vandewalle P, Susstrunk S, Vetterli M (2006) A frequency domain approach to registration of aliased images with application to super-resolution. EURASIP J Appl Signal Process (Special Issue on Super-Resolution) 24:1–14

    Article  Google Scholar 

  35. Wagner D, Mulloni A, Langlotz T, Schmalstieg D (2010) Real-time panoramic mapping and tracking on mobile phones. In: Virtual reality conference (VR). IEEE

  36. Wang Y, Donyanavard K, Cheng B (2010) Energy-aware real-time face recognition system on mobile cpu-gpu platform. In: International workshop on computer vision on GPU

  37. Winkler S, Rangaswamy K, Zhou Z (2007) Intuitive map navigation on mobile devices. In: Stephanidis C (ed) 4th international conference on universal access in human-computer interaction, part II, HCI international 2007, LNCS 4555. Springer, Beijing, pp 605–614

    Google Scholar 

  38. Xiong Y, Pulli K (2010) Fast panorama stitching for high-quality panoramic images on mobile phones. IEEE Trans Consumer Electron 56(2):298–306

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Center for Machine Vision Research, University of Oulu, 90570, Oulu, Finland

    Miguel Bordallo López, Jari Hannuksela & Olli Silvén

  2. Nokia Research Center, Tampere, Finland

    Markku Vehviläinen

Authors
  1. Miguel Bordallo López
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jari Hannuksela
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Olli Silvén
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Markku Vehviläinen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Miguel Bordallo López.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bordallo López, M., Hannuksela, J., Silvén, O. et al. Interactive multi-frame reconstruction for mobile devices. Multimed Tools Appl 69, 31–51 (2014). https://doi.org/10.1007/s11042-012-1252-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-012-1252-4

Keywords

Search

Navigation