Skip to main content
SpringerLink
Log in

An improved colour binary descriptor algorithm for mobile augmented reality

  • Original Article
  • Published:
Virtual Reality Aims and scope Submit manuscript

Abstract

The incorporation of both virtual content and real world in augmented reality (AR) allows real-time engagement with the virtual objects. The selection of an appropriate tracking algorithm is important to optimise the performance of mobile AR applications given the limited processing capabilities and memories of mobile devices like smartphones. Tracking in AR consists of four essential components, namely detector, descriptor, matcher, and pose estimator. Since a descriptor substantially affects the overall performance of a mobile AR application, it must have short computational time and remains invariant to scale, rotation, and lighting changes. Studies have proposed Fast Retina Keypoint (FREAK) descriptor as the most suitable descriptor for mobile AR applications. Unlike other greyscale descriptors, FREAK has shorter computational time and is less likely to be affected by scale and rotation changes. However, it overlooks the vital colour space information. Focusing on enhancing the efficiency and robustness of FREAK, this study proposed the use of CRH-FREAK (RGB + HSV) descriptor and applied the vertical concatenation technique that combined all extracted keypoints vertically. The robustness of the proposed descriptors against scale, rotation, and lighting changes was verified using Mikolajczyk and Amsterdam Library of Object Images (ALOI) datasets. The developed CRH-FREAK descriptors used six colour spaces to describe the keypoints, which made them slower than the original FREAK. However, the size reduction of CRH-FREAK from 512 bits to 128 bits in this study successfully reduced the computational time to 29.49 ms, which was found comparable to the original FREAK. The improved efficiency and robustness of a 128-bit CRH-FREAK descriptor benefit the future development of mobile AR applications that remain invariant to scale, rotation, and lighting changes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Source: Alahi et al. (2012)

Fig. 3

Source: Alahi et al. (2012)

Fig. 4
Fig. 5

Source: Wafy and Madbouly (2015)

Fig. 6

Source: Andono et al. (2014)

Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  • Alahi A, Ortiz R, Vandergheynst P FREAK: Fast retina keypoint. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012. IEEE, pp 510–517

  • Andono PN, Pramunendar RA, Supriyanto C, Hariadi M, Kondo K, Less DN (2014) An Evaluation of Color SIFT Descriptors for Underwater Images Proceedings of the Fourth IIEEJ International Workshop on Image Electronics and Visual Computing

  • Azuma RT (1997) A survey of augmented reality. Presence Teleoperators Virtual Environ 6:355–385

    Article  Google Scholar 

  • Bay H, Ess A, Tuytelaars T, Van Gool L (2008) Speeded-up robust features (SURF) computer vision and image understanding, vol 110, pp 346–359. doi:https://doi.org/10.1016/j.cviu.2007.09.014

  • Bernal J, Vilarino F, Sánchez J (2010) Feature detectors and feature descriptors: where we are now

  • Bianco S, Mazzini D, Pau D, Schettini R (2015) Local detectors and compact descriptors for visual search: a quantitative comparison. Digital Signal Process 44:1–13

    Article  Google Scholar 

  • Bleser G (2009) Towards visual-inertial slam for mobile augmented reality. der Technischen Universität Kaiserslautern

  • Bolyós C (2013) Scarlet–Real Time Mobile Augmented Reality Library. Paper presented at the Proceedings of CESCG 2013: The 17th Central European Seminar on Computer Graphics, Smolenice, Slovakia, 28–30 April 2013

  • Calonder M, Lepetit V, Özuysal M, Trzcinski T, Strecha C, Fua P (2012) BRIEF: Computing a local binary descriptor very fast IEEE Transactions on Pattern Analysis and Machine Intelligence, vol 34, pp 1281–1298. doi:https://doi.org/10.1109/TPAMI.2011.222

  • Cruz E, Orts-Escolano S, Gomez-Donoso F, Rizo C, Rangel JC, Mora H, Cazorla M (2019) An augmented reality application for improving shopping experience in large retail stores. Virtual Reality 23:281–291

    Article  Google Scholar 

  • Fan P, Men A, Chen M, Yang B Color-SURF: A surf descriptor with local kernel color histograms. In: 2009 IEEE International Conference on Network Infrastructure and Digital Content, 2009. IEEE, pp 726–730

  • Flavián C, Ibáñez-Sánchez S, Orús C (2019) The impact of virtual, augmented and mixed reality technologies on the customer experience. J Bus Res 100:547–560

    Article  Google Scholar 

  • Geusebroek JM, Burghouts GJ, Smeulders AWM (2005) The Amsterdam library of object images. Int J Comput Vis. https://doi.org/10.1023/B:VISI.0000042993.50813.60

    Article  Google Scholar 

  • Guan W, You S, Newmann U (2012) Efficient matchings and mobile augmented reality ACM transactions on multimedia computing, communications, and applications (TOMM). Vol 8, p 47

  • Hua J-Z, Liu G-H, Song S-X (2019) Content-based image retrieval using color volume histograms. Int J Pattern Recognit Artif Intell 33:1940010

    Article  Google Scholar 

  • Ihsan R, Sehat U (2013) A survey on augmented reality challenges and tracking acta graphica znanstveni časopis za tiskarstvo i grafičke komunikacije, vol 24, pp 29–46

  • Kottman M (2011) The color-BRIEF feature descriptor. in: spring conference on computer graphics SCCG, pp 28–30

  • Koyasu H, Nozaki K, Maekawa H (2014) Evaluation of image feature descriptors for marker-less AR applications advances in visual computing

  • Lam MC, Sadik MJ, Elias NF (2020a) The effect of paper-based manual and stereoscopic-based mobile augmented reality systems on knowledge retention. Virtual Reality. https://doi.org/10.1007/s10055-020-00451-9

    Article  Google Scholar 

  • Lam MC et al (2020b) Interactive augmented reality with natural action for chemistry experiment learning. TEM J 9:351

    Google Scholar 

  • Leutenegger S, Chli M, Siegwart RY (2011) BRISK: Binary Robust invariant scalable keypoints, pp 2548–2555. doi:https://doi.org/10.1109/ICCV.2011.6126542

  • Lowe G (2004) SIFT—The scale invariant feature transform. Int J 2:91–110

    Google Scholar 

  • Mahieu R, Tilak H (2015) Real-time mobile augmented reality using markerless subject tracking. Standford University Department of Electrical Engineering, Standford

    Google Scholar 

  • Majid NA (2018) Augmented reality to promote guided discovery learning for STEM learning. Int J Adv Sci Eng Inf Technol 8:1494–1500

    Article  Google Scholar 

  • Markatopoulou F, Pittaras N, Papadopoulou O, Mezaris V, Patras I (2015) A study on the use of a binary local descriptor and color extensions of local descriptors for video concept detection. In: MultiMedia Modeling, Springer, pp 282–293

  • Mikolajczyk KSC (2005) A performance evaluation of local descriptors. IEEE Trans Pattern Anal Mach Intell 27:1615–1630. https://doi.org/10.1109/TPAMI.2005.188

    Article  Google Scholar 

  • Nguyen TV, Tan D, Mirza B, Sepulveda (2016) J MARIM: mobile augmented reality for interactive manuals. In: Proceedings of the 2016 ACM on multimedia conference, 2016. ACM, pp 689–690

  • Obeidy WK, Arshad H, Chowdhury SA, Parhizkar B, Huang J (2013) Increasing the tracking efficiency of mobile augmented reality. In: International visual informatics conference, 2013. pp 447–457. doi:https://doi.org/10.1007/978-3-642-41939-3

  • Over P et al (2013) Trecvid 2013—an overview of the goals, tasks, data, evaluation mechanisms and metrics. In: Proceedings of TRECVID 2013, NIST USA, 2013 2013. pp 1–45

  • Rublee E, Rabaud V, Konolige K, Bradski G (2011) ORB: An efficient alternative to SIFT or SURF. pp 2564–2571. doi:https://doi.org/10.1109/ICCV.2011.6126544

  • Satoh K, Anabuki M, Yamamoto H, Tamura H (2001) A hybrid registration method for outdoor augmented reality. In: IEEE and ACM international symposium on Proceedings of augmented reality, IEEE, pp 67–76

  • Sung Y-T, Chang K-E, Liu T-C (2016) The effects of integrating mobile devices with teaching and learning on students’ learning performance: a meta-analysis and research synthesis. Comput Edu 94:252–275

    Article  Google Scholar 

  • Tan SY, Arshad H, Abdullah A (2018) An efficient and robust mobile augmented reality application . Int J Adv Sci Eng Inf Technol 8:1672–1672. https://doi.org/10.18517/ijaseit.8.4-2.6810

    Article  Google Scholar 

  • Tan SY, Arshad H, Abdullah A (2019) Distinctive accuracy measurement of binary descriptors in mobile augmented reality. PLoS ONE, p 14

  • Tang G, Liu Z, Xiong J (2019) Distinctive image features from illumination and scale invariant keypoints. Multimedia Tools Appl 78:23415–23442

    Article  Google Scholar 

  • Uchiyama H, Marchand E (2012) Object detection and pose tracking for augmented reality: recent approaches foundation in computer vision, pp 1–8

  • Ufkes A, Fiala M A markerless augmented reality system for mobile devices. In: 2013 International conference on computer and robot vision (CRV), IEEE, pp 226–233

  • Van De Sande KE, Gevers T, Snoek CG (2010) Evaluating color descriptors for object and scene recognition. IEEE Trans Pattern Anal Mach Intell 32:1582–1596

    Article  Google Scholar 

  • Wafy M, Madbouly A (2015) Increase Efficiency of SURF using RGB Color Space. Int J Adv Comput Sci Appl (IJACSA) 6:81–85

    Google Scholar 

  • Wu J, Cui Z, Sheng VS, Zhao P, Su D, Gong S (2013) A Comparative Study of SIFT and its Variants Measurement. Sci Rev 13:122–131

    Google Scholar 

  • Yang C-K, Chen Y-H, Chuang T-J, Shankhwar K, Smith S (2019) An augmented reality-based training system with a natural user interface for manual milling operations. Virtual Reality, pp 1–13

  • Zhang L, Gao G, Zhou C, Cui Z, Wang L (2018) An Efficient Feature Extraction Scheme for Mobile Anti-Shake in Augmented Reality. Tehnički vjesnik 25:1119–1124

    Google Scholar 

  • Zhou J, Cadavid S, Abdel-Mottaleb M (2011) Exploiting color sift features for 2d ear recognition. In: 2011 18th IEEE international conference on image processing, 2011. IEEE, pp 553–556

Download references

Acknowledgement

This work was supported by the Ministry of Higher Education Malaysia (FRGS/1/2018/ICT01/UKM/02/5).

Author information

Authors and Affiliations

  1. Center for Artificial Intelligence and Technology, Faculty of Information Science and Technology, Universiti Kebangsaan Malaysia, 43600, Bangi, Selangor, Malaysia

    Siok Yee Tan, Haslina Arshad & Azizi Abdullah

Authors
  1. Siok Yee Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haslina Arshad
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Azizi Abdullah
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Siok Yee Tan.

Ethics declarations

Conflict of interest

The authors have no conflicts of interest to declare.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tan, S.Y., Arshad, H. & Abdullah, A. An improved colour binary descriptor algorithm for mobile augmented reality. Virtual Reality 25, 1193–1219 (2021). https://doi.org/10.1007/s10055-021-00519-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10055-021-00519-0

Keywords

Search

Navigation