Skip to main content
Springer Nature Link
Log in

Auto ROI & mask R-CNN model for QR code beautification (ARM-QR)

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

The development of the Internet has enabled the QR code to become the most frequently applied two-dimensional barcode in daily life and in commercial advertisements, and its application continues to be more diversified to include warehouse management, electronic tickets, mobile payments, etc. The standard QR code consists of black and white modules, which display a monotonous visual effect. Since graph patterns are much easier to understand than text characters, showing the subject by patterns inside the QR code is the easiest way to understand implicit content.

This research involves the development of a methodology called ARM-QR, in which the QR code is integrated with full-color images, and deep learning technology is used to beautify it. First, the region of interest (ROI) of the color image is automatically identified using Mask R-CNN. The QR code’s visual beautification is further adjusted by the content of the object. Discrete wavelet transform and contrast sensitivity functions are also used to strengthen the visual perception of the QR code and reduce the impact of a low print resolution on the graphic legibility. The ARM-QR code’s visual quality is intensively verified by visual quality indices, which include the Peak Signal-to-Noise Ratio (PSNR), Mean-Square Error (MSE), Structural Similarity Index Metric (SSIM), and Gradient Magnitude Similarity Deviation (GMSD) based on evaluating the experimental data. The results of the experiment confirm that the visual beautification of the QR code generated in this research is of higher quality than that in other QR code beautification studies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22

Similar content being viewed by others

Data availability

All data are publicly available.

References

  1. Beegan, A.P., Iyer, L.R., Bell, A.E., et al. “Design and Evaluation of Perceptual Masks for Wavelet Image Compression,” in Proc. 10th IEEE Digital Signal Processing Workshop, IEEE CS Press, pp. 88–93, Oct 2002. Available: https://doi.org/10.1109/DSPWS.2002.1231082

  2. Brostow, G.J., Fauqueur, J., Cipolla, R.: Semantic object classes in video: a high-definition ground truth database. Pattern Recogn. Lett. 30(2), 88–97 (2009)

    Article  Google Scholar 

  3. Chang, J., Alain B., and Ostromoukhov, V. “Structure-aware error diffusion,” in Proc. ACM Trans Graph (TOG) 28(5): no. 162:1–162:8, Dec 2009.

  4. Chen, H., Sun, K., Tian, Z., Shen, C. et al., “BlendMask: Top-down meets bottom-up for instance segmentation,” in Conf. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 8573–8581, Jan 2020.

  5. Chen, L.C., Zhu, Y., Papandreou, G. et al., “Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation,” in Conf. Computer Vision – ECCV 2018, pp. 833–851.

  6. Chu, H.K., Chang, C.S., Lee, R.R. et al., “Halftone QR codes,” in Proc. ACM Trans Graph (TOG) 32(6): no. 217. ACM SIGGRAPH ASIA 2013, https://doi.org/10.1145/2508363.2508408

  7. Falcon, A. (2017) 40 Gorgeous QR code Artworks That Rock [Online]. Available: http://www.hongkiat.com/blog/qr-code-artworks/ (accessed on 16 November 2022)

  8. Garateguy, G.J., Arce, G.R., Lau, D.L., et al.: QR images: optimized image embedding in QR codes. IEEE Transact. Image Process (2014). https://doi.org/10.1109/TIP.2014.2321501

    Article  MathSciNet  MATH  Google Scholar 

  9. He, K., Gkioxari, G., Dollár, P., et al. “Mask R-CNN,” in Conf. IEEE International Conference on Computer Vision (ICCV), pp. 2961–2969, Mar 2017. Available: https://doi.org/10.1109/ICCV.2017.322

  10. Huang, B.B., Tang, S.X.: A contrast-sensitive visible watermarking scheme”. IEEE Multimed (2006). https://doi.org/10.1109/MMUL.2006.23

    Article  Google Scholar 

  11. Kyprianidis J.E., and Döllner, J. “Image abstraction by structure adaptive filtering,” in Proc. EG UKTheory and Practice of Computer Graphics, pp 51–58, 2008.

  12. Levicky, D., Foris, P.: Human Visual System Models in Digital Image Watermarking. Radioengineering 13(4), 38–43 (2004)

    Google Scholar 

  13. Li, L., Li, Y., Wang, B., et al.: “A new aesthetic QR code algorithm based on salient region detection and SPBVM”, in Conf, pp. 20–32. Security with Intelligent Computing and Big-data Services, Springer, Cham (2017)

    Google Scholar 

  14. Li, L., Wang, B., Lu, J., Zhang, S., et al.: A new aesthetic QR code algorithm based on salient region detection and SPBVM. J. Int Technol 20(3), 935–946 (2019)

    Google Scholar 

  15. Lin, S.S., Chang, Y.F., Le, T.N.H. et al. Generation of Photorealistic QR codes,” in Conf. SIGGRAPH Asia 2019 Posters, Nov 2019.

  16. Lin, T.Y., Dollár, P., Girshick, R., et al. “Feature Pyramid Networks for Object Detection,” in Conf. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2117–2125, Dec 2016. Available: https://doi.org/10.1109/CVPR.2017.106

  17. Lin, L., Zou, X., He, L. et al. “Aesthetic QR code generation with background contrast enhancement and user interaction,” in Conf. Third International Workshop on Pattern Recognition, July 2018. Available: https://doi.org/10.1117/12.2502054

  18. Lin, L., Wu, S., Liu, S., et al., “Interactive QR code beautification with full background image embedding,” in Proc. SPIE 10443, Second International Workshop on Pattern Recognition, 1044317, June 2017. Available: https://doi.org/10.1117/12.2280282

  19. Lin, Y.S., Luo, S.J., Chen, B.Y.: Artistic QR code embellishment. Computer. Graph. Forum 32(7), 137–146 (2013)

    Article  Google Scholar 

  20. Lin, S.S., Hu, M.C., Lee, C.H., et al.: Efficient QR code beautification with high quality visual content. IEEE Transact Multimed (2015). https://doi.org/10.1109/TMM.2015.2437711

    Article  Google Scholar 

  21. Lu, J., Cheng, W., Zhang, S.Q., et al.: A novel aesthetic QR code algorithm based on hybrid basis vector matrices. Symmetry (2018). https://doi.org/10.3390/sym10110543

    Article  Google Scholar 

  22. Mannos, J., Sakrison, D.: The effects of a visual fidelity criterion on the encoding of images. IEEE Trans Inf Theorem (1974). https://doi.org/10.1109/TIT.1974.1055250

    Article  MATH  Google Scholar 

  23. Ono, S., Morinaga, K., Nakayama, S.: Two-dimensional barcode decoration based on real-coded genetic algorithm. In: Proceedings of IEEE CEC, Hong-Kong, China, pp. 1068–1073 (2008)

  24. Qiao, S., Fang, X., Sheng, B., et al.: Structure-aware QR code abstraction. Vis Comp (2015). https://doi.org/10.1007/s00371-015-1107-x

    Article  Google Scholar 

  25. Rathi, J., Grewal, S.K.: Aesthetic QR: approaches for beautified, fast decoding, and secured QR codes”. IJ Inform. Eng. Elect. Bus. 3, 10–18 (2022)

    Google Scholar 

  26. Redmon, J., Divvala, S., Girshick, R., et al.: “You only look once: unified, real-time object detection”, in Conf. IEEE Conf Comp Vis Patt Recogn. (2015). https://doi.org/10.1109/CVPR.2016.91

    Article  Google Scholar 

  27. Russ Cox's method, (2012, April 12) QArt Codes [Online]. Available: https://research.swtch.com/qart

  28. Shelhamer, E., Long, J. and Darrell, T. “Fully Convolutional Networks for Semantic Segmentation,” in Conf. IEEE Transactions on Computer Vision and Pattern Recognition (CVPR), pp. 3431–3440, June 2015.

  29. Tsai, M.J.: A visible watermarking algorithm based on the content and contrast aware (COCOA) technique. J. Visual Commun. Image Represent. (2009). https://doi.org/10.1016/j.jvcir.2009.03.011

    Article  Google Scholar 

  30. Tsai, M.J., Hsieh, C.Y.: The visual color QR code algorithm (DWT-QR) based on wavelet transform and human vision system. Multimed Tools App (2019). https://doi.org/10.1007/s11042-019-7308-y

    Article  Google Scholar 

  31. Tsai, M.J., Peng, S.L.: QR code beautification by instance segmentation (IS-QR). Dig Signal Process (2023). https://doi.org/10.1016/j.dsp.2022.103887

    Article  Google Scholar 

  32. USC SIPI–The USC-SIPI image database [Online]. Available: http://sipi.usc.edu/services/database/Database.html (accessed 3 Jan, 2021)

  33. Viola, P., Jones, M.J.: “Rapid object detection using a boosted cascade of simple features”, in Conf. IEEE Comp Soc Conf Comp Vis Patt Recogn 1, 511–518 (2001)

    Google Scholar 

  34. Visualead Company, “Visual QR code” [Online]. Available online: http://www.visualead.com/ (accessed on 16 November 2022).

  35. Wang, Z., Bovik, A.C., Sheikh, H.R., et al.: Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Proc. (2004). https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  36. Watson, A.B., Yang, G.Y., Solomon, J.A., et al.: Visibility of wavelet quantization noise. IEEE Trans Image Proc. (1997). https://doi.org/10.1109/83605413

    Article  Google Scholar 

  37. Xu, M., Su, H., Li, Y., et al.: Stylized aesthetic QR code. IEEE Trans. Multimed. 21(8), 1960–1970 (2018)

    Article  Google Scholar 

  38. Xue, W., Zhang, L., Mou, X., et al.: Gradient magnitude similarity deviation: a highly efficient perceptual image quality index. IEEE Trans. Image Process. 23(2), 684–695 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  39. Zhang, L., Zhang, L., Mou, X., et al.: FSIM: a feature similarity index for image quality assessment. IEEE Trans Image Proc. (2001). https://doi.org/10.1109/TIP.2011.2109730

    Article  MATH  Google Scholar 

Download references

Acknowledgements

This work was partially supported by the National Science Council in Taiwan, Republic of China, under MOST 109-2410-H-009-022-MY3. In addition, the authors would like to thank the National Center for High-Performance Computing (NCHC) of National Applied Research Laboratories (NARLabs) in Taiwan for the provision of computational and storage resources.

Author information

Authors and Affiliations

  1. Institute of Information Management, National Yang Ming Chiao Tung University, Hsinchu City, Taiwan, ROC

    Min-Jen Tsai, Hung-Yu Wu & Di-Ting Lin

Authors
  1. Min-Jen Tsai
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Hung-Yu Wu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Di-Ting Lin
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

Min-Jen Tsai, Hung-Yu Wu and Di-Ting Lin wrote the main manuscript text, prepared all figures and tables. All authors reviewed the manuscript.

Corresponding author

Correspondence to Min-Jen Tsai.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tsai, MJ., Wu, HY. & Lin, DT. Auto ROI & mask R-CNN model for QR code beautification (ARM-QR). Multimedia Systems 29, 1245–1276 (2023). https://doi.org/10.1007/s00530-022-01046-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-022-01046-x

Keywords