Skip to main content
Springer Nature Link
Log in

Automatic Detection and Recognition of Products and Planogram Conformity Analysis in Real Time on Store Shelves

  • Conference paper
  • First Online:
Advances in Visual Computing (ISVC 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13599))

Included in the following conference series:

Abstract

This paper investigates the problem of robust retail product recognition on store shelves images with the aim of solving planogram conformity rate estimation problem. The proposed system is integrated into the Belive.ai plate-form which is built on more than 30.000 cameras already deployed in multiple brands stores all over different countries. The first aspect of our approach is to perform robust detection of any retail product in any shelf by applying the deep learning step trained on a large data set. In contrast, the idea here is to combine the first step with a fast version of ASIFT with CUDA acceleration and color histogram to find missing or new products in the shelf images. The second step was achieved by providing a pipeline that allows merging the information of shelves images sent by the cameras with the planograms provided by the brands to calculate the planogram conformity rate in real time.

The proposed method was validated with a dataset of 385 shelf images of 12 product categories from several stores of four different brands. Experimental results show that our approach is highly accurate in finding a product in all categories and for solving planogram conformity rate estimation problem.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Oh, K., Lee, M., Kim, G., Kim, S.: Detection of multiple salient objects through the integration of estimated foreground clues. Image Vis. Comput. 54, 31–44 (2016)

    Article  Google Scholar 

  2. Villamizar, M., Garrell, A., Sanfeliu, A., Moreno-Noguer, F.: Interactive multiple object learning with scanty human supervision. Comput. Vis. Image Underst. 149, 51–64 (2016)

    Article  Google Scholar 

  3. Haladová, Z., Šikudová, E.: Multiple instances detection in RGBD images. In: Chmielewski, L.J., Kozera, R., Shin, B.-S., Wojciechowski, K. (eds.) ICCVG 2014. LNCS, vol. 8671, pp. 246–253. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11331-9_30

    Chapter  Google Scholar 

  4. Aragon-Camarasa, G., Siebert, J.P.: Unsupervised clustering in Hough space for recognition of multiple instances of the same object in a cluttered scene. Pattern Recogn. Lett. 31(11), 1274–1284 (2010)

    Article  Google Scholar 

  5. Torralba, A., Murphy, K.P., Freeman, W.T.: Sharing visual features for multiclass and multiview object detection. IEEE Trans. Pattern Anal. Mach. Intell. 29(5), 854–869 (2007)

    Article  Google Scholar 

  6. He, H., Chen, S.: IMORL: Incremental multiple-object recognition and localization. IEEE Trans. Neural Netw. 19(10), 1727–1738 (2008)

    Article  Google Scholar 

  7. Foresti, G.L., Regazzoni, C.S.: A change-detection method for multiple object localization in real scenes. In: Proceedings of IECON’94-20th Annual Conference of IEEE Industrial Electronics, vol. 2, pp. 984–987. IEEE September 1994

    Google Scholar 

  8. Lowe, D.G.: Object recognition from local scale-invariant features. In: Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 2, pp. 1150–1157. IEEE, September 1999

    Google Scholar 

  9. Bay, H., Tuytelaars, T., Van Gool, L.: SURF: speeded up robust features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 404–417. Springer, Heidelberg (2006). https://doi.org/10.1007/11744023_32

    Chapter  Google Scholar 

  10. Thakoor, K.A., et al.: Attention biased speeded up robust features (ab-surf): a neurally-inspired object recognition algorithm for a wearable aid for the visually-impaired. In 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), pp. 1–6. IEEE, July 2013

    Google Scholar 

  11. Mohtaram, N., Radgui, A., Caron, G., Mouaddib, E.M.: mift: affine-mirror invariant feature transform. In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 893–897. IEEE, October 2018

    Google Scholar 

  12. Yu, G., Morel, J.M.: ASIFT: an algorithm for fully affine invariant comparison. Image Process. On Line 1, 11–38 (2011)

    Article  Google Scholar 

  13. Abdel-Hakim, A.E., Farag, A.A.: CSIFT: A SIFT descriptor with color invariant characteristics. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), vol. 2, pp. 1978–1983. IEEE, June 2006

    Google Scholar 

  14. Marder, M., Harary, S., Ribak, A., Tzur, Y., Alpert, S., Tzadok, A.: Using image analytics to monitor retail store shelves. IBM J. Res. Dev. 59(2/3), 1–3 (2015)

    Article  Google Scholar 

  15. Varol, G., Kuzu, R.S.: Toward retail product recognition on grocery shelves. In: Sixth International Conference on Graphic and Image Processing (ICGIP 2014), vol. 9443, pp. 46–52. SPIE, March 2015

    Google Scholar 

  16. Liu, J., Liu, Y.: Grasp recurring patterns from a single view. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2003–2010 (2013)

    Google Scholar 

  17. Papageorgiou, C.P., Oren, M., Poggio, T.: A general framework for object detection. In: Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271), pp. 555–562. IEEE, January 1998

    Google Scholar 

  18. Diplaros, A., Gevers, T., Patras, I.: Combining color and shape information for illumination-viewpoint invariant object recognition. IEEE Trans. Image Process. 15(1), 1–11 (2005)

    Article  Google Scholar 

  19. LeCun, Y.A., Bottou, L., Orr, G.B., Müller, K.-R.: Efficient BackProp. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. LNCS, vol. 7700, pp. 9–48. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35289-8_3

    Chapter  Google Scholar 

  20. Bengio, Y., Courville, A., Vincent, P.: Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1798–1828 (2013)

    Article  Google Scholar 

  21. Zientara, P., et al.: A multitask grocery assistance system for the visually impaired smart glasses, gloves, and shopping carts provide auditory and tactile feedback (2017)

    Google Scholar 

  22. Franco, A., Maltoni, D., Papi, S.: Grocery product detection and recognition. Exp. Syst. Appl. 81, 163–176 (2017)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. BELIVE AI Research Lab, Amiens, France

    Noureddine Mohtaram & Farouk Achakir

Authors
  1. Noureddine Mohtaram
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Farouk Achakir
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Noureddine Mohtaram .

Editor information

Editors and Affiliations

  1. University of Nevada, Reno, NV, USA

    George Bebis

  2. University of Illinois Urbana-Champaign, Urbana, IL, USA

    Bo Li

  3. National University of Singapore, Singapore, Singapore

    Angela Yao

  4. Microsoft Research Asia, Beijing, China

    Yang Liu

  5. University of Missouri, Columbia, MO, USA

    Ye Duan

  6. City University of Hong Kong, Kowloon, Hong Kong

    Manfred Lau

  7. Idaho National Laboratory, Idaho Falls, ID, USA

    Rajiv Khadka

  8. Salesforce, Seattle, WA, USA

    Ana Crisan

  9. Tufts University, Medford, MA, USA

    Remco Chang

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mohtaram, N., Achakir, F. (2022). Automatic Detection and Recognition of Products and Planogram Conformity Analysis in Real Time on Store Shelves. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2022. Lecture Notes in Computer Science, vol 13599. Springer, Cham. https://doi.org/10.1007/978-3-031-20716-7_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20716-7_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20715-0

  • Online ISBN: 978-3-031-20716-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics