Skip to main content
SpringerLink
Log in

A performance evaluation protocol for symbol spotting systems in terms of recognition and location indices

  • Original Paper
  • Published:
International Journal on Document Analysis and Recognition (IJDAR) Aims and scope Submit manuscript

Abstract

Symbol spotting systems are intended to retrieve regions of interest from a document image database where the queried symbol is likely to be found. They shall have the ability to recognize and locate graphical symbols in a single step. In this paper, we present a set of measures to evaluate the performance of a symbol spotting system in terms of recognition abilities, location accuracy and scalability. We show that the proposed measures allow to determine the weaknesses and strengths of different methods. In particular we have tested a symbol spotting method based on a set of four different off-the-shelf shape descriptors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Aksoy, S., Ye, M., Schauf, M., Song, M., Wang, Y., Haralick, R.: Algorithm performance contest. In: Proceedings of the Fifteenth International Conference on Pattern Recognition, ICPR00, pp. 870–876 (2000). doi:10.1109/ICPR.2000.903054

  2. Antonacopoulos, A., Bridson, D.: Performance analysis framework for layout analysis methods. In: Proceedings of the Ninth International Conference on Document Analysis and Recognition, ICDAR07, pp. 1258–1262 (2007). doi:10.1109/ICDAR.2007.4377117

  3. Antonacopoulos, A., Gatos, B., Bridson, D.: ICDAR 2005 page segmentation competition. In: Proceedings of the Eighth International Conference on Document Analysis and Recognition, ICDAR05, pp. 75–79 (2005). doi:10.1109/ICDAR.2005.184

  4. Antonacopoulos, A., Gatos, B., Bridson, D.: ICDAR 2007 page segmentation competition. In: Proceedings of the Ninth International Conference on Document Analysis and Recognition, ICDAR07, pp. 1279–1283 (2007) doi:10.1109/ICDAR.2007.203

  5. Antonacopoulos, A., Gatos, B., Karatzas, D.: ICDAR 2003 page segmentation competition. In: Proceedings of the Seventh International Conference on Document Analysis and Recognition, ICDAR03, pp. 688–692 (2003). doi:10.1109/ICDAR.2003.1227750

  6. Antonacopoulos, A., Karatzas, D., Bridson, D.: Ground truth for layout analysis performance evaluation. In: Document Analysis Systems, DAS06. Lecture Notes on Computer Science, vol. 3872, pp. 302–311. Springer, Berlin (2006). doi:10.1007/11669487_27

  7. Barber C., Dobkin D., Huhdanpaa H.: The quickhull algorithm for convex hulls. ACM Transactions on Mathematical Software 22(4), 469–483 (1996). doi:10.1145/235815.235821

    Article  MATH  MathSciNet  Google Scholar 

  8. Buckland M., Gey F.: The relationship between recall and precision. J. Am. Soc. Inform. Sci. 45(1), 12–19 (1994)

    Article  Google Scholar 

  9. Davis, J., Goadrich, M.: The relationship between precision–recall and ROC curves. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 233–240 (2006). doi:10.1145/1143844.1143874

  10. Delalandre, M., Pridmore, T., Valveny, E., Locteau, H., Trupin, E.: Building synthetic graphical documents for performance evaluation. In: Graphics Recognition. Recent Advances and New Opportunities. Lecture Notes on Computer Science, vol. 5046, pp. 288–298. Springer, Berlin (2008). doi:10.1007/978-3-540-88188-9_27

  11. Fawcett T.: An introduction to ROC analysis. Pattern Recognit. Lett. 27(8), 861–874 (2006). doi:10.1016/j.patrec.2005.10.010

    Article  MathSciNet  Google Scholar 

  12. Holz, F., Witschel, H., Heinrich, G., Heyer, G., Teresniak, S.: An evaluation framework for semantic search in p2p networks. In: Proceedings of the Seventh International Workshop on Innovative Internet Community Systems, I2CS07 (2007)

  13. Hripcsak G., Rothschild A.: Agreement, the F-measure, and reliability in information retrieval. J. Am. Med. Inform. Assoc 12(3), 296–298 (2005). doi:10.1016/j.jamia.2005.01.008

    Article  Google Scholar 

  14. Hu M.: Visual pattern recognition by moment invariants. IRE Trans. Inform. Theory 8, 179–187 (1962)

    Google Scholar 

  15. Huijsmans D., Sebe N.: How to complete performance graphs in content-based image retrieval: add generality and normalize scope. IEEE Trans. Pattern Anal. Mach. Intell. 27(2), 245–251 (2005). doi:10.1109/TPAMI.2005.30

    Article  Google Scholar 

  16. Kang, B., Kim, H., Lee, S.: Performance analysis of semantic indexing in text retrieval. In: Computational Linguistics and Intelligent Text Processing. Lecture Notes on Computer Science, vol. 2945, pp. 433–436. Springer, Berlin (2004).doi:10.1007/b95558

  17. Kauppinen H., Seppänen T., Pietikäinen M.: An experimental comparison of autoregressive and fourier-based descriptors in 2d shape classification. IEEE Trans. Pattern Anal. Mach. Intell. 17, 201–207 (1995). doi:10.1109/34.368168

    Article  Google Scholar 

  18. Lambert, G., Gao, H.: Discrimination properties of invariants using the line moments of vectorized contours. In: Proceedings of the 13th International Conference on Pattern Recognition, ICPR96, pp. 735–739 (1996). doi:10.1109/ICPR.1996.546920

  19. Liu W., Dori D.: A protocol for performance evaluation of line detection algorithms. Mach. Vis. Appl. 9(5), 240–250 (1997). doi:10.1007/s001380050045

    Article  Google Scholar 

  20. Liu W., Dori D.: Incremental arc segmentation algorithm and its evaluation. IEEE Trans. Pattern Anal.Mach. Intell. 20(4), 424–431 (1998). doi:10.1109/34.677280

    Article  Google Scholar 

  21. Lopresti, D., Nagy, G.: Issues in ground-truthing graphic documents. In: Graphics Recognition Algorithms and Applications. Lecture Notes on Computer Science, vol. 2390, pp. 46–66. Springer, Berlin (2001). doi:10.1007/3-540-45868-9

  22. Lu, C., Shukla, M., Subramanya, S., Wu, Y.: Performance evaluation of desktop search engines. In: Proceedings of the IEEE International Conference on Information Reuse and Integration, IRI07, pp. 110–115 (2007). doi:10.1109/IRI.2007.4296606

  23. Lucas, S.: ICDAR 2005 text locating competition results. In: Proceedings of the Eighth International Conference on Document Analysis and Recognition, ICDAR05, pp. 80–84 (2005). doi:10.1109/ICDAR.2005.231

  24. Lucas S., Panaretos A., Sosa L., Tang A., Wong S., Young R., Ashida K., Nagai H., Okamoto M., Yamamoto H., Miyao H., Zhu J., Ou W., Wolf C., Jolion J., Todoran L., Worring M., Lin X.: ICDAR 2003 robust reading competitions: entries, results, and future directions. Int. J. Doc. Anal. Recognit. 7(2), 105–122 (2005). doi:10.1007/s10032-004-0134-3

    Article  Google Scholar 

  25. Makhoul, J., Kubala, F., Schwartz, R., Weischedel, R.: Performance measures for information extraction. In: Proceedings of DARPA Broadcast News Workshop (1999)

  26. Marcus, J.: A novel algorithm for HMM word spotting performance evaluation and error analysis. In: Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP92, pp. 89–92 (1992). doi:10.1109/ICASSP.1992.226113

  27. Müller H., Müller W., Squire D., Marchand-Maillet S., Pun T.: Performance evaluation in content-based image retrieval: overview and proposals. Pattern Recognit. Lett. 22(5), 593–601 (2001). doi:10.1016/S0167-8655(00)00118-5

    Article  MATH  Google Scholar 

  28. Neumann, T., Bender, M., Michel, S., Weikum, G.: A reproducible benchmark for P2P retrieval. In: Proceedings of the First International Workshop on Performance and Evaluation of Data Management Systems, ExpDB06, pp. 1–8 (2006)

  29. Phillips I., Chhabra A.: Empirical performance evaluation of graphics recognition systems. IEEE Trans. Pattern Anal. Mach. Intell. 21(9), 849–870 (1999). doi:10.1109/34.790427

    Article  Google Scholar 

  30. Rath, T., Manmatha, R.: Features for word spotting in historical manuscripts. In: Proceedings of the Seventh International Conf. on Document Analysis and Recognition, ICDAR03, pp. 218–222 (2003). doi:10.1109/ICDAR.2003.1227662

  31. van Rijsbergen C.: Information Retrieval. Butterworth-Heinemann Newton, MA, USA (1979)

    Google Scholar 

  32. Rusiñol, M., Lladós, J.: Word and symbol spotting using spatial organization of local descriptors. In: The Eighth IAPR International Workshop on Document Analysis Systems, DAS08, pp. 489–496 (2008). doi:10.1109/DAS.2008.24

  33. Smeaton, A., Over, P., Kraaij, W.: Evaluation campaigns and TRECVid. In: Proceedings of the 8th ACM International Workshop on Multimedia Information Retrieval, pp. 321–330 (2006). doi:10.1145/1178677.1178722

  34. Song J., Su F., Tai C., Cai S.: An object-oriented progressive-simplification-based vectorization system for engineering drawings: Model, algorithm, and performance. IEEE Trans. Pattern Anal. Mach. Intell. 24(8), 1048–1060 (2002). doi:10.1109/TPAMI.2002.1023802

    Article  Google Scholar 

  35. Stoyan D., Stoyan H.: Fractals, Random Shapes and Point Fields (Methods of Geometrical Statistics). John Wiley & Sons, Chichester (1994)

    MATH  Google Scholar 

  36. Tabbone, S., Zuwala, D.: An indexing method for graphical documents. In: Proceedings of the Ninth International Conference on Document Analysis and Recognition, ICDAR07, pp. 789–793 (2007). doi:10.1109/ICDAR.2007.4377023

  37. Tombre, K., Lamiroy, B.: Graphics recognition—from re-engineering to retrieval. In: Proceedings of the Seventh International Conference on Document Analysis and Recognition, ICDAR03, pp. 148–155 (2003)

  38. Valveny, E., Dosch, P.: Symbol recognition contest: a synthesis. In: Graphics Recognition, Recent Advances and Perspectives. Lecture Notes on Computer Science, vol. 3088, pp. 368–385. Springer, Berlin (2004). doi:10.1007/b99011

  39. Valveny, E., Dosch, P.: Report on the second symbol recognition contest. In: Graphics Recognition. Ten Years Review and Future Perspectives. Lecture Notes on Computer Science, vol. 3926, pp. 381–397. Springer, Berlin (2006). doi:10.1007/11767978

  40. Valveny E., Dosch P., Winstanley A., Zhou Y., Yang S., Yan L., Liu W., Elliman D., Delalandre M., Trupin E., Adam S., Ogier J.: A general framework for the evaluation of symbol recognition methods. Int. J. Doc. Anal. Recognit. 9(1), 59–74 (2007). doi:10.1007/s10032-006-0033-x

    Article  Google Scholar 

  41. Wolf C., Jolion J.: Object count/area graphs for the evaluation of object detection and segmentation algorithms. Int. J. Doc. Anal. Recognit. 8(4), 280–296 (2006). doi:10.1007/s10032-006-0014-0

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dept. Ciències de la Computació, Computer Vision Center Edifici O, Universitat Autònoma de Barcelona, 08193, Bellaterra (Barcelona), Spain

    Marçal Rusiñol & Josep Lladós

Authors
  1. Marçal Rusiñol
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Josep Lladós
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marçal Rusiñol.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Rusiñol, M., Lladós, J. A performance evaluation protocol for symbol spotting systems in terms of recognition and location indices. IJDAR 12, 83–96 (2009). https://doi.org/10.1007/s10032-009-0083-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10032-009-0083-y

Keywords

Search

Navigation