Abstract
This paper addresses the problem of retrieving meaningful geometric information implied in image data. We outline a general algorithmic scheme to solve the problem in any geometric domain. The scheme, which depends on the domain, may lead to concrete algorithms when the domain is properly and formally specified. Taking plane Euclidean geometry \({\mathbb {E}}\) as an example of the domain, we show how to formally specify \({\mathbb {E}}\) and how to concretize the scheme to yield algorithms for the retrieval of meaningful geometric information in \({\mathbb {E}}\). For images of hand-drawn diagrams in \({\mathbb {E}}\), we present concrete algorithms to retrieve typical geometric objects and geometric relations, as well as their labels, and demonstrate the feasibility of our algorithms with experiments. An example is presented to illustrate how nontrivial geometric theorems can be generated from retrieved geometric objects and relations and thus how implied geometric knowledge may be discovered automatically from images.
Similar content being viewed by others
Notes
Note that the OCR Engine has to be trained with a dataset of samples before it is used for recognition.
We combine circle detection and line detection in \({\texttt {recognizeCircleAndLines}}(\mathrm {I}_2)\) because our algorithm takes into account the connection between the two kinds of detection. It may be desirable to split \({\texttt {recognizeCircleAndLines}}(\mathrm {I}_2)\) into two algorithms, one for each kind of detection.
Binarization serves to separate the background and the foreground of the grayscale image.
Resizing serves for improving the efficiency of retrieval as the size of images produced by photographing is usually larger than needed.
The first two conditions indicate that A and B are both incident to the line passing through \(P_0\) and \(P_{n-1}\), which can be verified by using condition (3) presented in Sect. 3.4, and are both between \(P_0\) and \(P_{n-1}\). The quantity \(\mu \) is introduced to measure the degree of correct recognition.
References
An W, Chen X, Wang D (2016) Searching for geometric theorems using features retrieved from diagrams. In: Kotsireas IS, Rump SM, Yap CK (eds) Mathematical aspects of computer and information sciences, LNCS 9582. Springer, Heidelberg, pp 383–397
ADG (1996–2014) Proceedings of international workshop on automated deduction in geometry. http://dblp.uni-trier.de/db/conf/adg/index.html. Accessed 4 Aug 2016
Balbiani P, Dugat V, Fariñas del Cerro L, Lopez A (1994) Eléments de géométrie mécanique. Hermès, Paris
Ballard DH (1981) Generalizing the Hough transform to detect arbitrary shapes. Pattern Recogn 13(2):111–122
Banerjee B, Chandrasekaran B (2010) A constraint satisfaction framework for executing perceptions and actions in diagrammatic reasoning. J Artif Intell Res 39(1):373–427
Beeson M, Wos L (2014) OTTER proofs in Tarskian geometry. In: Demri S, Kapur D, Weidenbach C (eds) Automated reasoning, LNAI 8562. Springer, Heidelberg, pp 495–510
Belongie S, Malik J, Puzicha J (2002) Shape matching and object recognition using shape contexts. IEEE Trans Pattern Anal Mach Intell 24(4):509–522
Bhatt M, Lee JH, Schultz CPL (2011) CLP(QS): a declarative spatial reasoning framework. In: Egenhofer M, Giudice N, Moratz R, Worboys M (eds) Spatial information theory, LNCS 6899. Springer, Heidelberg, pp 210–230
Buchberger B (1985) Gröbner bases: an algorithmic method in polynomial ideal theory. In: Bose NK (ed) Multidimensional systems theory. D. Reidel, Dordrecht, pp 184–232
Bulko WC (1988) Understanding text with an accompanying diagram. In: Ali M (ed) Proceedings of the 1st international conference on industrial and engineering applications of aritificial intelligence and expert systems, vol 2. ACM Press, New York, pp 894–898
Chen TC, Chung KL (2001) A new randomized algorithm for detecting lines. Real-Time Imaging 7(6):473–481
Chen TC, Chung KL (2001) An efficient randomized algorithm for detecting circles. Comput Vis Image Underst 83(2):172–191
Chen X, Wang D (2012) Management of geometric knowledge in textbooks. Data Knowl Eng 73:43–57
Chen X, Wang D (2013) Formalization and specification of geometric knowledge objects. Math Comput Sci 7(4):439–454
Chen X (2014) Representation and automated transformation of geometric statements. J Syst Sci Complex 27(2):382–412
Chen X, Song D, Wang D (2015) Automated generation of geometric theorems from images of diagrams. Ann Math Artif Intell 74(3–4):333–358
Chou S-C, Gao X-S, Zhang J-Z (1994) Machine proofs in geometry: automated production of readable proofs for geometry theorems. World Scientific, Singapore
Chou S-C, Gao X-S (2001) Automated reasoning in geometry. In: Handbook of automated reasoning, chap 10, vol I. MIT Press, Cambridge, pp 623–665
Cohn AG, Hazarika SM (2001) Qualitative spatial representation and reasoning: an overview. Fundam Inf 46(1–2):1–29
Dehlinger C, Dufourd J-F, Schreck P (2001) Higher-order intuitionistic formalization and proofs in Hilberts elementary geometry. In: Richter-Gebert J, Wang D (eds) Automated deduction in geometry, LNAI 2061. Springer, Heidelberg, pp 306–323
Duda RO, Hart PE (1972) Use of the Hough transformation to detect lines and curves in pictures. Commun ACM 15(1):11–15
Ferguson RW, Forbus KD (1998) Telling juxtapositions: using repetition and alignable difference in diagram understanding. In: Holyoak K, Gentner D, Kokinov B (eds) Advances in analogy research: integration of theory and data from the cognitive, computational, and neural sciences, NBU Series in Cognitive Science. New Bulgarian University, Sofia, pp 109–117
Forbus K, Usher J, Lovett A, Lockwood K, Wetzel J (2011) CogSketch: sketch understanding for cognitive science research and for education. Top Cogn Sci 3(4):648–666
Gelernter H (1995) Realization of a geometry theorem proving machine. In: Feigenbaum EA, Feldman J (eds) Computers and thought. MIT Press, Cambridge, pp 134–152
GeoGebra. http://www.geogebra.org/cms/. Accessed 4 Aug 2016
GEOTHER. http://www-polsys.lip6.fr/wang/GEOTHER/. Accessed 4 Aug 2016
Ghimire D, Lee J (2013) Geometric feature-based facial expression recognition in image sequences using multi-class AdaBoost and support vector machines. Sensors 13(6):7714–7734
Giaquinto M (2007) Visual thinking in mathematics: an epistemological study. Oxford University Press, New York
Glasgow J, Narayanan NH, Chandrasekaran B (1995) Diagrammatic reasoning: cognitive and computational perspectives. MIT Press, Cambridge
Gonthier G (2008) Formal proof—the four-color theorem. Not AMS 55(11):1382–1393
Goodrum AA (2000) Image information retrieval: an overview of current research. Inf Sci Res Spec Issue Inf Sci 3(2):63–66
Gorman JW, Mitchell OR, Kuhl FP (1988) Partial shape recognition using dynamic programming. IEEE Trans Pattern Anal Mach Intell 10(2):257–266
Hales T, Adams M, Bauer G et al (2015) A formal proof of the Kepler conjecture. arXiv preprint arXiv:1501.02155
Hammond T, Davis R (2002) Tahuti: a geometrical sketch recognition system for UML class diagrams. In: Proceeding of the special interest group on computer graphics 2006 courses, Article No. 25. ACM Press, New York
Hong H, Wang D, Winkler F (eds) (1995) Algebraic approaches to geometric reasoning. Special issue of annals of mathematics and artificial intelligence, vol 13. Baltzer, Basel, pp 1–2
Johnson LE (2015) Automated elementary geometry theorem discovery via inductive diagram manipulation. Master thesis, Massachusetts Institute of Technology
Jorge JA, Fonseca MJ (2000) A simple approach to recognise geometric shapes interactively. In: Chhabra AK, Dori D (eds) Graphics recognition recent advances, LNCS 1941. Springer, Heidelberg, pp 266–274
Jung CR, Schramm R (2004) Rectangle detection based on a windowed Hough transform. In: Proceedings of the 17th Braizilian symposium on computer graphics and image processing. IEEE Computer Society, New York, pp 113–120
Kapur D, Mundy JL (eds) (1989) Geometric reasoning. Special issue of artificial intelligence, vol 37. MIT Press, Cambridge, pp 1–3
Lafarge F, Descombes X (2010) Geometric feature extraction by a multi-marked point process. IEEE Trans Pattern Anal Mach Intell 32(9):1597–1609
Li K, Lu X, Ling H, Liu L, Feng T, Tang Z (2013) Detection of overlapped quadrangles in plane geometric figures. In: Proceedings of the 12th international conference on document analysis and recognition. IEEE Computer Society, Washington, pp 260–264
Matas J, Galambos C, Kittler J (2000) Robust detection of lines using the progressive probabilistic Hough transform. Comput Vis Image Underst 78(1):119–137
Materlik D (2003) Using sketch recognition to enhance the human-computer interface of geometry software. Diploma thesis, Free University of Berlin
McCharen JD, Overbeek RA, Wos L (1976) Problems and experiments for and with automated theorem-proving programs. IEEE Trans Comput 25(8):773–782
Montalvo FS (1990) Diagram understanding: the symbolic descriptions behind the scenes. In: Ichikawa T, Jungert E, Korfhage RR (eds) Visual languages and applications, series of languages and information systems. Plenum Press, New York, pp 5–27
Moon H, Chellappa R, Rosenfeld A (2002) Optimal edge-based shape detection. IEEE Trans Image Process 11(11):1209–1227
Mori S, Nishida H, Yamada H (1999) Optical character recognition. Wiley, New York
Nixon MS, Aguado AS (2008) Feature extraction & image processing, 2nd edn. Academic Press, Oxford, pp 87–88
Novak G (1995) Diagrams for solving physical problems. In: Glasgow J, Narayanan NH, Chandrasekaran B (eds) Diagrammatic reasoning: cognitive and computational perspectives. AAAI Press, Palo Alto, pp 753–774
Quaife A (1992) Automated development of fundamental mathematical theories. Kluwer, Dordrecht
Schultz C, Bhatt M (2015) Spatial symmetry driven pruning strategies for efficient declarative spatial reasoning. In: Fabrikant SI, Raubal M, Bertolotto M, Davies C, Freundschuh S, Bell S (eds) Spatial information theory, LNCS 9368. Springer, Heidelberg, pp 331–353
Seo MJ, Hajishirzi H, Farhadi A, Etzioni O (2014) Diagram understanding in geometry questions. In: Proceedings of the 28th AAAI conference on artificial intelligence. AAAI Press, Palo Alto, pp 2831–2838
Song D, Wang D, Chen X (2015) Discovering geometric theorems from scanned and photographed images of diagrams. In: Botana F, Quaresma P (eds) Automated deduction in geometry, LNAI 9201. Springer, Heidelberg, pp 149–165
Song D, Chen X (2016) Automated generation of keywords from images for geometric information search. In: Prodeedings of the 11th international workshop on automated deduction in geometry (ADG 2016) (Strasbourg, France, June 27–29, 2016), pp 172–189
Sonka M, Hlavac V, Boyle R (2014) Image processing, analysis, and machine vision, 4th edn. Cengage Learning, Stamford
Takahashi K (2012) PLCA: a framework for qualitative spatial reasoning based on connection patterns of regions. In: Qualitative spatio-temporal representation and reasoning: trends and future directions, chap 2. IGI Global, Hershey, pp 63–96
Tangelder JWH, Veltkamp RC (2008) A survey of content based 3D shape retrieval methods. Multimed Tools Appl 39(3):441–471
Tesseract-OCR. http://code.google.com/p/tesseract-ocr/. Accessed 4 Aug 2016
Ulgen F, Flavell AC, Akamatsu N (1995) Geometric shape recognition with fuzzy filtered input to a backpropagation neural network. IEICE Trans Inf Syst 78(2):174–183
Wang D (1996) Geometry machines: from AI to SMC. In: Calmet J, Campbell JA, Pfalzgraf J (eds) Artificial intelligence and symbolic mathematical computation, LNCS 1138. Springer, Heidelberg, pp 213–239
Wang D (2001) Elimination methods. Springer, New York
Wang D (2003) Automated generation of diagrams with Maple and Java. In: Joswig M, Takayama N (eds) Algebra, geometry, and software systems. Springer, Heidelberg, pp 277–287
Wu W-T (1994) Mechanical theorem proving in geometries: basic principles (translated from the Chinese by X. Jin and D. Wang). Springer, New York
Yuen HK, Princen J, Illingworth J, Kittler J (1990) Comparative study of Hough transform methods for circle finding. Image Vis Comput 8(1):71–77
Zhang TY, Suen CY (1984) A fast parallel algorithm for thinning digital patterns. Commun ACM 27(3):236–239
Acknowledgements
The authors wish to thank the referees for their helpful comments on an early version of the paper and acknowledge the support of the research funds from the State Key Laboratory of Software Development Environment under Grant Nos. SKLSDE-2015ZX-18 and SKLSDE-2016ZX-18 and from the Central Universities under Grant No. YWF-16-SXXY-01.
Author information
Authors and Affiliations
Corresponding author
Additional information
Responsible editor: Kristian Kersting.
Appendices
Appendix 1: Algorithmic description of \({\texttt {recognizeCircleAndLines}}(\mathrm {I}_2)\)
In Algorithm 4, the subalgorithm
-
\(\texttt {detectKeyPoints}(\mathrm {I}_2)\) is called to recognize key points in \(\mathrm {I}_2\) according to Definition 2;
-
\(\texttt {combineKp2PntInt}(\mathrm {K}_p)\) is called to collect points of interest from clusters of key points in \(\mathrm {K}_p\);
-
\(\texttt {assignPntOfInterest}(\mathrm {C}_{rv}, \mathrm {P}_I)\) is called to assign s to \(c.\textit{spoint}\) and e to \(c.\textit{epoint}\) of each \(c\in \mathrm {C}_{rv}\), where s is a point of interest nearest to \(c.\textit{spoint}\) and e is a point of interest nearest to \(c.\textit{epoint}\);
-
\(\texttt {sortCurves}(\mathrm {C}_{rv})\) is called to sort the elements of \(\mathrm {C}_{rv}\) in a descending order with respect to length.
The above subalgorithms are rather simple and we do not present them formally. The other subalgorithms \(\texttt {detectCurves}(\mathrm {I}_2,\mathrm {K}_p)\), \(\texttt {computeCrvAttributes}(\mathrm {C}_{rv})\), and \(\texttt {determineEntityType}(\mathrm {C}_{rv})\) of Algorithm 4 are formulated as Algorithms 5, 6, and 7 respectively.
In Algorithm 5, \(\texttt {getNextPoint}(S)\) is called to find the next point in the 8-neighbours of the last point of S which does not occur in S, which can be added to S, and which will lead to a key point.
In Algorithm 6,
-
\(\texttt {random}(n)\) generates a randomized integer in [1, n];
-
\(\texttt {getRadi}(T_c[1], T_c[p_{idx}], T_c[n])\) computes the radius of the circle c passing through three points \(T_c[1], T_c[p_{idx}], T_c[n]\);
-
\(\texttt {distance}(T_c[j], \texttt {line}(T_c[1], T_c[n]))\) computes the distance from the point \(T_c[j]\) to the line passing through \(T_c[1]\) and \(T_c[n]\);
-
\(\texttt {polyfit}(T_c[1,\ldots ,N_2],t)\) computes the coefficients of a polynomial of degree t which fits the sequence \(T_c[1,\ldots ,N_2]\) of points;
-
\(\texttt {polySlope}(coef_s,P)\) computes the slope of a line tangent to the curve defined by the polynomial with coefficients \(coef_s\) at the point P;
-
\(\texttt {tangentVecAngle}(slope,T_c[1,\ldots ,N_2])\) computes the directed angle from the vector (1, 0) to \((v_x,slope)\), where \(v_x=1\) if the X-coordinate of \(T_c[1]\) is bigger than \(T_c[N_2]\), and \(v_x=-1\) otherwise.
In Algorithm 7, \(\texttt {combine2curves}(\mathrm {C}_{rv}[i],\mathrm {C}_{rv}[m_i])\) combines the two curves \(\mathrm {C}_{rv}[i]\) and \(\mathrm {C}_{rv}[m_i]\) of points into one curve and then updates the attributes of the new curve.
Appendix 2: Selected experimental results
In Table 5, basic geometric entities and basic geometric relations, listed in the subcolumns under “Basic geometric information”, are represented in the form of T: n, where T is the type of the geometric entity or relation and n is the number of instances of geometric entities or relations of type T retrieved from the image shown in the column “Image of diagram”; “Size” denotes the image size in pixels; “Time” denotes the running time in seconds for retrieving the geometric information from the image.
Rights and permissions
About this article
Cite this article
Song, D., Wang, D. & Chen, X. Retrieving geometric information from images: the case of hand-drawn diagrams. Data Min Knowl Disc 31, 934–971 (2017). https://doi.org/10.1007/s10618-017-0494-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10618-017-0494-1