Skip to main content
SpringerLink
Log in

Ontology Evaluation

  • Chapter
  • First Online:
Handbook on Ontologies

Part of the book series: International Handbooks on Information Systems ((INFOSYS))

Summary

The evaluation of ontologies is still an emerging field. A set of preliminary ideas and frameworks have been suggested in the literature. This chapter collects ontology quality criteria and lays out a common framework for aspects of ontology evaluation. It will present in depth descriptions of these ontology aspects and how to evaluate them. The techniques and ideas collected and presented here will help to uncover errors in ontologies. This chapter concentrates on the automatic, domain- and task-independent evaluation of an ontology.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 349.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 449.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 449.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Harith Alani. Position paper: Ontology construction from online ontologies. In Les Carr, David De Roure, Arun Iyengar, Carole A. Goble, and Michael Dahlin, editors, Proceedings of the 15th International Conference on World Wide Web, WWW 2006, Edinburgh, Scotland, UK, May 23–26, 2006, pages 491–495. ACM, New York, 2006.

    Google Scholar 

  2. Harith Alani and Christopher Brewster. Metrics for ranking ontologies. In Denny Vrandečić, Mari del Carmen Suárez-Figueroa, Aldo Gangemi, and York Sure, editors. Proceedings of the 4th International Workshop on Evaluation of Ontologies for the Web (EON2006) at the 15th International World Wide Web Conference (WWW 2006), Edinburgh, Scotland, May 2006, pages 24–30.

    Google Scholar 

  3. Sean Bechhofer, Frank van Harmelen, Jim Hendler, Ian Horrocks, Deborah L. McGuinness, Peter F. Patel-Schneider, and Lynn Andrea Stein. OWL Web Ontology Language Abstract Reference, 2004. W3C Rec. 10 February 2004.

    Google Scholar 

  4. Dave Beckett. RDF/XML syntax specification (revised). W3C Recommendation, February 2004.

    Google Scholar 

  5. Tim Berners-Lee. Universal Resource Identifiers in WWW: A Unifying Syntax for the Expression of Names and Addresses of Objects on the Network as used in the World-Wide Web. Technical Report 1630, Internet Engineering Task Force, June 1994.

    Google Scholar 

  6. Tim Berners-Lee, James Hendler, and Ora Lassila. The Semantic Web. Scientific American, 284(5):34–43, 2001.

    Article  Google Scholar 

  7. Tim Bray, Dave Hollander, Andrew Layman, and Richard Tobin. Namespaces in XML 1.0 (second edition), 2006. W3C Rec. 16 August 2006.

    Google Scholar 

  8. Tim Bray, Jean Paoli, C. Michael Sperberg-McQueen, Eve Maler, and Francois Yergeau. Extensible markup language (XML) 1.0 (fourth edition), 2006. W3C Rec. 16 August 2006.

    Google Scholar 

  9. Christopher Brewster, Harith Alani, Srinandan Dasmahapatra, and Yorick Wilks. Data-driven ontology evaluation. In Proceedings of the Language Resources and Evaluation Conference (LREC 2004), pages 164–168, Lisbon, Portugal, 2004. European Language Resources Association.

    Google Scholar 

  10. Dan Brickley and Libby Miller. The Friend Of A Friend (FOAF) vocabulary specification, July 2005.

    Google Scholar 

  11. James Clark and Makoto Murata. RELAX NG Specification, December 2001. OASIS committee specification.

    Google Scholar 

  12. Mathieau d’Aquin, Claudio Baldassarre, Larian Gridinoc, Sofia Angeletou, Marta Sabou, and Enrico Motta. Characterizing knowledge on the semantic web with watson. In Denny Vrandečić, Raúl García-Castro, Asunción Gómez-Pérez, York Sure, and Zhisheng Huang, editors, Proceedings of the Workshop on Evaluation of Ontologies and Ontology-Based Tools, 5th International EON Workshop (EON2007) at ISWC/ASWC’07, pages 1–10, Busan, Korea, November 2007.

    Google Scholar 

  13. Li Ding, Tim Finin, Anupam Joshi, Rong Pan, R. Scott Cost, Yun Peng, Pavan Reddivari, Vishal Doshi, and Joel Sachs. Swoogle: A search and metadata engine for the semantic web. In CIKM’04: Proceedings of the Thirteenth ACM International Conference on Information and Knowledge Management, pages 652–659. ACM, New York, NY, 2004.

    Google Scholar 

  14. David C. Fallside and Priscilla Walmsley. XML schema part 0: Primer second edition, 2004. W3C Rec. 28 October 2004.

    Google Scholar 

  15. Christine Fellbaum. WordNet: An Electronic Lexical Database (Language, Speech, and Communication). MIT, New York, 1998.

    MATH  Google Scholar 

  16. Roy Fielding, James Gettys, Jeffrey Mogul, Henrik Frystyk, Larry Masinter, Paul Leach, and Tim Berners-Lee. Hypertext Transfer Protocol – HTTP/1.1. RFC 2616, June 1999.

    Google Scholar 

  17. Aldo Gangemi, Carola Catenacci, Massimiliano Ciaramita, and Jens Lehmann. Ontology evaluation and validation: An integrated formal model for the quality diagnostic task. Technical report, Laboratory of Applied Ontologies – CNR, Rome, Italy, 2005. http://www.loa-cnr.it/Files/OntoEval4OntoDev_Final.pdf.

  18. Aldo Gangemi, Carola Catenaccia, Massimiliano Ciaramita, and Jos Lehmann. Qood grid: A metaontology-based framework for ontology evaluation and selection. In Denny Vrandečić, Mari del Carmen Suárez-Figueroa, Aldo Gangemi, and York Sure, editors. Proceedings of the 4th International Workshop on Evaluation of Ontologies for the Web (EON2006) at the 15th International World Wide Web Conference (WWW 2006), Edinburgh, Scotland, May 2006, pages 8–15.

    Google Scholar 

  19. Asunción Gómez-Pérez. Ontology evaluation. In Steffen Staab and Rudi Studer, editors, Handbook on Ontologies in Information Systems, First Edition, International Handbooks on Information Systems, chapter 13, pages 251–274. Springer, Berlin, 2004.

    Google Scholar 

  20. Jan Grant and Dave Beckett. RDF test cases. W3C Recommendation, February 2004.

    Google Scholar 

  21. Stephan Grimm and Boris Motik. Closed world reasoning in the semantic web through epistemic operators. In Bernardo Cuenca Grau, Ian Horrocks, Bijan Parsia, and Peter Patel-Schneider, editors, Second International Workshop on OWL: Experiences and Directions (OWLED 2006), Galway, Ireland, 2005.

    Google Scholar 

  22. Thomas R. Gruber. Towards principles for the design of ontologies used for knowledge sharing. International Journal of Human-Computer Studies, 43(5/6):907–928, 1995.

    Article  Google Scholar 

  23. Michael Grüninger and Mark S. Fox. Methodology for the design and evaluation of ontologies. In IJCAI95 Workshop on Basic Ontological Issues in Knowledge Sharing, Montreal, 1995.

    Google Scholar 

  24. Peter Haase and Guilin Qi. An analysis of approaches to resolving inconsistencies in DL-based ontologies. In Proceedings of International Workshop on Ontology Dynamics (IWOD’07), pages 97–109, June 2007.

    Google Scholar 

  25. Masahiro Hori, Jérôme Euzenat, and Peter F. Patel-Schneider. OWL Web Ontology Language XML presentation syntax, 2003. W3C Note 11 June 2003.

    Google Scholar 

  26. Matthew Horridge. The Protégé OWL unit test framework, 2005. Website at http://www.co-ode.org/downloads/owlunittest/.

  27. Matthew Horridge, Nick Drummond, John Goodwin, Alan Rector, Robert Stevens, and Hai Wang. The manchester owl syntax. In OWLED2006 Second Workshop on OWL Experiences and Directions, Athens, GA, USA, 2006.

    Google Scholar 

  28. Ian Horrocks and Peter F. Patel-Schneider. Reducing OWL Entailment to Description Logic Satisfiability. Journal of Web Semantics, 1(4):7–26, 2004.

    Article  Google Scholar 

  29. Aleks Jakulin and Dunja Mladenić. Ontology grounding. In Proceedings of 8th International Multi-conference Information Society IS-2005, pages 170–173, 2005.

    Google Scholar 

  30. Aditya Kalyanpur, Bijan Parsia, Evren Sirin, Bernardo Cuenca Grau, and James Hendler. Swoop: A web ontology editing browser. Journal of Web Semantics: Science, Services and Agents on the World Wide Web, 4(2):144–153, June 2006.

    Google Scholar 

  31. Graham Klyne and Jeremy Carroll. Resource Description Framework (RDF): Concepts and abstract syntax. W3C Recommendation 10 February 2004.

    Google Scholar 

  32. Konstantinos Kotis, George A. Vouros, and Jerónimo P. Alonso. HCOME: A tool-supported methodology for engineering living ontologies. Semantic Web and Databases, pages 155–166, 2005.

    Google Scholar 

  33. Markus Krötzsch, Denny Vrandečić, Max Völkel, Heiko Haller, and Rudi Studer. Semantic wikipedia. Journal of Web Semantics, 5:251–261, 2007.

    Article  Google Scholar 

  34. Joey Lam. Methods for Resolving Inconsistencies in Ontologies. PhD thesis, University of Aberdeen, 2007.

    Google Scholar 

  35. Adolfo Lozano-Tello and Asunción Gómez-Pérez. OntoMetric: A method to choose the appropriate ontology. Journal of Database Management Special Issue on Ontological analysis, Evaluation, and Engineering of Business Systems Analysis Methods, 15(2), 2004.

    Google Scholar 

  36. Boris Motik and Ian Horrocks. Problems with OWL syntax. In OWLED2006 Second Workshop on OWL Experiences and Directions, Athens, GA, 2006.

    Google Scholar 

  37. Boris Motik, Ian Horrocks, and Ulrike Sattler. Adding Integrity Constraints to OWL. In Christine Golbreich, Aditya Kalyanpur, and Bijan Parsia, editors, Third International Workshop on OWL: Experiences and Directions 2007 (OWLED 2007), Innsbruck, Austria, June 6–7, 2007.

    Google Scholar 

  38. Leo Obrst, Werner Ceusters, Inderjeet Mani, Steve Ray, and Barry Smith. The evaluation of ontologies. In Christopher J.O. Baker and Kei-Hoi Cheung, editors, Revolutionizing Knowledge Discovery in the Life Sciences, chapter 7, pages 139–158. Springer, Berlin, 2007.

    Google Scholar 

  39. Eyal Oren, Renaud Delbru, Michele Catasta, Richard Cyganiak, Holger Stenzhorn, and Giovannia Tummarello. Sindice.com: A document-oriented lookup index for open linked data. International Journal of Metadata, Semantics and Ontologies, 3(1), 2008.

    Google Scholar 

  40. Chintan Patel, Kaustubh Supekar, Yugyung Lee, and E. K. Park. OntoKhoj: A semantic web portal for ontology searching, ranking and classification. In Proceedings of Fifth ACM International Workshop on Web Information and Data Management, pages 58–61, New York, NY, 2003.

    Google Scholar 

  41. Peter Patel-Schneider, Patrick Hayes, and Ian Horrocks. OWL Web Ontology Language Abstract Syntax and Semantics, 2004. W3C Rec. 10 February 2004.

    Google Scholar 

  42. Addison Phillips and Mark Davis. Tags for Identifying Languages. Technical Report 4646, Internet Engineering Task Force, September 2006.

    Google Scholar 

  43. Robert Porzel and Rainer Malaka. A task-based approach for ontology evaluation. In Paul Buitelaar, Siegrfried Handschuh, and Bernardo Magnini, editors, Proceedings of ECAI 2004 Workshop on Ontology Learning and Population, Valencia, Spain, August 2004.

    Google Scholar 

  44. Sebastian Rudolph, Johanna Völker, and Pascal Hitzler. Supporting lexical ontology learning by relational exploration. In Uta Priss, Simon Polovina, and Richard Hill, editors, Conceptual Structures: Knowledge Architectures for Smart Applications, Proc. ICCS 2007, volume 4604 of LNAI, pages 488–491. Springer, Sheffield, UK, 2007.

    Google Scholar 

  45. Marta Sabou, Jorge Gracia, Sofia Angeletou, Mathieu d’Aquin, and Enrico Motta. Evaluating the semantic web: A task-based approach. In Proceedings of the 6th International Semantic Web Conference (ISWC’07), pages 423–437, November 2007.

    Google Scholar 

  46. Leo Sauermann and Richard Cyganiak. Cool URIs for the semantic web. Interest group note, W3C, March 2008.

    Google Scholar 

  47. Andrea Schaerf. Reasoning with individuals in concept languages. Data and Knowlegde Engineering, 13(2):141–176, 1994.

    Article  Google Scholar 

  48. Michael K. Smith, Chris Welty, and Deborah McGuinness. OWL Web Ontology Language Guide, 2004. W3C Rec. 10 February 2004.

    Google Scholar 

  49. Aaron Swartz. RDF site summary (RSS) 1.0, December 2000. Official specification.

    Google Scholar 

  50. Samir Tartir, I. Budak Arpinar, Michael Moore, Amit P. Sheth, and Boanerges Aleman-Meza. OntoQA: Metric-based ontology quality analysis. In Proceedings of IEEE Workshop on Knowledge Acquisition from Distributed, Autonomous, Semantically Heterogeneous Data and Knowledge Sources, pages 45–53, 2005.

    Google Scholar 

  51. Christoph Tempich, Helena Sofia Pinto, York Sure, and Steffen Staab. An argumentation ontology for DIstributed, Loosely-controlled and evolvInG Engineering processes of oNTologies (DILIGENT). In Asunción Gómez-Pérez and Jérôme Euzenat, editors, Proceedings of the Second European Semantic Web Conference, volume 3532, pages 241–256, Heraklion, Crete, Greece. Springer, Berlin, 2005.

    Google Scholar 

  52. Paola Velardi, Roberto Navigli, Alessandro Cucchiarelli, and Francesca Neri. Evaluation of OntoLearn, a methodology for automatic population of domain ontologies. In Paul Buitelaar, Philipp Cimiano, and Bernardo Magnini, editors, Ontology Learning from Text: Methods, Applications and Evaluation. IOS Press, Amsterdam, 2005.

    Google Scholar 

  53. Johanna Völker, Denny Vrandečić, and York Sure. Automatic evaluation of ontologies (AEON). In Yolanda Gil, Enrico Motta, V. Richard Benjamins, and Mark A. Musen, editors, Proceedings of the Fourth International Semantic Web Conference (ISWC’05), volume 3729 of LNCS. Springer, Berlin, 2005.

    Google Scholar 

  54. Denny Vrandečić. Explicit knowledge engineering patterns with macros. In Chris Welty and Aldo Gangemi, editors, Proceedings of the Ontology Patterns for the Semantic Web Workshop at the ISWC 2005, Galway, Ireland, November 2005.

    Google Scholar 

  55. Denny Vrandečić and Aldo Gangemi. Unit tests for ontologies. In Mustafa Jarrar, Claude Ostyn, Werner Ceusters, and Andreas Persidis, editors, Proceedings of the 1th International Workshop on Ontology Content and Evaluation in Enterprise, LNCS, Montpellier, France, October. Springer, Berlin, 2006.

    Google Scholar 

  56. Denny Vrandečić and Aldo Gangemi. Unit tests for ontologies. In Mustafa Jarrar, Claude Ostyn, Werner Ceusters, and Andreas Persidis, editors, Proceedings of the 1st International Workshop on Ontology Content and Evaluation in Enterprise, LNCS, Montpellier, France, October. Springer, Berlin, 2006.

    Google Scholar 

  57. Denny Vrandečić and York Sure. How to design better ontology metrics. In Wolfgang May and Michael Kifer, editors, Proceedings of the 4th European Semantic Web Conference (ESWC’07), Innsbruck, Austria. Springer, Berlin, 2007.

    Google Scholar 

  58. Denny Vrandečić, Johanna Völker, Peter Haase, Duc Thanh Tran, and Philipp Cimiano. A metamodel for annotations of ontology elements in OWL DL. In York Sure, Saartje Brockmans, and Jürgen Jung, editors, Proceedings of the Second Workshop on Ontologies and Meta-Modeling, Karlsruhe, Germany, October 2006. GI Gesellschaft für Informatik.

    Google Scholar 

  59. Taowei David Wang and Bijan Parsia. Ontology performance profiling and model examination: First steps. In Karl Aberer, Key-Sun Choi, Natasha Fridman Noy, Dean Allemang, Kyung-Il Lee, Lyndon J. B. Nixon, Jennifer Golbeck, Peter Mika, Diana Maynard, Riichiro Mizoguchi, Guus Schreiber, and Philippe Cudré-Mauroux, editors, Proceedings of the 6th International Semantic Web Conference/2nd Asian Semantic Web Conference (ISWC/ASWC’07), volume 4825 of Lecture Notes in Computer Science, pages 595–608, Busan, Korea, November. Springer, Berlin, 2007.

    Google Scholar 

  60. Taowei David Wang, Bijan Parsia, and James A. Hendler. A survey of the web ontology landscape. In Isabel Cruz, Stefan Decker, Dean Allemang Chris Preist, Daniel Schwabe, Peter Mika, Michael Uschold, and Lora Aroyo, editors, Proceedings of the Fifth International Semantic Web Conference (ISWC’06), volume 4273 of Lecture Notes in Computer Science, pages 682–694, Athens, Georgia, November 2006. Springer.

    Google Scholar 

  61. Jonathan Yu, James A. Thom, and Audrey Tam. Ontology evaluation using wikipedia categories for browsing. In Proceedings of the ACM Sixteenth Conference on Information and Knowledge Management (CIKM), Lisboa, Portugal, November 2007. ACM.

    Google Scholar 

Download references

Acknowledgements

The ideas presented in this chapter have been collected and explored in discussions with many others. I want to especially thank York Sure, Aldo Gangemi, Philipp Cimiano, Johanna Völker, Enrico Motta, Elena Paslaru Bontas Simperl, Marta Sabou, Markus Krötzsch, Anupriya Ankolekar, Boris Motik, Stephan Grimm, Hans-Jörg Happel, Malvina Nissim, Jos Lehmann, Valentina Presutti, Joey Lam, Christoph Tempich, Peter Haase, Dan Connolly, Rinke Hoekstra, Sebastian Rudolph, Chris Welty, Aleks Jakulin, and the participants of the EON 2006 workshop in Edinburgh, Scotland, and the EON 2007 workshop in Busan, South Korea.

Author information

Authors and Affiliations

  1. Institut AIFB, Universität Karlsruhe (TH), Karlsruhe, Germany

    Denny Vrandečić

Authors
  1. Denny Vrandečić
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Denny Vrandečić .

Editor information

Editors and Affiliations

  1. Inst. Informatik, Universität Koblenz-Landau, Universitaetsstr. 1, Koblenz, 56016, Germany

    Steffen Staab

  2. Inst. Angewandte Informatik und, Universität Karlsruhe, Karlsruhe, 76128, Germany

    Rudi Studer

Rights and permissions

Reprints and permissions

Copyright information

© 2009 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Vrandečić, D. (2009). Ontology Evaluation. In: Staab, S., Studer, R. (eds) Handbook on Ontologies. International Handbooks on Information Systems. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-92673-3_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-92673-3_13

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-70999-2

  • Online ISBN: 978-3-540-92673-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation