Evaluating the concept specialization distance from an end-user perspective: The case of AGROVOC

The common understanding of generalization/specialization relations assumes the relation to be equally strong between a classifier and any of its related classifiers and also at every level of the hierarchy. Assigning a grade of relative distance to represent the level of similarity between the related pairs of classifiers could correct this situation, which has been considered as an oversimplification of the psychological account of the real-world relations. The paper aims to discuss these issues.

The evaluation followed an end-user perspective. In order to obtain a consistent data set of specialization distances, a group of 21 persons was asked to assign values to a set of relations from a selection of terms from the AGROVOC thesaurus. Then two sets of representations of the relations between the terms were built, one according to the calculated concept of specialization weights and the other one following the original order of the thesaurus. In total, 40 persons were asked to choose between the two sets following an A/B test-like experiment. Finally, short interviews were carried out after the test to inquiry about their decisions.

The results show that the use of this information could be a valuable tool for search and information retrieval purposes and for the visual representation of knowledge organization systems (KOS). Furthermore, the methodology followed in the study turned out to be useful for detecting inconsistencies in the thesaurus and could thus be used for quality control and optimization of the hierarchical relations.

The use of this relative distance information, namely, “concept specialization distance,” has been proposed mainly at a theoretical level. In the current experiment, the authors evaluate the potential use of this information from an end-user perspective, not only for text-based interfaces but also its application for the visual representation of KOS. Finally, the methodology followed for the elaboration of the concept specialization distance data set showed potential for detecting possible inconsistencies in KOS.