Elsevier

Journal of Web Semantics

Volume 39, August 2016, Pages 81-96
Journal of Web Semantics

Visual query interfaces for semantic datasets: An evaluation study

https://doi.org/10.1016/j.websem.2016.01.002Get rights and content

Abstract

The rapid growth of the Linked Open Data cloud, as well as the increasing ability to lift relational enterprise datasets to a semantic, ontology-based level means that vast amounts of information are now available in a representation that closely matches the conceptualizations of the potential users of this information. This makes it interesting to create ontology based, user-oriented tools for searching and exploring this data. Although initial efforts were intended for tech users with knowledge of SPARQL/RDF, there are ongoing proposals designed for lay users. One of the most promising approaches is to use visual query interfaces, but more user studies are needed to assess their effectiveness. In this paper, we compare the effect on usability of two important paradigms for ontology-based query interfaces: form-based and graph-based interfaces. In order to reduce the number of variables affecting the comparison, we performed a user study with two state-of-the-art query tools developed by ourselves, sharing a large part of the code base: the graph-based tool OptiqueVQS*, and the form-based tool PepeSearch. We evaluated these tools in a formal comparison study with 15 participants searching a Linked Open Data version of the Norwegian Company Registry. Participants had to respond to 6 non-trivial search tasks using alternately OptiqueVQS* and PepeSearch. Even without previous training, retrieval performance and user confidence were very high, thus suggesting that both interface designs are effective for searching RDF datasets. Expert searchers had a clear preference for the graph-based interface, and mainstream searchers obtained better performance and confidence with the form-based interface. While a number of participants spontaneously praised the capability of the graph interface for composing complex queries, our results evidence that graph interfaces are difficult to grasp. In contrast, form interfaces are more learnable and relieve problems with disorientation for mainstream users. We have also observed positive results introducing faceted search and dynamic term suggestion in semantic search interfaces.

Introduction

The increasing availability of Linked Data is changing the ways that developers design for interaction with Web content. Heath  [1] outlines this shift in metaphor, away from the current document-centric Web, to one in which users are interacting with things (data and objects) and the connections between them. This is especially true for interfaces for retrieving information from the Web. Existing Linked Data can be used in everyday tasks, such as making decisions about a car purchase or researching the potential success of opening a new organic-food shop. However, it is unclear how average Web users can find and digest Linked Data in order to fulfill their information needs, i.e. without requiring specific knowledge of RDF and SPARQL. The ubiquitous document retrieval style interfaces for organizing and locating Web pages (i.e. documents) are not meant for finding Linked Data. One of the challenges of designing for the Semantic Web includes finding new ways to allow people to use this content  [2]. Unfortunately, most tools available are not easily used by searchers having few-to-no technical skills  [3]. Most are SPARQL query interfaces that require a user to learn how to write SPARQL syntax to access triplestore endpoints.

The lack of intuitive, non-technical, and novice-friendly search interfaces essentially blocks many ordinary users from gaining access to published Linked Data. To overcome this problem, there have been attempts at developing Linked Data search tools that hide SPARQL syntax from the user. Of these tools, the present types of search statement input styles include: natural language, keyword-based, form or template-based, and other visual approaches such as graphs. Some user interfaces (UIs) employ only one style of interaction at input and others use multiple (mixed) styles. For example, K-search  [4] presents an ontology tree view combined with a form-based query entry interface. Results of queries, the retrieved RDF content, are often presented as tables and lists, or as graphical data objects.

Although various types of search interfaces have been developed and reported in the literature, there is a lack of empirical evidence of the effectiveness of these approaches  [5]. Innovative search interfaces for querying RDF triples are often described technically and with little discussion of design for human interaction. It is not often that they have conducted usability tests. Elbedweihy et al.  [6] published one of the few papers on evaluating search query approaches for the Semantic Web. They state the importance of working towards a comprehensive evaluation framework that provides guidance to developers, including design criteria for task type (e.g. search for facts) and user type (e.g. domain experts lacking technical skills), stating “there are very few studies that have focused on assessing the usability of semantic search systems”.

As we have established, user studies are scarce and this motivates our work, which pursues the proposal of a portable search tool for mainstream users, the non-savvy searcher having little to no knowledge of Semantic Web technical standards, looking for facts and having well-defined searches (as opposed to ill-defined or vaguely defined information needs). For example, users such as journalists who need to know how many people in Norway voted in the last election within a specific geographical region.

The test case for the project was based on a freely accessible government data site, a Linked Open Data version of the Norwegian Company Registry.1 Based on the prior work discussed above, we identified form-based interfaces as best for the mainstream user  [6], and the PepeSearch interface was therefore constructed as a multi-class form, taking inspiration from facet-based interfaces. This design is compared to the graph-based interface of OptiqueVQS  [7] that displays the underlying classes and relationships in a visual query-building environment intended for domain experts.

In this paper we present the results of a comparison study between OptiqueVQS* and PepeSearch. We collect user feedback on both interfaces, asking participants to complete specific search tasks and to report their satisfaction with the tools. Our contribution is to determine whether graph-based and form-based interfaces are effective for non-tech users in terms of retrieval performance and usability. We hypothesize that for mainstream users, form interfaces will outperform graph interfaces, and that satisfaction scores will be higher. We look at interface strengths and weaknesses, user feelings of being in control, and disorientation. In addition, we investigate user confidence in the result sets. This study fits into a larger framework that will bring further knowledge to the community of developers working on Semantic Web challenges  [2], [8], addressing some of the current shortcomings of understanding the average user’s perspective and needs for Semantic Web tools.

We organize the paper by first presenting relevant background on the definition of mainstream users and semantic search, as well as previously developed interfaces and evaluation studies that are related to our design. We then describe the two interfaces in detail before moving on to report the methods. Results are presented in detail followed by a discussion section that includes both limitations and plans for future work.

Section snippets

Semantic search for mainstream users

The Semantic Web is moving out of the stage where only programmers and those experts with the technical skills necessary for working with raw RDF are using it. Efforts to expose data openly to citizens and customers for analysis and reuse have been made. Examples of early adopters of Linked Data for the mainstream have included open government data, such as Data.gov in the USA  [9]. News media have also worked toward opening up information sources, for instance the Guardian’s Open Platform.2

Experimental setup

With our experimental tools in place, we aimed to gain insight into the following hypotheses:

  • 1.

    Visual query interfaces, like PepeSearch and OptiqueVQS*, are both effective for non-tech users without specific knowledge of SPARQL or RDF.

  • 2.

    Form-based interfaces such as PepeSearch are adequate for mainstream searchers needing simple and learnable user interfaces.

  • 3.

    Graph-based interfaces such as OptiqueVQS* are adequate for expert searchers requiring expressive queries.

We designed our study as a formal

Evaluation results

We present here the results of the study, beginning with a quantitative analysis of search performance and participants’ confidence. Then, we discuss the usability results, as well as the main strengths and weaknesses of the experimental tools.

Discussion

As a wrap-up, we interlink the insight obtained from the qualitative and quantitative methods, and generalize our results to a comparison between graph-based and form-based search interfaces. Table 10 presents the main findings of this analysis with the supporting data.

One of the main goals of this study was to assess whether the two experimental search tools are effective for searching RDF datasets. Obtained results show that both OptiqueVQS* and PepeSearch seem adequate for this purpose,

Conclusions and future work

This paper presents the results of a user study and contributes to further guidelines when designing semantic search interfaces. It was our goal to demonstrate that the PepeSearch and the OptiqueVQS* interfaces can be used to successfully search an RDF datastore. To briefly summarize, we have evidence that expert searchers prefer graph-based to form-based interfaces, mainly due to the increased expressiveness of graph-based query editors. However, form-based interfaces are more easily learned

References (48)

  • A. Soylu et al.

    Experiencing OptiqueVQS: a multi-paradigm and ontology-based visual query system for end users

    Univer. Access Inf. Soc.

    (2015)
  • T. Heath, M. Dzbor, E. Motta, Supporting user tasks and context: Challenges for semantic web research, in: Proceedings...
  • L. Ding, D. Difranzo, A. Graves, J.R. Michaelis, X. Li, D.L. McGuinness, J. Hendler, Data-gov wiki: Towards linking...
  • A. Dadzie et al.

    Approaches to visualising linked data: A survey

    Semant. Web

    (2011)
  • L. Battle, Preliminary analysis of users and tasks for the semantic web, in: Proceedings of the 3rd International...
  • E. Kaufmann et al.

    How useful are natural language interfaces to the semantic web for casual end-users?

  • K. Elbedweihy

    Effective, usable and learnable semantic search

    (2014)
  • M. Hearst

    Search User Interfaces

    (2009)
  • B.M. Wildemuth et al.

    Search tasks and their role in studies of search behaviors

  • A. Broder

    A taxonomy of web search

    SIGIR forum

    (2002)
  • T. Berners-Lee et al.

    The semantic web

    Sci. Am.

    (2001)
  • J. Nielsen, Usability 101: Introduction to Usability, 2012, URL:...
  • A. Freitas et al.

    Querying heterogeneous datasets on the linked data web: Challenges, approaches, and trends

    IEEE Internet Comput.

    (2012)
  • D.V. Camarda, S. Mazzini, A. Antonuccio, LodLive, exploring the web of data, in: Proceedings of the 8th International...
  • Cited by (0)

    This research has been partially funded by the Norwegian Research Council through the Semicolon II project (http://www.semicolon.no/), and the European Commission through the Optique (FP7 GA 318338) and BYTE (FP7 GA 619551) projects. The authors also want to thank the participants in the user study.

    View full text