Visual query interfaces for semantic datasets: An evaluation study☆
Introduction
The increasing availability of Linked Data is changing the ways that developers design for interaction with Web content. Heath [1] outlines this shift in metaphor, away from the current document-centric Web, to one in which users are interacting with things (data and objects) and the connections between them. This is especially true for interfaces for retrieving information from the Web. Existing Linked Data can be used in everyday tasks, such as making decisions about a car purchase or researching the potential success of opening a new organic-food shop. However, it is unclear how average Web users can find and digest Linked Data in order to fulfill their information needs, i.e. without requiring specific knowledge of RDF and SPARQL. The ubiquitous document retrieval style interfaces for organizing and locating Web pages (i.e. documents) are not meant for finding Linked Data. One of the challenges of designing for the Semantic Web includes finding new ways to allow people to use this content [2]. Unfortunately, most tools available are not easily used by searchers having few-to-no technical skills [3]. Most are SPARQL query interfaces that require a user to learn how to write SPARQL syntax to access triplestore endpoints.
The lack of intuitive, non-technical, and novice-friendly search interfaces essentially blocks many ordinary users from gaining access to published Linked Data. To overcome this problem, there have been attempts at developing Linked Data search tools that hide SPARQL syntax from the user. Of these tools, the present types of search statement input styles include: natural language, keyword-based, form or template-based, and other visual approaches such as graphs. Some user interfaces (UIs) employ only one style of interaction at input and others use multiple (mixed) styles. For example, K-search [4] presents an ontology tree view combined with a form-based query entry interface. Results of queries, the retrieved RDF content, are often presented as tables and lists, or as graphical data objects.
Although various types of search interfaces have been developed and reported in the literature, there is a lack of empirical evidence of the effectiveness of these approaches [5]. Innovative search interfaces for querying RDF triples are often described technically and with little discussion of design for human interaction. It is not often that they have conducted usability tests. Elbedweihy et al. [6] published one of the few papers on evaluating search query approaches for the Semantic Web. They state the importance of working towards a comprehensive evaluation framework that provides guidance to developers, including design criteria for task type (e.g. search for facts) and user type (e.g. domain experts lacking technical skills), stating “there are very few studies that have focused on assessing the usability of semantic search systems”.
As we have established, user studies are scarce and this motivates our work, which pursues the proposal of a portable search tool for mainstream users, the non-savvy searcher having little to no knowledge of Semantic Web technical standards, looking for facts and having well-defined searches (as opposed to ill-defined or vaguely defined information needs). For example, users such as journalists who need to know how many people in Norway voted in the last election within a specific geographical region.
The test case for the project was based on a freely accessible government data site, a Linked Open Data version of the Norwegian Company Registry.1 Based on the prior work discussed above, we identified form-based interfaces as best for the mainstream user [6], and the PepeSearch interface was therefore constructed as a multi-class form, taking inspiration from facet-based interfaces. This design is compared to the graph-based interface of OptiqueVQS [7] that displays the underlying classes and relationships in a visual query-building environment intended for domain experts.
In this paper we present the results of a comparison study between OptiqueVQS* and PepeSearch. We collect user feedback on both interfaces, asking participants to complete specific search tasks and to report their satisfaction with the tools. Our contribution is to determine whether graph-based and form-based interfaces are effective for non-tech users in terms of retrieval performance and usability. We hypothesize that for mainstream users, form interfaces will outperform graph interfaces, and that satisfaction scores will be higher. We look at interface strengths and weaknesses, user feelings of being in control, and disorientation. In addition, we investigate user confidence in the result sets. This study fits into a larger framework that will bring further knowledge to the community of developers working on Semantic Web challenges [2], [8], addressing some of the current shortcomings of understanding the average user’s perspective and needs for Semantic Web tools.
We organize the paper by first presenting relevant background on the definition of mainstream users and semantic search, as well as previously developed interfaces and evaluation studies that are related to our design. We then describe the two interfaces in detail before moving on to report the methods. Results are presented in detail followed by a discussion section that includes both limitations and plans for future work.
Section snippets
Semantic search for mainstream users
The Semantic Web is moving out of the stage where only programmers and those experts with the technical skills necessary for working with raw RDF are using it. Efforts to expose data openly to citizens and customers for analysis and reuse have been made. Examples of early adopters of Linked Data for the mainstream have included open government data, such as Data.gov in the USA [9]. News media have also worked toward opening up information sources, for instance the Guardian’s Open Platform.2
Experimental setup
With our experimental tools in place, we aimed to gain insight into the following hypotheses:
- 1.
Visual query interfaces, like PepeSearch and OptiqueVQS*, are both effective for non-tech users without specific knowledge of SPARQL or RDF.
- 2.
Form-based interfaces such as PepeSearch are adequate for mainstream searchers needing simple and learnable user interfaces.
- 3.
Graph-based interfaces such as OptiqueVQS* are adequate for expert searchers requiring expressive queries.
We designed our study as a formal
Evaluation results
We present here the results of the study, beginning with a quantitative analysis of search performance and participants’ confidence. Then, we discuss the usability results, as well as the main strengths and weaknesses of the experimental tools.
Discussion
As a wrap-up, we interlink the insight obtained from the qualitative and quantitative methods, and generalize our results to a comparison between graph-based and form-based search interfaces. Table 10 presents the main findings of this analysis with the supporting data.
One of the main goals of this study was to assess whether the two experimental search tools are effective for searching RDF datasets. Obtained results show that both OptiqueVQS* and PepeSearch seem adequate for this purpose,
Conclusions and future work
This paper presents the results of a user study and contributes to further guidelines when designing semantic search interfaces. It was our goal to demonstrate that the PepeSearch and the OptiqueVQS* interfaces can be used to successfully search an RDF datastore. To briefly summarize, we have evidence that expert searchers prefer graph-based to form-based interfaces, mainly due to the increased expressiveness of graph-based query editors. However, form-based interfaces are more easily learned
References (48)
- et al.
Evaluating the usability of natural language query languages and interfaces to semantic web knowledge bases
Web Semant.: Sci., Serv. Agents World Wide Web
(2010) - et al.
Repeatable and reliable semantic search evaluation
Web Semant.: Sci., Serv. Agents World Wide Web
(2013) - et al.
Evaluating question answering over linked data
Web Semant.: Sci., Serv. Agents World Wide Web
(2013) - et al.
Semantic search of tools for collaborative learning with the ontoolsearch system
Comput. & Edu.
(2010) A taskonomy for the semantic web
Semant. Web
(2010)- T. Heath, J. Domingue, P. Shabajee, User interaction and uptake challenges to successfully deploying semantic web...
- et al.
Linked data — The story so far
International journal on semantic web and information systems, Spec. Issue Link. data
(2009) - R. Bhagdev, S. Chapman, F. Ciravegna, V. Lanfranchi, D. Petrelli, Hybrid search: Effectively combining keywords and...
Semantic web user interfaces: A systematic mapping study
(2011)- et al.
Evaluating semantic search query approaches with expert and casual users
Experiencing OptiqueVQS: a multi-paradigm and ontology-based visual query system for end users
Univer. Access Inf. Soc.
Approaches to visualising linked data: A survey
Semant. Web
How useful are natural language interfaces to the semantic web for casual end-users?
Effective, usable and learnable semantic search
Search User Interfaces
Search tasks and their role in studies of search behaviors
A taxonomy of web search
SIGIR forum
The semantic web
Sci. Am.
Querying heterogeneous datasets on the linked data web: Challenges, approaches, and trends
IEEE Internet Comput.
Cited by (0)
- ☆
This research has been partially funded by the Norwegian Research Council through the Semicolon II project (http://www.semicolon.no/), and the European Commission through the Optique (FP7 GA 318338) and BYTE (FP7 GA 619551) projects. The authors also want to thank the participants in the user study.