1 Introduction

In the domain of humanities, scholars analyze old literary works in order to derive several kinds of knowledge. One important kind of such knowledge regards the primary sources, i.e. the works of other authors that an author makes reference to in his texts. Usually, scholars encode this knowledge in natural language, and report it in commentaries, which the readers use to understand several aspects of the literary works. However, encoding knowledge in natural language does not allow using computers to make inferences that could be useful to scholars, for example about the distribution of the primary sources and of the authors cited in a work.

Our study is part of the “Towards a Digital Dante Encyclopedia” project, an Italian National Research project supporting scholars, amongst others, in formally expressing the knowledge about primary sources present in Dante’s works and more generally in literary texts. The goal of our work is to represent this kind of knowledge by (i) creating an ontology providing a formal representation of the terms required for expressing knowledge about the primary sources of an author’s works, and (ii) developing a semantic digital library based on the ontology that allows scholars to make inferences on the collected data and visualize them in a friendly and easy way.

In the Digital Humanities literature, there are ontologies focusing on different aspects of textual information. Each of these ontologies represents a set of possible interpretations of the source text(s). Up to now, an ontology for representing knowledge about primary sources of literary texts has not been developed. In the previous stages of the project, we developed an ontology [7] using the RDF language [13] to represent Dante’s works and the knowledge about their primary sources. Then, we populated the ontology with the knowledge extracted from some authoritative commentaries, obtaining an RDF knowledge base. On top of this knowledge base, we developed DanteSourcesFootnote 1, a focused Digital Library (DL) endowed with web services that allow visualizing information on Dante Alighieri’s primary sources in form of charts and tables. In this paper we present the results of a survey, developed using Google Docs, to collect suggestions and comments from end-users on their interactions with DanteSources in order to evaluate its usability.

The paper is structured as follows: Sect. 2 reports a review of related works; in Sect. 3 the functionalities of DanteSources are described. Section 4 presents the results of the evaluation and a discussion of them. Finally, in Sect. 5 we report our conclusive remarks.

2 Related Works

In the last thirty years several major Digital Humanities projects about Dante Alighieri and his works were published online. Most of them focused on offering digital editions of Dante’s works with commentaries, textual search, and multimedia.

The Dartmouth Dante Project (DDP)Footnote 2 was launched in 1985 with the aim of publishing the full searchable text of the Divine Comedy and several of its commentaries in digital format [10]. On top of the DDP in 2013 the web application Dante LabFootnote 3 was developed. This application allows the concurrent visualization of the original text of the Comedy, some translations i.e. English, French and German, and more than 75 commentaries, each of them fully searchable. The Princeton Dante ProjectFootnote 4 includes the full searchable text of the Divine Comedy, Dante’s minor works, several commentaries and multimedia resources [11]. The World of DanteFootnote 5 is a multi-media research tool that contains the whole text of the Divine Comedy with a basic semantic annotation that identifies people, places, deities and structures, and connections to digital objects such as images and music. However, the semantic knowledge is not represented through a formal ontology and is not available in a machine-readable format. An interactive timeline of Dante’s life and relevant maps are also available [14]. Digital DanteFootnote 6 offers the full text of all Dante’s works with commentaries, illustrations and recorded readings. It also features a subproject called Intertextual DanteFootnote 7, the first digital attempt at connecting passages of Dante’s works with the corresponding fragments of cited primary sources. However, the project is limited to the works of the Roman poet Ovid [15]. Dante OnlineFootnote 8 includes the full text of all Dante’s works, a biography with an interactive timeline, and a database listing hundreds of manuscripts of the Divine Comedy with bibliographic information. DanteworldsFootnote 9 is a multimedia project built around the Divine Comedy, featuring illustrations by historical and contemporary artists, audio lectures and study resources. More recently, two major Italian research projects were published: DanteSearchFootnote 10 and Dante Medieval Archive (DaMA)Footnote 11. DanteSearch is a complete lemmatization, grammatical and syntactic annotation of Dante’s works allowing users to perform morphological and syntactic queries on the full text of the author’s works. On the other hand, DaMA is a digital archive containing the full text of Dante’s works, commentaries and several primary sources in XML-TEI format. With the advent of the Semantic Web, several projects were started with a focus on the application of the new semantic technologies (RDF, OWLFootnote 12) to the Humanities. For instance, González-Blanco et al. [9] describe a semantic model to connect repertoires of poetic writings. Vitale [16] presents an ontology for the 3D visualization of cultural heritage; Lana et al. [12] describe an ontology for annotating geographical places in texts. In this context, we developed DanteSources DL, which uses the technologies of the Semantic Web to represent the knowledge included in the works of Dante Alighieri, focusing on primary sources.

3 The Digital Library DanteSources

In the previous stages of the project “Towards a Digital Dante Encyclopaedia”, we developed an ontology [7] expressed in RDF language [13] to represent Dante’s works and the knowledge about their primary sources. We populated the ontology with the knowledge extracted from the following authoritative commentaries: Vita Nova [3], Vita Nuova [1], De vulgari eloquentia [2], Convivio [5], and Monarchia [6]. The obtained RDF graph was eventually stored in a Virtuoso triple store [8]. The focused DanteSources DL allows extracting and displaying the information stored in the knowledge base to support scholars in discovering and exploring Dante’s primary sources. DanteSources is a web application developed in Java using Javascript and Ajax functions. The DL extracts knowledge by running SPARQL queries on the RDF knowledge base. DanteSources shows the knowledge about the primary sources cited by Dante in the form of tables and column bar charts. In particular, we used the HighchartsFootnote 13 JavaScript library to implement these charts. Highcharts allows exporting the charts in various well-known formats: PDF, PNG, JPEG, SVG. Furthermore, we implemented an additional JavaScript function allowing users to automatically export and download all the data in CSV formatFootnote 14. This feature is particularly important since it allows scholars to obtain and manage raw data, in order to apply further data analyses in addition to the ones already provided by the DL. Currently, eight different predefined SPARQL queries are available to extract data. They can be distinguished into three different groups. The first group includes three queries. In order to make these queries, a search form allows users to choose either one Dante’s work or all his works and, in addition, a specific subpart of the work (e.g. a book). The queries produce column bar charts regarding the distribution of the works, the authors and the thematic areas cited by Dante. For example, Fig. 1 shows the chart of the distribution of some primary sources cited in the first book of Convivio.

Fig. 1.
figure 1

The chart represents the distribution of some primary sources cited in the first book of Convivio

Full size image

Additional information about primary sources, authors and thematic areas are available by clicking on their names. In particular, clicking on the title of a primary source, the DL shows a table reporting information about: (i) the book, (ii) the chapter, (ii) the paragraph or poem and (iv) the fragment of the Dante’s work in which the primary source is cited, (v) the type of reference, (vii) the reference to a fragment of the primary source cited in the commentary, (viii) the thematic area and (ix) the author of the primary source. An example of this table is reported in Fig. 2.

Fig. 2.
figure 2

Part of the table regarding Ethica Nicomachea cited by Dante in the first book of Convivio

Full size image

Similarly, when a user clicks on the name of a cited author, the DL shows a table reporting all the works of that author cited in the Dante’s text chosen by the user. Furthermore, for each primary source the DL reports the book, the chapter and the paragraph (or poem) of the Dante’s text in which the author is cited. Figure 3 shows an example of table related to a cited author.

Fig. 3.
figure 3

Part of the table regarding Aristotle cited in Monarchia

Full size image

Finally, for what concerns the thematic areas, clicking on one name the user obtains the following information for each thematic area: the primary sources included in that area, their authors and the book, chapter and paragraph (or poem) of the Dante’s work where the thematic area is cited.

The three queries of the second group allow visualizing several charts that report the distribution of a particular primary source, a cited author or a thematic area respectively. In order to improve the usability of the DL, we implemented two different search forms for these queries:

  1. (1)

    An autocomplete menu where the user can type the title of the cited work, the author, or the thematic area;

    Fig. 4.
    figure 4

    The distribution of Aristotle’s Metaphysics in all Dante’s works (a), in the four books of Convivio (b) and in the chapters of the second book of Convivio (c).

    Full size image
  2. (2)

    An alphabetically ordered list in which the user can select the title of the primary source, the author, or the thematic area.

The data regarding the distributions are not only available for an entire Dante’s work, like Convivio, but also for its subparts like books, chapters or poems. Indeed, by clicking on one bar in the chart representing the distribution of the information onto a Dante’s work, it is possible to visualize information about its subparts. For example, it is possible to visualize the distribution of the selected information on all the books of Convivio and also its distribution on each chapter of the second book of Convivio. Figure 4 shows three views of the distribution of a primary source on three different levels: all Dante’s works (a), one particular Dante’s work (b), a subpart of a Dante’s work (c).

The last group includes two queries that allow visualizing the distribution of the three types of reference to primary sources: explicit, strict and generic, focusing either on Dante’s works or on a single primary source. Indeed, with the support of Dante’s experts we defined three types of reference:

  1. 1.

    explicit, if the reference is explicitly made by Dante, e.g. “As the Philosopher says at the beginning of the First Philosophy”, where the Philosopher is Aristotle and the First Philosophy is Metaphysics;

  2. 2.

    strict, if the reference is indicated by a scholar and refers to a specific work, e.g. “SI MANUCA: it is the bread of the angels, the manna as called in the Old Testament (Ps. 77, 25 Panem angelorum manducavit homo)”;

  3. 3.

    generic, if the reference is indicated by a scholar, and refers to a concept (e.g. Medieval comments to Aristotle’s works).

Figure 5 shows the distribution of the three types of reference in the works of Dante included in our knowledge base.

Fig. 5.
figure 5

The distribution of the three types of reference in Dante’s works

Full size image

As for the second group of queries, by clicking on the bars of the chart, the data regarding the distribution of the types of reference are available both for an entire Dante’s work and for its subparts, i.e. books and chapters.

4 Evaluation

4.1 The Online Survey

The survey included 28 questions about the main aspects regarding the usability of the DL. After a brief overview to identify the sample, we focused our attention on the features of the DL: the navigation usability, the readability of the charts and tables, the usability of the search functionalities, the satisfaction of the results obtained doing a query. A specific attention was put on the effective responsiveness of the layout and on the use of the DL on different devices, e.g. PCs, smartphones, tablets. Questions were presented through multiple choice and text area. The same questions required a personal judgment on selected features. A Likert 5-scale values from 1 (totally negative) to 5 (totally positive) was used to express the opinion.

4.2 Participants

We collected suggestions and comments from 26 users; some of them did not answer all the questions. Our sample included 18 females and 10 males. 82 % of the participants were in the range of 31–50 years old, 14 % were people between 10 and 30 years old and 3 % were in the range of 51–65 years old.

25 % of the users were students, 20 % were researchers, 12 % were university professors, 4 % were school teachers and the 37 % of the users did not specify their profession. Concerning technological skills, 51 % declared to regularly use smartphones with android system, 18 % an iPhone or iPod touch, 11 % an iPad, 11 % a tablet with Android system, 7 % of the users responded that they never use mobile devices. 100 % of the users said that they use the Internet at least once per day to search information about their interest fields.

4.3 Results

We asked the users to freely use and explore DanteSources. Afterwards, we asked an opinion about the general interaction with the interface: 63 % of the users expressed a totally positive opinion (score 5), 22.2 % assigned score 4, 7.4 % score 3 and the remaining 7.4 % score 2. No one expressed a totally negative opinion (score 1). Figure 6 shows a graphical representation of the results.

Fig. 6.
figure 6

Opinions of the users on the interaction with the interface

Full size image

Regarding the navigation, 55.6 % of the users said that the navigation is simple and clear and the user always knowing where s/he currently and where to go afterwards, 25.9 % assigned score 4, 14.8 % score 3, and only one user score 2. No one expressed a totally negative opinion (score 1). A graphical representation of the results is reported in Fig. 7.

Fig. 7.
figure 7

Readability of the charts

Full size image

As far as the readability of the obtained charts is concerned, 51.9 % of the users assigned score 5, 25.9 % score 4, 14.8 % score 3, and the remaining 7,4 % score 2. No one chose score 1. In particular, the users expressed a positive opinion on the possibility to export in CSV format all the data visualized into the charts and on the possibility of visualizing ten results at a time using the scrollbar. On the other hand, some comments reported the continuous use of the scrollbar as a limit, when the results are many.

Then, we asked to evaluate the tables which the user can access from each chart and to leave a textual comment to describe the main observed problems. Regarding the charts about primary sources, several users reported the usefulness and the completeness of the knowledge presented in the tables. Some users highlighted the lack of text visualization for the cited fragment within the text of the primary source. One user said that it is not intuitive that clicking on the title of the primary sources in the chart, it is possible to visualize a table. This possibility is described in natural language below the chart but this user suggested that it could be more useful having it above. As to the tables regarding the cited authors, the majority of the comments reported that the only missing information is the textual fragment of the primary sources.

For what concerns the tables about the thematic areas, many comments reported the lack of a textual explanation on how the thematic areas were defined, as well as the absence of the textual fragment of the primary sources.

For the second group of queries, we asked about the usefulness of the two different search menus, the autocomplete menu and the alphabetically ordered list. 80.8 % expressed a totally positive opinion, assigning score 5, and the remaining 19.2 % assigned score 4. The users reported the usefulness of these search functionalities especially to avoid the failure of the search and the absence of feedback (Fig. 8).

Fig. 8.
figure 8

Usefulness of the two different search menus

Full size image

Regarding the position of the information on the screen, 25.9 % of the users assigned score 5, 48.1 % score 4, 22.2 % score 3, 3.7 % score 2. No one expressed a totally negative opinion (score 1). The main reported problem regards the presence of too much blank space between the search forms and the results, which forces the users to vertically scroll the page. Furthermore, some comments reported that the explanations about the functionalities of the charts should have a more relevant position on the page.

Only 18.5 % of the users accessed DanteSources from a mobile device. The used mobile devices were: smartphone with Android system, iPhone, iPad. 57.1 % of this 18.5 % declared that interaction on a mobile device was very easy and clear (score 5), the remaining 42.9 % assigned score 4. No problems were reported in the comments.

76.9 % of the users said that the response times of the DL were very fast and they obtained an instantaneous response (value 5). 15.4 % assigned score 4 and the remaining 7.7 % score 3. No one chose the more negative scores 2 and 1 (Fig. 9).

Fig. 9.
figure 9

Ease of response of DanteSources

Full size image

Regarding the data obtained through the search functionalities, 61.5 % of the users said that they obtained all the information they needed, while 34.6 % reported that they obtained a lot of information, although they would like to receive additional data. Only one user said that s/he did not retrieve the information s/he needed. This last user specified that s/he is a linguist and for this reason s/he would need to other types of information, like Named Entities, parsing of the texts and collocations. Unfortunately, our project was focused not on the syntax but on the semantics of the primary sources. Many users expressed the desire to visualize the text of the fragments of the primary sources as well as, on request, the entire text of the primary sources and of the Dante’s works. Other comments suggested visualizing the entire text of the commentaries. Unfortunately, they are protected by copyright, so it is not possible to show them.

We asked the user about the major problems encountered using DanteSources. Several users reported that they did not have significant problems, few users suggested making the information more compact on the screen and one user said that the interactive image on the homepage is not intuitive to use.

Finally, the users left suggestions to improve the DL. The three main suggestions regarded (i) the possibility of visualizing the texts of the cited fragments, as well as the entire texts of the primary sources and of the Dante’s works; (ii) the introduction of the possibility to directly visualize the information on the primary sources written by a specific cited author; (iii) the enrichment of the knowledge base with other Dante’s works, especially with the Divine Comedy.

About the usefulness of DanteSources, 65.5 % of the users declared that the DL is very useful (score 5), 26.9 % assigned score 4, 3.8 % score 3 and the remaining 3.8 % score 2 (Fig. 10).

Fig. 10.
figure 10

General usefulness of DanteSources

Full size image

4.4 Discussion

Regarding the answers and the opinions obtained from the users through the survey, we observe that the interaction with the application is generally satisfying. Several users said that the navigation is clear and easy and the results of the queries are complete and well presented. Also the response times of the DL are satisfactory. At the same time, the users made suggestions to improve the interface usability and the user experience. In brief, the suggestions are related to:

  1. a.

    reducing blank spaces in the layout, especially between the search forms and the results and moving the explanation of additional features of the charts in a more evident position;

  2. b.

    visualizing the fragments of the primary sources and, upon request, the whole texts of the primary sources and of the Dante’s works;

  3. c.

    visualizing the data shown in the charts also in the form of tables or lists;

  4. d.

    making available the corresponding works when the users search for a specific author;

  5. e.

    enriching the knowledge base with other Dante’s works.

Regarding the (a) suggestion, we have planned to reduce the blank spaces and modify the position of the explanation of the charts’ additional features. About the (b), we have already developed a function that for each Dante’s fragment allows visualizing the fragment of the corresponding primary source. Up to now we implemented this function for the primary sources included in DaMA, since this archive contains several primary sources coded in XML-TEI format. Regarding the suggestion to also visualize the entire texts of Dante’s works, these are already preset in a dedicated page of the DL as external links to DaMA. However, the users didn’t find them easily. For this reason, we have planned to add this links also in the pages of the search results. Concerning the suggestion (c), the users did not consider the possibility to export the data in CSV format in order to have the results in form of simple table. We supposed that the CSV could be a format not very well known and used in the Humanities, so we have thought to replace the current label “Export in CSV” into a more clear label like “Export as a table (in CSV format)”. About request (d), when a user searches for a specific cited author, we have decided to add a feature that shows a table of that author’s works. Regarding the suggestion (e), at this moment we are working to add Rime [4] in our knowledge base.

5 Conclusions

In this paper we have presented a survey conducted on the interaction with the DanteSources Digital Library via PCs and mobile devices to collect suggestions and comments on possible usability issues observed by the users.

We have had a total of 26 users. Positive responses on the general interaction have emerged from the survey: more than 60 % of the users have expressed a distinctly positive response. About 80 % have expressed a satisfactory opinion on the navigation (score 4 and 5) and on the readability of charts and tables (score 4 and 5) as well. 100 % of the users who accessed DanteSources from a mobile device have declared that interaction was easy and clear (score 4 and 5). On the other hand, the users have identified some problems and lacks in the interface, in particular: the presence of too much blank space between the search forms and the results, the hidden position of the explanation of the charts’ functionalities, the lack of text visualization for the cited fragments and for the entire texts of the primary sources and Dante’s works. Furthermore, the users have suggested adding a functionality that allows visualizing the list of works when the users search for a specific author. The users have also suggested enriching the knowledge base with other Dante’s works, especially with Divine Comedy. We have planned to modify and enrich DanteSources taking into account the problems highlighted by the users and the given suggestions.

The Dante’s experts who have taken part in our evaluation have said that having the information about primary sources available, for the first time, in digital format, improves and makes more efficient their searches. Indeed, in their commentaries, scholars typically express knowledge on the primary sources in natural language. This limits scholars in their advances insofar as it prevents automatic inferences of new information that may be useful for their studies. For instance, in the case of references to primary sources, the inferences may concern the total amount of references to a certain work or to an author. Eventually, the digitization of the knowledge about primary sources allows experts to have a comprehensive overview of the data, to visualize it as charts and tables, and to export it to make their own analyses. For this reason, DanteSources allows scholars to explore the dynamics of the multi-faceted culture of Dante in relation to the diverse stages of his biography and to study the evolution in time of Dante’s cultural background.