THC-DAT helps in reading a multi-topic document: Results from a user-centered evaluation of a within-document analysis tool

With a mass of electronic multi-topic documents available, there is an increasing need for evaluating emerging analysis tools to help users and digital libraries analyze these documents better. The purpose of this paper is to evaluate the effectiveness, efficiency and user satisfaction of THC-DAT, a within-document analysis tool, in reading a multi-topic document.

The authors reviewed related literature first, then performed a user-centered, comparative evaluation of two within-document analysis tools, THC-DAT and BOOKMARK. THC-DAT extracts a topic hierarchy tree using hierarchical latent Dirichlet allocation (hLDA) method and takes the context information into account. BOOKMARK provides similar functionality to the Table of Contents bookmarks in Adobe Reader. Three novel kinds of tasks were devised for participants to finish on two tools, with objective results to assess reading effectiveness and efficiency. And post-system questionnaires were employed to obtain participants’ subjective judgments about the tools.

The results confirm that THC-DAT is significantly more effective than BOOKMARK, while not inferior in efficiency. There is some evidence that suggests THC-DAT can slow down the process of approaching cognitive overload and improve users’ willingness to undertake difficult task. Based on qualitative data from questionnaires, the results indicate that users were more satisfied when using THC-DAT than BOOKMARK.

Adopting THC-DAT in digital libraries or electrical document reading systems contributes to promoting users’ reading performance, willingness to undertake difficult task and general satisfaction. Moreover, THC-DAT is of great value to addressing cognitive overload problem in the information retrieval field.

This paper evaluates a novel within-document analysis tool in analyzing a multi-topic document, and proved that this tool is superior to the benchmark in effectiveness and user satisfaction, and not inferior in efficiency.