Abstract
Information Extraction (IE) research has made remarkable progress in Natural Language Processing using intrinsic measures, but little attention has been paid to human analysts as downstream processors. In one experiment, when participants were presented text with or without markup from an IE pipeline, they showed better text comprehension without markup. In a second experiment, the markup was hand-generated to be as relevant and accurate as possible to find conditions under which markup improves performance. This experiment showed no significant difference between performance with and without markup, but a significant majority of participants preferred working with markup to without. Further, preference for markup showed a fairly strong correlation with participants’ ratings of their own trust in automation. These results emphasize the importance of testing IE systems with actual users and the importance of trust in automation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Cunningham, H.: Information extraction, automatic. In: Encyclopedia of Language and Linguistics, 2nd edn., pp. 665–677. Elsevier, New York (2005)
Marrero, M., Urbano, J., Sánchez-Cuadrado, S., Morato, J., GĂłmez-BerbĂs, J.M.: Named entity recognition: fallacies, challenges and opportunities. Comput. Stan. Interfaces 35, 482–489 (2013)
Zaroukian, E.: Information extraction for optimized human understanding and decision making. In: Proceedings of ICCRTS (2018)
Ruddy, M.: ELICIT – the experimental laboratory for the investigation of collaboration, information sharing, and trust. In: Proceedings of the 12th ICCRTS (2007)
Li, Q., Ji, H.: Incremental joint extraction of entity mentions and relations. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pp. 402–412. ACL, New York (2014)
Li, Q., Ji, H., Huang, L.: Joint event extraction via structured prediction with global features. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pp. 73–82. ACL, New York (2013)
Krausman, A.: Understanding audio communication delay in distributed team interaction: impact on trust, shared understanding, and workload. In: Proceedings of the IEEE CogSIMA Conference, pp. 1–3. IEEExplore (2017)
NASA: NASA Task Load Index (TLX), v. 1.0 Manual (1986)
Williams, T.R.: Guidelines for designing and evaluating the display of information on the Web. Tech. Commu. 47(3), 383–396 (2000)
Jian, J.Y., Bisantz, A.M., Drury, C.G.: Foundations for an empirically determined scale of trust in automated systems. Int. J. Cogn. Ergon. 4(1), 53–71 (2000)
Acknowledgments
Many thanks to Stephen Tratz, Claire Bonial, Jeffrey Micher, Clare Voss, Jon Bakdash, Lucia Donatelli, and Jeff Hoye for their assistance in designing, deploying, and interpreting this work. This research was supported in part by an appointment to the Student Research Participation Program at the Army Research Laboratory administered by the Oak Ridge Institute for Science and Education through an interagency agreement between U.S. Department of Energy and ARL.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 This is a U.S. government work and not under copyright protection in the U.S.; foreign copyright protection may apply
About this paper
Cite this paper
Zaroukian, E., Caylor, J., Vanni, M., Kase, S. (2020). Human Interaction with the Output of Information Extraction Systems. In: Ahram, T. (eds) Advances in Artificial Intelligence, Software and Systems Engineering. AHFE 2019. Advances in Intelligent Systems and Computing, vol 965. Springer, Cham. https://doi.org/10.1007/978-3-030-20454-9_46
Download citation
DOI: https://doi.org/10.1007/978-3-030-20454-9_46
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-20453-2
Online ISBN: 978-3-030-20454-9
eBook Packages: EngineeringEngineering (R0)