Abstract
In this paper we present Jarvis, a multimodal explorer and navigation system for biocuration data, from both curated sources and text-derived datasets. This system harnesses voice and haptic control for a bioinformatic research context, specifically manipulation of data visualizations such as heatmaps and word clouds showing related terms in the dataset. We combine external speech systems with Clustergrammer [1] for the generation of bioinformatic queries, the BoB interface [2] for answering queries in that domain, and the VoxML framework [12] for manipulating the results and semantic grounding. We deploy the resulting system to iOS on an iPad for use by researchers over a test dataset of gene expression in tumor samples. The intent is to integrate multimodal control (here voice and haptics), so as to facilitate interaction with and analysis of data, taking advantages of using both modalities.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
“Voxphrases” in Fig. 1 are voxemes representing words as manipulable 3D objects. These are always billboarded to display facing the camera.
References
Fernandez, N.F., et al.: Clustergrammer, a web-based heatmap visualization and analysis tool for high-dimensional biological data (2017). https://doi.org/10.1038/sdata.2017.151
Burstein, M., et al.: Using multiple contexts to interpret collaborative task dialogs. In: Advanced in Cognitive Systems (2019)
Clark, N.R., Ma’ayan, A.: Introduction to statistical methods for analyzing large data sets: gene-set enrichment analysis. Sci. Signal. 4(190), p. tr4 (2011). ISSN 1945–0877. https://doi.org/10.1126/scisignal.2001966. https://stke.sciencemag.org/content/4/190/tr4.full.pdf. https://stke.sciencemag.org/content/4/190/tr4
Friedman, S., et al.: Learning by reading: extending and localizing against a model. Adv. Cogn. Syst. 5, 77–96 (2017)
Goldstone, W.: Unity Game Development Essentials. Packt Publishing Ltd., Birmingham (2009)
Gyori, B.M., et al.: From word models to executable models of signaling networks using automated assembly. Mol. Syst. Biol. 13(11), 954 (2017)
Jehl, P., et al.: ProViz–a web-based visualization tool to investigate the functional and evolutionary features of protein sequences. Nucl. Acids Res. 44(W1), W11–W15 (2016). ISSN 0305-1048. https://doi.org/10.1093/nar/gkw265. https://academic.oup.com/nar/article-pdf/44/W1/W11/18787253/gkw265.pdf
Johnston, M.: Building multimodal applications with EMMA. In: Proceedings of the 2009 International Conference on Multimodal Interfaces, pp. 47–54 (2009)
Johnston, M.: Multimodal integration for interactive conversational systems. In: The Handbook of Multimodal-Multisensor Interfaces: Language Processing, Software, Commercialization, and Emerging Directions, vol. 3, pp. 21–76 (2019)
Kim, J.-D.: Biomedical Natural Language Processing (2017)
Krishnaswamy, N., Pustejovsky, J.: An evaluation framework for multimodal interaction. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) (2018)
Krishnaswamy, N., Pustejovsky, J.: VoxSim: a visual platform for modeling motion language. In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations. Osaka, Japan: The COLING 2016 Organizing Committee, December 2016, pp. 54–58. https://www.aclweb.org/anthology/C16-2012
Lee, J., et al.: BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36(4), 1234–1240 (2020)
McDonald, D., et al.: Extending biology models with deep NLP over scientific articles. In: Workshops at the Thirtieth AAAI Conference on Artificial Intelligence (2016)
McInnes, L., Healy, J.: UMAP: uniform manifold approximation and projection for dimension reduction. ArXiv abs/1802.03426 (2018)
O’Halloran, K.L., et al.: A digital mixed methods research design: integrating multimodal analysis with data mining and information visualization for big data analytics. J. Mixed Methods Res. 12(1), 11–30 (2018)
Papineni, K., et al.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, pp. 311–318 (2002)
Pustejovsky, J., Krishnaswamy, N.: VoxML: a visualization modeling language. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pp. 4606–4613 (2016)
Schonlau, M.: The clustergram: a graph for visualizing hierarchical and nonhierarchical cluster analyses. Stata J. 2(4), 391–402 (2002)
Selfridge, E., Johnston, M.: Interact: tightly-coupling multimodal dialog with an interactive virtual assistant. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp. 381–382 (2015)
Shannon, P., et al.: Cytoscape: a software environment for integrated models of biomolecular interaction networks. Genome Res. 13(11), 2498–2504 (2003). https://doi.org/10.1101/gr.1239303
Tao, Y., et al.: Information visualization techniques in bioinformatics during the postgenomic era. Drug Discov. Today BIOSILICO 2(6), 237–245 (2004)
Todorov, P.V., et al.: INDRA-IPM: interactive pathway modeling using natural language with automated assembly. Bioinformatics 35(21), 4501–4503 (2019)
Yan, Z., et al.: Building task-oriented dialogue systems for online shopping. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)
Acknowledgements
This work is supported in part by US Defense Advanced Research Projects Agency (DARPA), Contract W911NF-15-C-0238; and DTRA grant DTRA-16-1-0002; Approved for Public Release, Distribution Unlimited. The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government. We would like to thank everyone at the Boston office of Smart Information Flow Technologies, particularly Laurel Bobrow, Robert Bobrow, Mark Burstein, David McDonald, and Matthew McLure; and Benjamin Gyori and John Bachman at Harvard Medical School. All remaining errors are, of course, those of the authors alone.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Hutchens, M., Krishnaswamy, N., Cochran, B., Pustejovsky, J. (2020). Jarvis: A Multimodal Visualization Tool for Bioinformatic Data. In: Stephanidis, C., et al. HCI International 2020 – Late Breaking Papers: Interaction, Knowledge and Social Media. HCII 2020. Lecture Notes in Computer Science(), vol 12427. Springer, Cham. https://doi.org/10.1007/978-3-030-60152-2_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-60152-2_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-60151-5
Online ISBN: 978-3-030-60152-2
eBook Packages: Computer ScienceComputer Science (R0)