Skip to main content
Springer Nature Link
Log in

Explainable interactive projections of images

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Dimension reductions (DR) help people make sense of image collections by organizing images in the 2D space based on similarities. However, they provide little support for explaining why images were placed together or apart in the 2D space. Additionally, they do not provide support for modifying and updating the 2D representation to explore new relationships and organizations of images. To address these problems, we present an interactive DR method for images that uses visual features extracted by a deep neural network to project the images into 2D space and provides visual explanations of image features that contributed to the 2D location. In addition, it allows people to directly manipulate the 2D projection space to define alternative relationships and explore subsequent projections of the images. With an iterative cycle of semantic interaction and explainable-AI feedback, people can explore complex visual relationships in image data. Our approach to human–AI interaction integrates visual knowledge from both human-mental models and pre-trained deep neural models to explore image data. We demonstrate our method through examples with collaborators in agricultural science and other applications. Additionally, we present a quantitative evaluation that assesses how well our method captures and incorporates feedback.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. The implementation of our method can be found at https://github.com/infovis-vt/Andromeda_IMG.

References

  1. Cunningham, P.: Dimension reduction. In: Machine Learning Techniques for Multimedia, pp 91–112. Springer (2008)

  2. Endert, A., Chang, R., North, C., Zhou, M.: Semantic interaction: coupling cognition and computation through usable interactive analytics. IEEE Comput. Gr. Appl. 35(4), 94–99 (2015)

    Article  Google Scholar 

  3. Self, J.Z., Dowling, M., Wenskovitch, J., Crandell, I., Wang, M., House, L., et al.: Observation-level and parametric interaction for high-dimensional data analysis. ACM Trans. Interact. Intell. Syst. 8(2), 1–36 (2018)

    Article  Google Scholar 

  4. Cheng, T.Y., Huertas-Company, M., Conselice, C.J., Aragon-Salamanca, A., Robertson, B.E., Ramachandra, N.: Beyond the Hubble sequence-exploring galaxy morphology with unsupervised machine learning. Mon. Not. R. Astron. Soc. 503(3), 4446–4465 (2021)

    Article  Google Scholar 

  5. Bojarski, M., Choromanska, A., Choromanski, K., Firner, B., Jackel, L., Muller, U., et al.: Visualbackprop: efficient visualization of CNNs. Preprint at arXiv:1611.05418 (2016)

  6. Han, H., Faust, R., Norambuena, B.F.K., Prabhu, R., Smith, T., Li, S., et al.: Explainable interactive projections for image data. In: Advances in Visual Computing: 17th International Symposium, ISVC 2022, San Diego, CA, USA, October 3–5, 2022, Proceedings, Part I., pp. 77–90. Springer (2022)

  7. Tukey, J.W., Wilk, M.B.: Data analysis and statistics: an expository overview. In: Proceedings of the November 7-10, 1966, Fall Joint Computer Conference, pp. 695–709 (1966)

  8. Cavallo, M., Demiralp, Ç.: A visual interaction framework for dimensionality reduction based data exploration. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–13 (2018)

  9. Jeong, D.H., Ziemkiewicz, C., Fisher, B., Ribarsky, W., Chang, R.: iPCA: an interactive system for PCA-based visual analytics. In: Computer Graphics Forum, vol. 28, pp. 767–774. Wiley Online Library (2009)

  10. Espadoto, M., Appleby, G., Suh, A., Cashman, D., Li, M., Scheidegger, C.E., et al.: UnProjection: leveraging inverse-projections for visual analytics of high-dimensional data. IEEE Transactions on Visualization and Computer Graphics (2021)

  11. dos Santos Amorim, E.P., Brazil, E.V., Daniels, J., Joia, P., Nonato, L.G., Sousa, M.C.: iLAMP: exploring high-dimensional spacing through backward multidimensional projection. In: 2012 IEEE Conference on Visual Analytics Science and Technology, pp. 53–62. IEEE (2012)

  12. Eler, D.M., Nakazaki, M.Y., Paulovich, F.V., Santos, D.P., Andery, G.F., Oliveira, M.C.F., et al.: Visual analysis of image collections. Vis. Comput. 25(10), 923–937 (2009)

    Article  Google Scholar 

  13. Paulovich, F.V., Eler, D.M., Poco, J., Botha, C.P., Minghim, R., Nonato, L.G.: Piece wise Laplacian-based projection for interactive data exploration and organization. In: Computer Graphics Forum, vol. 30, pp. 1091–1100, Wiley Online Library (2011)

  14. Joia, P., Coimbra, D., Cuminato, J.A., Paulovich, F.V., Nonato, L.G.: Local affine multidimensional projection. IEEE Trans. Vis. Comput. Gr. 17(12), 2563–2571 (2011)

    Article  Google Scholar 

  15. Mamani, G.M., Fatore, F.M., Nonato, L.G., Paulovich, F.V.: User-driven feature space transformation. In: Computer Graphics Forum, vol. 32. pp. 291–299. Wiley Online Library (2013)

  16. Brown, E.T., Liu, J., Brodley, C.E., Chang, R.: Dis-function: learning distance functions interactively. In: IEEE Conference on Visual Analytics Science and Technology. vol. 2012, pp. 83–92. IEEE (2012)

  17. Fujiwara, T., Wei, X., Zhao, J., Ma, K.L.: Interactive dimensionality reduction for comparative analysis. IEEE Trans. Vis. Comput. Gr. 28(1), 758–768 (2022). https://doi.org/10.1109/TVCG.2021.3114807

    Article  Google Scholar 

  18. Endert, A., Fiaux, P., North, C.: Semantic interaction for visual text analytics. In: Proc. of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’12. pp. 473-482. ACM, New York. Available from: https://doi.org/10.1145/2207676.2207741 (2012)

  19. Endert, A., Fiaux, P., North, C.: Semantic interaction for sensemaking: inferring analytical reasoning for model steering. IEEE Trans. Vis. Comput. Gr. 18(12), 2879–2888 (2012)

    Article  Google Scholar 

  20. Dowling, M., Wycoff, N., Mayer, B., Wenskovitch, J., House, L., Polys, N., et al.: Interactive visual analytics for sensemaking with big text. Big Data Res. 16, 49–58 (2019)

    Article  Google Scholar 

  21. Dowling, M., Wenskovitch, J., Hauck, P., Binford, A., Polys, N., North, C.: A bidirectional pipeline for semantic interaction. In: Proc. Workshop on Machine Learning from User Interaction for Visualization and Analytics (at IEEE VIS 2018), vol. 11, p. 74 (2018)

  22. Wang, M., Wenskovitch, J., House, L., Polys, N., North, C.: Bridging cognitive gaps between user and model in interactive dimension reduction. Vis. Inform. 5(2), 13–25 (2021)

    Article  Google Scholar 

  23. Endert, A., Han, C., Maiti, D., House, L., North, C.: Observation-level interaction with statistical models for visual analytics. In: IEEE Conference on Visual Analytics Science and Technology, vol. 2011, pp. 121–130. IEEE (2011)

  24. House, L., Leman, S., Han, C.: Bayesian visual analytics: Bava. Stat. Anal. Data Min. ASA Data Sci. J. 8(1), 1–13 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  25. Leman, S.C., House, L., Maiti, D., Endert, A., North, C.: Visual to parametric interaction (v2pi). PloS One 8(3), e50474 (2013)

    Article  Google Scholar 

  26. Krokos, E., Cheng, H.C., Chang, J., Nebesh, B., Paul, C.L., Whitley, K., et al.: Enhancing deep learning with visual interactions. ACM Trans. Interact. Intell. Syst. (TiiS) 9(1), 1–27 (2019)

    Article  Google Scholar 

  27. Bian, Y., North, C.: Deepsi: interactive deep learning for semantic interaction. In: 26th International Conference on Intelligent User Interfaces, pp. 197–207 (2021)

  28. Bian, Y., North, C., Krokos, E., Joseph, S.: Semantic, explanation of interactive dimensionality reduction. In: IEEE Visualization Conference (VIS), vol. 2021, pp. 26–30. IEEE (2021)

  29. Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: European Conference on Computer Vision, pp. 818–833. Springer (2014)

  30. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2929 (2016)

  31. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)

  32. Pirolli, P., Card, S.: The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In: Proceedings of International Conference on Intelligence Analysis, vol. 5, pp. 2–4. McLean, VA (2005)

  33. Salau, A.O., Jain, S.: Feature extraction: a survey of the types, techniques, applications. In: 2019 International Conference on Signal Processing and Communication (ICSC), pp. 158–164. IEEE (2019)

  34. Chen, J., Zou, L.H., Zhang, J., Dou, L.H.: The comparison and application of corner detection algorithms. J. Multimed. 4(6) (2009)

  35. Lowe, D.G.: Object recognition from local scale-invariant features. In: Proceedings of the Seventh IEEE International Conference on Computer Vision. vol. 2, pp. 1150–1157. IEEE (1999)

  36. Ghosh, S.K., Biswas, B., Ghosh, A.: A novel noise removal technique influenced by deep convolutional autoencoders on mammograms. In: Deep Learning in Data Analytics. pp. 25–43. Springer (2022)

  37. Yu, S., Jia, S., Xu, C.: Convolutional neural networks for hyperspectral image classification. Neurocomputing 219, 88–98 (2017)

    Article  Google Scholar 

  38. Villaret, M., et al.: Affective state-based framework for e-Learning systems. In: Artificial Intelligence Research and Development: Proceedings of the 23rd International Conference of the Catalan Association for Artificial Intelligence. vol. 339, p. 357. IOS Press (2021)

  39. Baumgartl, H., Buettner, R.: Developing efficient transfer learning strategies for robust scene recognition in mobile robotics using pre-trained convolutional neural networks. Preprint at arXiv:2107.11187 (2021)

  40. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  41. Han, H., Prabhu, R., Smith, T., Dhakal, K., Wei, X., Li, S., et al.: Interactive deep learning for exploratory sorting of plant images by visual phenotypes (2022)

  42. Banerjee, S.: Animal image dataset. https://www.kaggle.com/datasets/iamsouravbanerjee/animal-image-dataset-90-different-animals?select=animals

  43. Rousseeuw, P.J.: Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20, 53–65 (1987)

    Article  MATH  Google Scholar 

  44. Wenskovitch, J., North, C.: Interactive AI: Designing for the ‘Two Black Boxes’ Problem, pp. 1–10. IEEE Computer Society, Hybrid Human-Artificial Intelligence Special Issue, Washington (2020)

  45. Bian, Y., Wenskovitch, J., North, C.: Deepva: bridging cognition and computation through semantic interaction and deep learning. Preprint at arXiv:2007.15800 (2020)

Download references

Acknowledgements

This material is based upon work supported by the National Science Foundation under Grant # 2127309 to the Computing Research Association for the CIFellows 2021 Project. This project was funded, in part, with an integrated internal competitive grant from the College of Agriculture and Life Sciences at Virginia Tech.

Author information

Authors and Affiliations

  1. Computer Science, Virginia Tech, Blacksburg, VA, 24061, USA

    Huimin Han, Rebecca Faust, Brian Felipe Keith Norambuena, Jiayue Lin & Chris North

  2. School of Plant and Environmental Sciences, Virginia Tech, Blacksburg, VA, 24061, USA

    Song Li

  3. Department of Computing and Systems Engineering, Universidad Católica del Norte, 1249004, Antofagasta, Chile

    Brian Felipe Keith Norambuena

Authors
  1. Huimin Han
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Rebecca Faust
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Brian Felipe Keith Norambuena
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Jiayue Lin
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Song Li
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Chris North
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Rebecca Faust.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Han, H., Faust, R., Keith Norambuena, B.F. et al. Explainable interactive projections of images. Machine Vision and Applications 34, 100 (2023). https://doi.org/10.1007/s00138-023-01452-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-023-01452-9

Keywords