research-article

Exploring Hyperspectral Histopathology Image Segmentation from a Deformable Perspective

Authors:

Yan WangAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 242 - 251

https://doi.org/10.1145/3581783.3611796

Published: 27 October 2023 Publication History

Abstract

Hyperspectral images (HSIs) offer great potential for computational pathology. However, limited by the spectral redundancy and the lack of spectral prior in popular 2D networks, previous HSI based techniques do not perform well. To address these problems, we propose to segment HSIs from a deformable perspective, which processes different spectral bands independently and fuses spatiospectral features of interest via deformable attention mechanisms. In addition, we propose Deformable Self-Supervised Spectral Regression (DF-S3R), which introduces two self-supervised pre-text tasks based on the low rank prior of HSIs enabling the network learning with spectrum-related features. During pre-training, DF-S3R learns both spectral structures and spatial morphology, and the jointly pre-trained architectures help alleviate the transfer risk to downstream fine-tuning. Compared to previous works, experiments show that our deformable architecture and pre-training method perform much better than other competitive methods on pathological semantic segmentation tasks, and the visualizations indicate that our method can trace the critical spectral characteristics from subtle spectral disparities. Code will be released at https://github.com/Ayakax/DFS3R.

References

[1]

V. Backman, M. B. Wallace, L. T. Perelman, and et al. 2000. Detection of preinvasive cancer cells. Nature, Vol. 406 (2000), 35--36.

[2]

Hangbo Bao, Li Dong, and Furu Wei. 2022. BEiT: BERT Pre-Training of Image Transformers. In ICLR.

[3]

Marcel Bengs, Nils Gessert, Wiebke Laffers, Dennis Eggert, Stephan Westermann, Nina A. Müller, Andreas O. H. Gerstner, Christian Betz, and Alexander Schlaefer. 2020. Spectral-spatial Recurrent-Convolutional Networks for In-Vivo Hyperspectral Tumor Type Classification. In MICCAI.

[4]

Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang. 2022. Swin-Unet: Unet-Like Pure Transformer for Medical Image Segmentation. In ECCV Workshops.

[5]

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-End Object Detection with Transformers. In ECCV, Vol. 12346. 213--229.

[6]

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A Simple Framework for Contrastive Learning of Visual Representations. In ICML.

[7]

Xinlei Chen and Kaiming He. 2021. Exploring Simple Siamese Representation Learning. In CVPR.

[8]

Ö zgü n cC icc ek, Ahmed Abdulkadir, Soeren S. Lienkamp, Thomas Brox, and Olaf Ronneberger. 2016. 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation. In MICCAI.

[9]

Jean-Bastien Grill, Florian Strub, Florent Altché, and et al. 2020. Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning. In NeurIPS.

[10]

Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R. Roth, and Daguang Xu. 2021. Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images. In MICCAI Workshops.

[11]

Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett A. Landman, Holger R. Roth, and Daguang Xu. 2022. UNETR: Transformers for 3D Medical Image Segmentation. In WACV.

[12]

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. 2022. Masked Autoencoders Are Scalable Vision Learners. In CVPR.

[13]

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. 2020. Momentum Contrast for Unsupervised Visual Representation Learning. In CVPR.

[14]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In CVPR.

[15]

Danfeng Hong, Zhu Han, Jing Yao, Lianru Gao, Bing Zhang, Antonio Plaza, and Jocelyn Chanussot. 2022. SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers. IEEE Trans. Geosci. Remote. Sens., Vol. 60 (2022), 1--15.

[16]

Fabian Isensee, Paul F. Jaeger, Simon A. A. Kohl, Jens Petersen, and Klaus H. Maier-Hein. 2021. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods (2021).

[17]

S. Karim, Akeel Q., and et al. 2023. Hyperspectral Imaging: A Review and Trends towards Medical Imaging. Current Medical Imaging, Vol. 19, 5 (2023), 417--427.

[18]

Bing Liu, Anzhu Yu, Xuchu Yu, Ruirui Wang, Kuiliang Gao, and Wenyue Guo. 2021b. Deep Multiview Learning for Hyperspectral Image Classification. IEEE Trans. Geosci. Remote. Sens., Vol. 59, 9 (2021), 7758--7772.

[19]

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021a. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In ICCV.

[20]

Ilya Loshchilov and Frank Hutter. 2017. Fixing Weight Decay Regularization in Adam. CoRR, Vol. abs/1711.05101 (2017).

[21]

Zhongtian Ma, Zhiguo Jiang, and Haopeng Zhang. 2022. Hyperspectral Image Classification Using Feature Fusion Hypergraph Convolution Neural Network. IEEE Trans. Geosci. Remote. Sens., Vol. 60 (2022), 1--14.

[22]

Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. 2016. V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation. In 3DV.

[23]

Lichao Mou, Pedram Ghamisi, and Xiao Xiang Zhu. 2018. Unsupervised Spectral-Spatial Feature Learning via Deep Residual Conv-Deconv Network for Hyperspectral Image Classification. IEEE Trans. Geosci. Remote. Sens., Vol. 56, 1 (2018), 391--406.

[24]

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In MICCAI.

[25]

Olga Russakovsky, Jia Deng, Hao Su, and et al. 2015. ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vis., Vol. 115, 3 (2015), 211--252.

Digital Library

[26]

Massimo Salvi, U. Rajendra Acharya, Filippo Molinari, and Kristen M. Meiburger. 2021. The impact of pre- and post-image processing techniques on deep learning frameworks: A comprehensive review for digital pathology image analysis. Comput. Biol. Medicine, Vol. 128 (2021), 104129.

[27]

Haoqing Wang, Yehui Tang, Kai Han, Jianyuan Guo, Zhi-Hong Deng, and Yunhe Wang. 2023. Masked Image Modeling with Local Multi-Scale Reconstruction. In CVPR.

[28]

Qian Wang, Li Sun, Yan Wang, Mei Zhou, Menghan Hu, Jiangang Chen, Ying Wen, and Qingli Li. 2021. Identification of Melanoma From Hyperspectral Pathology Image Using 3D Convolutional Networks. IEEE Trans. Medical Imaging, Vol. 40, 1 (2021), 218--227.

[29]

Yizhou Wang, Shixiang Tang, Feng Zhu, and et al. 2022. Revisiting the Transferability of Supervised Pretraining: an MLP Perspective. In CVPR.

[30]

Enze Xie, Wenhai Wang, Zhiding Yu, Animashree Anandkumar, Jose M. Alvarez, and Ping Luo. 2021. SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. In NeurIPS.

[31]

Xingran Xie, Yan Wang, and Qingli Li. 2022a. S3R: Self-supervised Spectral Regression for Hyperspectral Histopathology Image Classification. In MICCAI.

[32]

Zhenda Xie, Zheng Zhang, Yue Cao, and et al. 2022b. SimMIM: a Simple Framework for Masked Image Modeling. In CVPR.

[33]

Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. 2022c. SimMIM: A Simple Framework for Masked Image Modeling. In CVPR.

[34]

Zhang Ying, Wang Yan, Zhang Benyan, and Li Qingli. 2022. A hyperspectral dataset of precancerous lesions in gastric cancer and benchmarks for pathological diagnosis. Journal of Biophotonics (2022), e202200163.

[35]

Boxiang Yun, Yan Wang, Jieneng Chen, Huiyu Wang, Wei Shen, and Qingli Li. 2021. SpecTr: Spectral Transformer for Hyperspectral Pathology Image Segmentation. CoRR, Vol. abs/2103.03604 (2021).

[36]

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Sté phane Deny. 2021. Barlow Twins: Self-Supervised Learning via Redundancy Reduction. In ICML.

[37]

Qing Zhang, Qingli Li, Guanzhen Yu, Li Sun, Mei Zhou, and Junhao Chu. 2019. A Multidimensional Choledoch Database and Benchmarks for Cholangiocarcinoma Diagnosis. IEEE Access, Vol. 7 (2019), 149414--149421.

[38]

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. 2021. Deformable DETR: Deformable Transformers for End-to-End Object Detection. In ICLR.

[39]

L. Zhuang and M. K. Ng. 2021. FastHyMix: Fast and Parameter-Free Hyperspectral Image Mixed Noise Removal. IEEE Transactions on Neural Networks and Learning Systems, Vol. PP, 99 (2021), 1--15.

Cited By

Zhang YHuang ZHong ZWu SWu JTan KCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Mixed Prototype Correction for Causal Inference in Medical Image ClassificationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681395(4377-4386)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681395
Dong HZhou RYun BZhou HZhang BLi QWang Y(2024)Multi-stage Multi-granularity Focus-Tuned Learning Paradigm for Medical HSI SegmentationMedical Image Computing and Computer Assisted Intervention – MICCAI 202410.1007/978-3-031-72111-3_43(456-466)Online publication date: 6-Oct-2024
https://doi.org/10.1007/978-3-031-72111-3_43

Index Terms

Exploring Hyperspectral Histopathology Image Segmentation from a Deformable Perspective
1. Computing methodologies
  1. Machine learning
    1. Machine learning algorithms
      1. Feature selection

Recommendations

Exploring the Spectral Prior for Hyperspectral Image Super-Resolution
In recent years, many single hyperspectral image super-resolution methods have emerged to enhance the spatial resolution of hyperspectral images without hardware modification. However, existing methods typically face two significant challenges. First, ...
S $^{3}$ R: Self-supervised Spectral Regression for Hyperspectral Histopathology Image Classification
Medical Image Computing and Computer Assisted Intervention – MICCAI 2022
Abstract
Benefited from the rich and detailed spectral information in hyperspectral images (HSI), HSI offers great potential for a wide variety of medical applications such as computational pathology. But, the lack of adequate annotated data and the high ... $^{}$ $^{}$ $^{}$ $^{}$
Hyperspectral image super-resolution through clustering-based sparse representation
Abstract
Promoting the spatial resolution of hyperspectral sensors is expected to improve computer vision tasks. However, due to the physical limitations of imaging sensors, the hyperspectral image is often of low spatial resolution. In this paper, we ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Shanghai Natural Science Foundation
National Natural Science Foundation of China
Science and Technology Commission of Shanghai Municipality

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
191
Total Downloads

Downloads (Last 12 months)108
Downloads (Last 6 weeks)2

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang YHuang ZHong ZWu SWu JTan KCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Mixed Prototype Correction for Causal Inference in Medical Image ClassificationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681395(4377-4386)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681395
Dong HZhou RYun BZhou HZhang BLi QWang Y(2024)Multi-stage Multi-granularity Focus-Tuned Learning Paradigm for Medical HSI SegmentationMedical Image Computing and Computer Assisted Intervention – MICCAI 202410.1007/978-3-031-72111-3_43(456-466)Online publication date: 6-Oct-2024
https://doi.org/10.1007/978-3-031-72111-3_43

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten