Development of an automated combined positive score prediction pipeline using artificial intelligence on multiplexed immunofluorescence images

https://doi.org/10.1016/j.compbiomed.2022.106337Get rights and content

Highlights

  • CPS using AI-based algorithm is close to the CPS by the pathologist.

  • CPS using mIF images is close to the CPS using pharmDX PD-L1 IHC images.

  • Artefacts in mIF can be detected by hybrid of AI-based and rule-based algorithms.

  • AI-based CPS can save noticeable time and cost in cancer diagnosis.

Abstract

Immunotherapy targeting immune checkpoint proteins, such as programmed cell death ligand 1 (PD-L1), has shown impressive outcomes in many clinical trials but only 20%–40% of patients benefit from it. Utilizing Combined Positive Score (CPS) to evaluate PD-L1 expression in tumour biopsies to identify patients with the highest likelihood of responsiveness to anti-PD-1/PD-L1 therapy has been approved by the Food and Drug Administration for several solid tumour types. Current CPS workflow requires a pathologist to manually score the two-colour PD-L1 chromogenic immunohistochemistry image. Multiplex immunofluorescence (mIF) imaging reveals the expression of an increased number of immune markers in tumour biopsies and has been used extensively in immunotherapy research. Recent rapid progress of Artificial Intelligence (AI)-based imaging analysis, particularly Deep Learning, provides cost effective and high-quality solutions to healthcare. In this article, we propose an imaging pipeline that utilizes three-colour mIF images (DAPI, PD-L1, and Pan-cytokeratin) as input and predicts the CPS using AI techniques. Our novel pipeline is composed of three modules employing algorithms of image processing, machine learning, and deep learning techniques. The first module of quality check (QC) detects and removes the image regions contaminated with sectioning and staining artefacts. The QC module ensures that only image regions free of the three common artefacts are used for downstream analysis. The second module of nuclear segmentation uses deep learning to segment and count nuclei in the DAPI images wherein our specialized method can accurately separate touching nuclei. The third module of cell phenotyping calculates CPS by identifying and counting PD-L1 positive cells and tumour cells. These modules are data-efficient and require only few manual annotations for training purposes. Using tumour biopsies from a clinical trial, we found that the CPS from the AI-based models shows a high Spearman correlation (78%, p = 0.003) to the pathologist-scored CPS.

Introduction

Clinical outcomes of immunotherapy by blocking the PD-1/PD-L1 checkpoint pathway are remarkable for patients with recurrent or unresectable late-stage cancer, however, only 20%–40% of patients achieve long-term durable responses [[1], [2], [3]]. Identifying patients who are likely to respond to this treatment, and further understanding mechanisms of resistance to immune checkpoint inhibitors (ICI) are crucial for directing new therapies and reducing medical cost. PD-L1 quantification using immunohistochemistry (IHC) is the most widely validated biomarker analysis to stratify patients who would benefit from ICI, and several PD-L1 IHC based companion diagnostic tests are already approved by the Food and Drug Administration (FDA) [4]. The Combined Positive Score (CPS) on the tumour biopsies stained with Agilent PD-L1 IHC 22C3 PharmDx assay [5,6], has been used to identify patients with an increased likelihood of responsiveness to anti-PD-1 therapy in cancers such as head and neck squamous cell carcinoma (HNSCC) [7], urothelial carcinoma (UC) [8,9], gastric or gastroesophageal junction (GEJ) adenocarcinoma [[10], [11], [12]], and advanced cervical cancer [13,14]. CPS is calculated as the ratio of the number of PD-L1 positive (+) cells to the number of viable tumour cells and multiplied by 100. Current scoring practice requires the pathologist to score PD-L1 expression on at least 100 tumour cells with inclusion and exclusion rules depending on the cancer types [15]. The manual scoring is laborious, time consuming, and dependent on human interpretation. Thus, developing an automated CPS algorithm is beneficial in easing the burden on the pathologists and facilitates the search for reliable immune biomarkers for other ICI [4].

Few automatic CPS scoring on two-colour chromogenic PD-L1 IHC images have been reported [16,17], however, as far as we know we cannot find any publication on automatic CPS scoring using immunofluorescence (IF) images. In comparison to IHC, more immune markers can be labelled using multiplexed immunofluorescence staining method on the same biopsies with high intensity amplification [18,19]. This advantage provides the researchers with the access to understand the complex spatial contexture and cell interactions in tumour microenvironment which has shown to be highly relevant to tumour progression and cancer therapeutic outcomes [20,21]. Thus, developing an automatic CPS scoring using mIF images can expand the power of companion diagnostic using clinical biopsies to benefit cancer patients.

In this study, we propose an automated end-to-end CPS prediction pipeline using mIF images. We employed three main AI-based modules: quality check (QC), nuclear segmentation, and cell phenotyping to design our CPS prediction pipeline. The QC module identifies and excludes the common artefacts in the mIF images, which are different from the IHC image artefacts [18]. We evaluated this pipeline using the biopsies from recurrent HNSCC patients, who had enrolled in a clinical study sponsored by Rakuten Medical Inc. ([22,23]). The IHC images using PD-L1 IHC 22C3 pharmDx and the corresponding adjacent paired mIF images using GE-MultiOmyx platform were prepared for all biopsies. We verified the performance of our AI-based CPS prediction by comparing with the gold-standard CPS of the same biopsies manually scored by a certified pathologist.

Section snippets

Patients and tumour specimens

Tumour biopsies were collected from the patients with recurrent HNSCC (rHNSCC) who had enrolled in a Rakuten Medical sponsored study (NCT02422979). Haematoxylin and eosin (H&E) staining, PD-L1 IHC staining using PD-L1 IHC 22C3 pharmDx kit (Agilent, Santa Clara, CA, USA) [5], the mIF staining with PD-L1 antibody (SP142, Abcam, Cat: ab236238) and PanCK antibody (PCK-26, Sigma, Cat: C5992), and the mIF imaging acquisition using GE MultiOmyx platform (GE Research, Niskayuna, NY, USA) were completed

Performance in detecting artefacts, nuclear segmentation, and cell phenotyping in mIF images

We evaluated the detection of blur on CPS-test dataset images, air-bubble artefacts from 28 images among the Akoya-set, and tissue-fold artefacts using 95 images also from the Akoya-set. For air-bubble detection, 18 images contained air-bubbles and 10 images did not. For tissue-fold and blur detection [50], images contained different sizes of tissue-fold or blur regions whereas 30 did not. We obtained image classification F1-scores of 74%, 88.8%, and 87.4% for blur, tissue-fold, and air-bubble,

Discussion

Immunotherapy targeting the PD-1/PD-L1 checkpoint and deep learning for image analysis are two breakthroughs in the last decade. Determining PD-L1 expression by IHC is the most widely studied predictive biomarker to stratify patients for cancer immunotherapy targeting the PD-L1 checkpoint pathway. Many ongoing and future clinical trials of immunotherapy also contain biomarker testing, including PD-L1 IHC, in enriching patient responses [4]. Deep Learning has shown great promise in clinical

Conclusions

Optimizing the cancer treatment efficacy for individual cancer patients while reducing the cost is the key goal for the health care industry. We addressed those challenges by developing an automatic CPS scoring pipeline to stratify cancer patients who are likely benefit from immune checkpoint therapy. This proof-of-concept study shows that our automatic CPS scoring using artificial intelligence on multiplexed immunofluorescence (mIF) images is concordant with the CPS gold standard by the

Statement of author contributions

MD, SM, CWL conceived and designed the study; AV, SS, DM implemented the algorithm and analysed data; MGG and CWL provided the data; AV, SS, DM, JJ, SM, CWL reviewed and interpreted the analysis. All authors were involved in writing the paper and had final approval of the submitted and published versions.

Data availability statement

Please contact the corresponding author for data availability. All requests for data and algorithms will be reviewed by Rakuten Medical Inc. To verify if the request is subject to any intellectual property or confidentiality obligations. Any data and materials will be released via a Material Transfer Agreement.

Funding statement

The studies described herein were funded by Rakuten Medical, Inc. Abhishek Vahadane (AV), Shreya Sharma (SS), Devraj Mandal (DM), Madan Dabbeeru (MD), and Shantanu Majumdar (SM) are employed by Rakuten India Enterprise; Josephine Jakthong (JJ) is a consulting pathologist for Rakuten Medical, Inc.; Miguel Garcia-Guzman (MGG) and Chung-Wein Lee (CWL) are employed by Rakuten Medical, Inc.

Conflict of interest disclosure

The authors have no conflicts of interest to disclose.

Ethics approval statement

The original study in which patient biopsies were collected was performed in accordance with the principles of the Declaration of Helsinki (1964), Good Clinical Practice guidelines, and applicable US Code of Federal Regulations (CFR), 21 CFR Part 50 & 312. The study protocol, informed consent form, and any other appropriate documents were approved by the Institutional Review Board/Independent Ethics Committee for each participating center.

Patient consent statement

The patients signed informed consent forms prior to enrolment in the Rakuten Medical, Inc. Sponsored trial.

Permission to reproduce material from other sources

The article does not include data from other resources.

Clinical trial registration

NCT02422979

Declaration of competing interest

None Declared.

Acknowledgements

The authors wish to thank the patients and families who participated in the Rakuten Medical sponsored parent clinical study for their participation. The last author thanks the language editing and manuscript proofreading by Dr. Laura Spuhler and Ms. Cindy Chen.

References (64)

  • E.B. Garon et al.

    Five-Year overall survival for patients with advanced non‒small-cell lung cancer treated with pembrolizumab: results from the phase I KEYNOTE-001 study

    J. Clin. Oncol.

    (2019)
  • D.B. Doroshow et al.

    PD-L1 as a biomarker of response to immune-checkpoint inhibitors

    Nat. Rev. Clin. Oncol.

    (2021)
  • PD-L1 IHC 22C3 pharmDx Interpretation Manual - Head and Neck Squamous Cell Carcinoma

    (2019)
  • US Food and Drug Administration. Summary of Safety and Effectiveness Data (SSED) for PD-L1 IHC 22C3 pharmDx. In:...
  • J. Vuky et al.

    Long-term outcomes in KEYNOTE-052: phase II study investigating first-line pembrolizumab in cisplatin-ineligible patients with locally advanced or metastatic urothelial cancer

    J. Clin. Oncol.

    (2020)
  • Y.J. Bang et al.

    Pembrolizumab alone or in combination with chemotherapy as first-line therapy for patients with advanced gastric or gastroesophageal junction adenocarcinoma: results from the phase II nonrandomized KEYNOTE-059 study

    Gastric Cancer

    (2019)
  • C.S. Fuchs et al.

    Safety and efficacy of pembrolizumab monotherapy in patients with previously treated advanced gastric and gastroesophageal junction cancer: phase 2 clinical KEYNOTE-059 trial

    JAMA Oncol.

    (2018)
  • L. Fashoyin-Aje et al.

    FDA approval summary: pembrolizumab for recurrent locally advanced or metastatic gastric or gastroesophageal junction adenocarcinoma expressing PD-L1

    Oncol.

    (2019)
  • J.S. Frenel et al.

    Safety and efficacy of pembrolizumab in advanced, programmed death ligand 1-positive cervical cancer: results from the phase ib KEYNOTE-028 trial

    J. Clin. Oncol.

    (2017)
  • H.C. Chung et al.

    Efficacy and safety of pembrolizumab in previously treated advanced cervical cancer: results from the phase II KEYNOTE-158 study

    J. Clin. Oncol.

    (2019)
  • K. Kulangara et al.

    Clinical utility of the combined positive score for programmed death ligand-1 expression and the approval of pembrolizumab for treatment of gastric cancer

    Arch. Pathol. Lab Med.

    (2019)
  • Y. Park et al.

    PD-L1 testing in gastric cancer by the combined positive score of the 22C3 PharmDx and SP263 assay with clinically relevant cut-offs

    Cancer Res. Treat.

    (2020)
  • B. Puladi et al.

    Automated PD-L1 scoring using artificial intelligence in head and neck squamous cell carcinoma

    Cancers

    (2021)
  • W.C.C. Tan et al.

    Overview of multiplex immunohistochemistry/immunofluorescence techniques in the era of cancer immunotherapy

    Cancer Commun.

    (2020)
  • F. Petitprez et al.

    The tumor microenvironment in the response to immune checkpoint blockade therapies

    Front. Immunol.

    (2020)
  • D. Bruni et al.

    The immune contexture and Immunoscore in cancer prognosis and therapeutic efficacy

    Nat. Rev. Cancer

    (2020)
  • H. Kobayashi et al.

    Near-Infrared photoimmunotherapy of cancer

    Acc. Chem. Res.

    (2019)
  • M. Mitsunaga et al.

    Cancer cell–selective in vivo near infrared photoimmunotherapy targeting specific membrane molecules

    Nat. Med.

    (2011)
  • M.J. Gerdes et al.

    Highly multiplexed single-cell analysis of formalin-fixed, paraffin-embedded cancer tissue

    Proc. Natl. Acad. Sci. U. S. A.

    (2013)
  • MultiOmyx™ Hyperplexed Immunofluorescence Assay

    (2019)
  • A. Janowczyk et al.

    HistoQC: an open-source quality control tool for digital pathology slides

    JCO clinic. cancer inform.

    (2019)
  • X.M. Lopez et al.

    An automated blur detection method for histological whole slide imaging

    PLoS One

    (2013)
  • Cited by (0)

    View full text