A deep-learning framework to predict cancer treatment response from histopathology images through imputed transcriptomics – Nature

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.
Advertisement
Nature Cancer (2024)
1328 Accesses
157 Altmetric
Metrics details
Advances in artificial intelligence have paved the way for leveraging hematoxylin and eosin-stained tumor slides for precision oncology. We present ENLIGHT–DeepPT, an indirect two-step approach consisting of (1) DeepPT, a deep-learning framework that predicts genome-wide tumor mRNA expression from slides, and (2) ENLIGHT, which predicts response to targeted and immune therapies from the inferred expression values. We show that DeepPT successfully predicts transcriptomics in all 16 The Cancer Genome Atlas cohorts tested and generalizes well to two independent datasets. ENLIGHT–DeepPT successfully predicts true responders in five independent patient cohorts involving four different treatments spanning six cancer types, with an overall odds ratio of 2.28 and a 39.5% increased response rate among predicted responders versus the baseline rate. Notably, its prediction accuracy, obtained without any training on the treatment data, is comparable to that achieved by directly predicting the response from the images, which requires specific training on the treatment evaluation cohorts.
This is a preview of subscription content, access via your institution

Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
Prices may be subject to local taxes which are calculated during checkout
TCGA histological images and their corresponding gene expression profiles were downloaded from the Genomic Data Commons Data Portal (https://portal.gdc.cancer.gov). The TransNEO-breast dataset is available at https://ega-archive.org/studies/EGAS00001004582. The NCI-brain and bintrafusp alfa cohorts were obtained from the NCI. The trastuzumab1 cohort is a subset of the TransNEO-breast dataset. Trastuzumab2 cohort data were downloaded from The Cancer Imaging Archive database (https://www.cancerimagingarchive.net). The ALKi dataset was made available by the University of Colorado, and the PARPi dataset was made available by Sheba Medical Center. Restrictions apply to the distribution of these data, which were used under license for the current study. Access to the ALKi dataset may be requested from S.P. (sharon.pine@cuanschutz.edu). Access to the PARPi dataset may be requested from T.G. (Talia.Golan@sheba.health.gov.il). All DeepPT-predicted expression and relevant response data, along with the code to calculate performance measures, are available via GitHub at https://github.com/PangeaResearch/enlight-deeppt-data. Source data are provided with this paper. All other data supporting the findings of this study are available from the corresponding authors upon reasonable request.
The DeepPT framework is available for academic research purposes via Zenodo at https://doi.org/10.5281/zenodo.11125591 (ref. 61). ENLIGHT scores, given expression profiles (either measured directly from the tumor or predicted from slides), can be calculated using a web service at https://ems.pangeabiomed.com/. Any additional information is available from the corresponding authors upon request.
Golub, T. R. et al. Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science 286, 531–537 (1999).
Article  CAS  PubMed  Google Scholar 
Curtis, C. et al. The genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups. Nature 486, 346–352 (2012).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Doroshow, D. B. & Doroshow, J. H. Genomics and the history of precision oncology. Surg. Oncol. Clin. N. Am. 29, 35–49 (2020).
Article  PubMed  Google Scholar 
Rosenthal, J. et al. Building tools for machine learning and artificial intelligence in cancer research: best practices and a case study with the PathML toolkit for computational pathology. Mol. Cancer Res. 20, 202–206 (2022).
Article  CAS  PubMed  Google Scholar 
Ström, P. et al. Artificial intelligence for diagnosis and grading of prostate cancer in biopsies: a population-based, diagnostic study. Lancet Oncol. 21, 222–232 (2020).
Article  PubMed  Google Scholar 
Fu, Y. et al. Pan-cancer computational histopathology reveals mutations, tumor composition and prognosis. Nat. Cancer 1, 800–810 (2020).
Article  CAS  PubMed  Google Scholar 
Noorbakhsh, J. et al. Deep learning-based cross-classifications reveal conserved spatial behaviors within tumor histological images. Nat. Commun. 11, 6367 (2020).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Kim, J. et al. Unsupervised discovery of tissue architecture in multiplexed imaging. Nat. Methods 19, 1653–1661 (2022).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Coudray, N. et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24, 1559–1567 (2018).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Swiderska-Chadaj, Z. et al. Learning to detect lymphocytes in immunohistochemistry with deep learning. Med. Image Anal. 58, 101547 (2019).
Article  PubMed  Google Scholar 
Yu, K.-H. et al. Classifying non-small cell lung cancer types and transcriptomic subtypes using convolutional neural networks. J. Am. Med. Inform. Assoc. 27, 757–769 (2020).
Article  PubMed  PubMed Central  Google Scholar 
Couture, H. D. et al. Image analysis with deep learning to predict breast cancer grade, ER status, histologic subtype, and intrinsic subtype. NPJ Breast Cancer 4, 30 (2018).
Article  PubMed  PubMed Central  Google Scholar 
Campanella, G. et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat. Med. 25, 1301–1309 (2019).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Xu, H. et al. Spatial heterogeneity and organization of tumor mutation burden with immune infiltrates within tumors based on whole slide images correlated with patient survival in bladder cancer. J. Pathol. Inform. 13, 100105 (2022).
Article  PubMed  PubMed Central  Google Scholar 
Kather, J. N. et al. Pan-cancer image-based detection of clinically actionable genetic alterations. Nat. Cancer 1, 789–799 (2020).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Qu, H. et al. Genetic mutation and biological pathway prediction based on whole slide images in breast carcinoma using deep learning. NPJ Precis. Oncol. 5, 87 (2021).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Schaumberg, A. J., Rubin, M. A. & Fuchs, T. J. H&E-stained whole slide image deep learning predicts SPOP mutation state in prostate cancer. Preprint at bioRxiv https://doi.org/10.1101/064279 (2018).
Tsou, P. & Wu, C.-J. Mapping driver mutations to histopathological subtypes in papillary thyroid carcinoma: applying a deep convolutional neural network. J. Clin. Med. 8, 1675 (2019).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Chang, P. et al. Deep-learning convolutional neural networks accurately classify genetic mutations in gliomas. AJNR Am. J. Neuroradiol. 39, 1201–1207 (2018).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Kather, J. N. et al. Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nat. Med. 25, 1054–1056 (2019).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Kim, R. H. et al. A deep learning approach for rapid mutational screening in melanoma. Preprint at bioRxiv https://doi.org/10.1101/610311 (2019).
Chen, M. et al. Classification and mutation prediction based on histopathology H&E images in liver cancer using deep learning. NPJ Precis. Oncol. 4, 14 (2020).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Ghaffari Laleh, N. et al. Benchmarking weakly-supervised deep learning pipelines for whole slide classification in computational pathology. Med. Image Anal. 79, 102474 (2022).
Article  PubMed  Google Scholar 
Mobadersany, P. et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc. Natl Acad. Sci. USA 115, E2970–E2979 (2018).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Cheng, J. et al. Integrative analysis of histopathological images and genomic data predicts clear cell renal cell carcinoma prognosis. Cancer Res. 77, e91–e100 (2017).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Beck, A. H. et al. Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Sci. Transl. Med. 3, 108ra113 (2011).
Article  PubMed  Google Scholar 
Bulten, W. et al. Automated deep-learning system for Gleason grading of prostate cancer using biopsies: a diagnostic study. Lancet Oncol. 21, 233–241 (2020).
Article  PubMed  Google Scholar 
Courtiol, P. et al. Deep learning-based classification of mesothelioma improves prediction of patient outcome. Nat. Med. 25, 1519–1525 (2019).
Article  CAS  PubMed  Google Scholar 
Boehm, K. M. et al. Multimodal data integration using machine learning improves risk stratification of high-grade serous ovarian cancer. Nat. Cancer 3, 723–733 (2022).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Chen, Y. et al. Computational pathology improves risk stratification of a multi-gene assay for early stage ER+ breast cancer. NPJ Breast Cancer 9, 40 (2023).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Zheng, H., Momeni, A., Cedoz, P.-L., Vogel, H. & Gevaert, O. Whole slide images reflect DNA methylation patterns of human tumors. NPJ Genom. Med. 5, 11 (2020).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Wang, H. et al. Mitosis detection in breast cancer pathology images by combining handcrafted and convolutional neural network features. J. Med. Imaging (Bellingham) 1, 034003 (2014).
Article  PubMed  Google Scholar 
Turkki, R., Linder, N., Kovanen, P. E., Pellinen, T. & Lundin, J. Antibody-supervised deep learning for quantification of tumor-infiltrating immune cells in hematoxylin and eosin stained breast cancer samples. J. Pathol. Inform. 7, 38 (2016).
Article  PubMed  PubMed Central  Google Scholar 
Saltz, J. et al. Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images. Cell Rep. 23, 181–193 (2018).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Page, D. B. et al. Spatial analyses of immune cell infiltration in cancer: current methods and future directions: a report of the International Immuno-Oncology Biomarker Working Group on Breast Cancer. J. Pathol. 260, 514–532 (2023).
Article  CAS  PubMed  Google Scholar 
Wang, X. et al. Spatial interplay patterns of cancer nuclei and tumor-infiltrating lymphocytes (TILs) predict clinical benefit for immune checkpoint inhibitors. Sci. Adv. 8, eabn3966 (2022).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Hu, J. et al. Using deep learning to predict anti-PD-1 response in melanoma and lung cancer patients from histopathology images. Transl. Oncol. 14, 100921 (2021).
Article  CAS  PubMed  Google Scholar 
Johannet, P. et al. Using machine learning algorithms to predict immunotherapy response in patients with advanced melanoma. Clin. Cancer Res. 27, 131–140 (2021).
Article  CAS  PubMed  Google Scholar 
Zhang, F. et al. Predicting treatment response to neoadjuvant chemoradiotherapy in local advanced rectal cancer by biopsy digital pathology image features. Clin. Transl. Med. 10, e110 (2020).
Article  PubMed  PubMed Central  Google Scholar 
Honomichl, N. HER2 and trastuzumab treatment response H&E slides with tumor ROI annotations (HER2 tumor ROIs). The Cancer Imaging Archive https://doi.org/10.7937/E65C-AM96 (2022).
He, B. et al. Integrating spatial gene expression and breast tumour morphology via deep learning. Nat. Biomed. Eng. 4, 827–834 (2020).
Article  CAS  PubMed  Google Scholar 
Wang, Y. et al. Predicting molecular phenotypes from histopathology images: a transcriptome-wide expression–morphology analysis in breast cancer. Cancer Res. 81, 5115–5126 (2021).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Monjo, T., Koido, M., Nagasawa, S., Suzuki, Y. & Kamatani, Y. Efficient prediction of a spatial transcriptomics profile better characterizes breast cancer tissue sections without costly experimentation. Sci. Rep. 12, 4133 (2022).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Schmauch, B. et al. A deep learning model to predict RNA-Seq expression of tumours from whole slide images. Nat. Commun. 11, 3877 (2020).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Levy-Jurgenson, A., Tekpli, X., Kristensen, V. N. & Yakhini, Z. Spatial transcriptomics inferred from pathology whole-slide images links tumor heterogeneity to survival in breast and lung cancer. Sci. Rep. 10, 18802 (2020).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Alsaafin, A., Safarpoor, A., Sikaroudi, M., Hipp, J. D. & Tizhoosh, H. R. Learning to predict RNA sequence expressions from whole slide images with applications for search and classification. Commun. Biol. 6, 304 (2023).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Dinstag, G. et al. Clinically oriented prediction of patient response to targeted and immunotherapies from the tumor transcriptome. Med 4, 15–30 (2023).
Article  CAS  PubMed  Google Scholar 
Sammut, S.-J. et al. Multi-omic machine learning predictor of breast cancer therapy response. Nature 601, 623–629 (2022).
Article  CAS  PubMed  Google Scholar 
Zheng, Y. et al. Digital profiling of cancer transcriptomes from histology images with grouped vision attention. Preprint at bioRxiv https://doi.org/10.1101/2023.09.28.560068 (2023).
Hanahan, D. & Weinberg, R. A. Hallmarks of cancer: the next generation. Cell 144, 646–674 (2011).
Article  CAS  PubMed  Google Scholar 
Iorio, F. et al. Pathway-based dissection of the genomic heterogeneity of cancer hallmarks’ acquisition with SLAPenrich. Sci. Rep. 8, 6713 (2018).
Article  PubMed  PubMed Central  Google Scholar 
Subramanian, A. et al. Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles. Proc. Natl Acad. Sci. USA 102, 15545–15550 (2005).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Richardsen, E. et al. Evaluation of the proliferation marker Ki-67 in a large prostatectomy cohort. PLoS ONE 12, e0186852 (2017).
Article  PubMed  PubMed Central  Google Scholar 
Whitfield, M. L., George, L. K., Grant, G. D. & Perou, C. M. Common markers of proliferation. Nat. Rev. Cancer 6, 99–106 (2006).
Article  CAS  PubMed  Google Scholar 
Liberzon, A. et al. The Molecular Signatures Database (MSigDB) hallmark gene set collection. Cell Syst. 1, 417–425 (2015).
Article  CAS  PubMed  PubMed Central  Google Scholar 
Farahmand, S. et al. Deep learning trained on hematoxylin and eosin tumor region of interest predicts HER2 status and trastuzumab treatment response in HER2+ breast cancer. Mod. Pathol. 35, 44–51 (2022).
Article  CAS  PubMed  Google Scholar 
Strauss, J. et al. Bintrafusp alfa, a bifunctional fusion protein targeting TGF-β and PD-L1, in patients with human papillomavirus-associated malignancies. J. Immunother. Cancer 8, e001395 (2020).
Article  PubMed  PubMed Central  Google Scholar 
Kanopoulos, N., Vasanthavada, N. & Baker, R. L. Design of an image edge detection filter using the Sobel operator. IEEE J. Solid-State Circuits 23, 358–367 (1988).
Article  Google Scholar 
Macenko, M. et al. A method for normalizing histology slides for quantitative analysis. in 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro 1107–1110 (IEEE, 2009); https://doi.org/10.1109/ISBI.2009.5193250
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 770–778 (IEEE, 2016); https://doi.org/10.1109/cvpr.2016.90
Hoang, D.-T. et al. DeepPT: a deep learning model for predicting transcriptomics from histopathology images. Zenodo https://doi.org/10.5281/zenodo.11125591 (2024).
Download references
This work was partially supported by grant DP190103402 from the Australian Research Council (D.-T.H., E.A.S.) and by the Intramural Research Program of the National Institutes of Health (NIH), National Cancer Institute (NCI), Center for Cancer Research (CCR) (E.D.S., S.S., N.S., E.R.). This work used the supercomputational resources of the Australian National Computational Infrastructure (AUNCI) and the Australian National University Merit Allocation Scheme (ANUMAS). D.-T.H. would like to thank T. Bui for the helpful discussion.
Biological Data Science Institute, College of Science, Australian National University, Canberra, Australian Capital Territory, Australia
Danh-Tai Hoang, Katherine Caley & Eric A. Stone
Pangea Biomed Ltd., Tel Aviv, Israel
Gal Dinstag, Doreen S. Ben-Zvi, Efrat Elis, Tuvik Beker & Ranit Aharonov
Cancer Data Science Laboratory, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA
Eldad D. Shulman, Sanju Sinha, Neelam Sinha, Peng Jiang & Eytan Ruppin
Department of Immunology, University of Pittsburgh, Pittsburgh, PA, USA
Leandro C. Hermida
Tumor Microenvironment Center, UPMC Hillman Cancer Center, University of Pittsburgh, Pittsburgh, PA, USA
Leandro C. Hermida
Breast Cancer Now Toby Robins Research Centre, Institute of Cancer Research, London, UK
Stephen-John Sammut
The Royal Marsden Hospital NHS Foundation Trust, London, UK
Stephen-John Sammut
Laboratory of Pathology, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA
Christopher H. Dampier & Kenneth Aldape
Oncology Institute, Sheba Medical Center at Tel-Hashomer, Tel Aviv University, Tel Aviv, Israel
Chani Stossel & Talia Golan
Division of Medical Oncology, University of Colorado Anschutz Medical Campus, Aurora, CO, USA
Tejas Patil & Sharon R. Pine
Thoracic and GI Malignancies Branch, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA
Arun Rajan
Center for Immuno-Oncology, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA
Wiem Lassoued, Julius Strauss, Shania Bailey & Jason Redman
Surgical Oncology Program, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA
Clint Allen
Laboratory of Genitourinary Cancer Pathogenesis, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA
Scott Wilkinson & Adam G. Sowalsky
School of Clinical Medicine, University of Cambridge, Cambridge, UK
Carlos Caldas
Genitourinary Malignancy Branch, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA
James L. Gulley
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
D.-T.H., S.S., E.A.S. and E.R. initialed the study. D.-T.H. developed the DeepPT framework and performed predictions of gene expression from H&E slides. G.D. performed pathway enrichment analysis, conducted the analysis of prognostic signatures and applied ENLIGHT to predict patient response from the predicted gene expression, with assistance from D.S.B.-Z. and E.E. L.C.H. assisted with data curation and gene expression preprocessing. D.-T.H. and E.D.S. analyzed the correlation between the predicted gene expression and the abundance of tumor-infiltrating lymphocytes. E.D.S. conducted the gene set enrichment analysis. K.C. assisted with image preprocessing. S.-J.S., C.H.D., C.S., T.P., A.R., W.L., J.S., S.B., C.A., J.R., T.G., S.W., A.G.S., S.R.P., C.C., J.L.G. and K.A. provided external and clinical datasets. N.S. and P.J. interpreted the results and provided feedback on the study. T.B., K.A., R.A., E.A.S. and E.R. supervised the study. D.-T.H., G.D., R.A. and E.R. wrote the paper with assistance and feedback from all the other coauthors.
Correspondence to Danh-Tai Hoang, Eric A. Stone or Eytan Ruppin.
D.-T.H., E.A.S., E.R., G.D., R.A. and T.B. are listed as inventors on a patent (application no. 63/349,829, United States, 2022) filed based on the methodology outlined in this study. G.D., D.S.B., E.E., T.B. and R.A. are employees of Pangea Biomed. E.R. is a cofounder of Medaware, Metabomed and Pangea Biomed (divested from the latter). E.R. serves as a non-paid scientific consultant to Pangea Biomed under a collaboration agreement between Pangea Biomed and the NCI. The other authors declare no competing interests.
Nature Cancer thanks Ulysses Balis and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
(a) The feature compression subnetwork consists of an input layer of 2,048 neurons, a bottleneck of 512 neurons, and an output layer of 2,048 neurons. (b) The MLP regression subnetwork consists of an input layer of 512 neurons, a hidden layer of 512 neurons, and an output layer with the number of neurons reflecting the number of genes. (c) In the ensemble learning strategy (bagging), five models were trained independently with five internal training-validation splits; these five model predictions were averaged to make the final prediction. (d) In the model selection strategy, the ‘best’ model with the highest performance on the validation set was chosen to make predictions on the test set. Of note, DeepPT uses ensemble learning.
The violin plots depict the correlations between the predicted and measured expression values across the cohort samples obtained by HE2RNA (light pink) and DeepPT (light blue) for all genes (a), the top 1,000 genes (b), the top 2,000 genes (c), and the top 3,000 genes (d) with the highest correlations. The results presented in this figure were measured by the mean of 5 folds, as reported in44. Except for this figure, all other results presented in this study were measured across the entire test samples, consistent with the approach used in49. P-values were calculated using the one-sided Mann-Whitney U test. In violin plots, the central mark is the median. The number of patients in each cohort is shown in parentheses.
Source data
UMAP visualization of 2,048 histopathological features that were extracted by using pre-trained ResNet50 CNN. 4,000 image tiles from each dataset were selected randomly to illustrate. Each point represents each feature vector of one image tile.
(a) Performance of ENLIGHT-DeepPT (light blue bars) and the respective drug target(s) expression (gray bars). (b) Performance of ENLIGHT-DeepPT when using the same methodology described in47 (light blue bars) and a version of ENLIGHT-DeepPT that incorporates the target expression in the scoring method for antibodies (gray bars). (c) Performance of ENLIGHT-DeepPT when using the same methodology described in47 to generate genetic interaction networks that constitutes ENLIGHT’s predictive biomarkers (light blue bars) and a revised methodology (gray bars) where we restricted ENLIGHT’s biomarker to only include genes that showed high positive correlation (R > 0.4) between actual and DeepPT-predicted values among the respective TCGA cohort (that is, according to the cancer type of each of the five drug response datasets). Results are shown for each of the three datasets where antibody drugs were used and the aggregation of them. Odds Ratio (OR) for each dataset were obtained by using the same clinical decision threshold that has been previously established in47. The number of patients in each cohort is shown in parentheses.
Source data
The number of tiles in each slide image from TCGA and NCI-Brain datasets ranges from 100 to 8,000 (a-e), while the number of tiles in each TransNEO-Breast slide image is much smaller, ranging from 100 to 1,000 (f).
Source data
The median expression over samples of each gene commonly varies from 10 to 100,000 for every dataset considered in this study (a to f).
Source data
(a-d) Difference between ResNet features and AutoEncoder features. Histograms of median and standard deviation of ResNet features (a-b) and AutoEncoder features (c-d). The TCGA-BRCA cohort was selected as an example. (e-f) Model performance on external validation datasets. The violin plots depict the distribution of Pearson correlations between the predicted and experimentally measured expression values across the cohort samples for the top 1,000 genes with the highest correlation. The bar charts indicate the number of genes exhibiting Pearson correlations between the predicted and measured expression values across the cohort samples that are above 0.4. The results are presented separately for each external validation set, TransNeo-Breast (e) (n = 160 patients) and NCI-Brain (f) (n = 226 patients). P-values were calculated using the one-sided Mann-Whitney U test. In violin plots, the central mark is the median.
Source data
The violin plots depict the distribution of Pearson correlation for the top 1,000 genes, and the bar charts indicate the number of genes exhibiting Pearson correlations between the predicted and measured expression values across the cohort samples above 0.4. The results were obtained from model selection strategy (gray) and ensemble learning strategy (light blue). P-values were calculated using the one-sided Mann-Whitney U test. In violin plots, the central mark is the median. The number of patients in each cohort is shown in parentheses.
Source data
Supplementary Table 1. Number of slides, number of patients and number of selected tiles for each cohort. Supplementary Table 2. Details of the five datasets analyzed using ENLIGHT–DeepPT. Supplementary Table 3. AUC and accuracy for the five datasets analyzed using ENLIGHT–DeepPT.
Statistical source data.
Statistical source data.
Statistical source data.
Statistical source data.
Statistical source data.
Statistical source data.
Statistical source data.
Statistical source data.
Statistical source data.
Statistical source data.
Reprints and permissions
Hoang, DT., Dinstag, G., Shulman, E.D. et al. A deep-learning framework to predict cancer treatment response from histopathology images through imputed transcriptomics. Nat Cancer (2024). https://doi.org/10.1038/s43018-024-00793-2
Download citation
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s43018-024-00793-2
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Advertisement
Nature Cancer (Nat Cancer) ISSN 2662-1347 (online)
© 2024 Springer Nature Limited
Sign up for the Nature Briefing: Cancer newsletter — what matters in cancer research, free to your inbox weekly.

source

Leave a Comment