Skip to main content
SpringerLink
Log in

Multi-scale generative adversarial network for improved evaluation of cell–cell interactions observed in organ-on-chip experiments

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Organs On a Chip (OOCs) represent a sophisticated approach for exploring biological mechanisms and developing therapeutic agents. In conjunction with high-quality time-lapse microscopy (TLM), OOCs allow for the visualization of reconstituted complex biological processes, such as multi-cell-type migration and cell–cell interactions. In this context, increasing the frame rate is desirable to reconstruct accurately cell-interaction dynamics. However, a trade-off between high resolution and carried information content is required to reduce the overall data volume. Moreover, high frame rates increase photobleaching and phototoxicity. As a possible solution for these problems, we report a new hybrid-imaging paradigm based on the integration of OOC/TLMs with a Multi-scale Generative Adversarial Network (GAN) predicting interleaved video frames with the aim to provide high-throughput videos. We tested the performance of the predictive capability of GAN on synthetic videos, as well as on real OOC experiments dealing with tumor–immune cell interactions. The proposed approach offers the possibility to acquire a reduced number of high-quality TLM images without any major loss of information on the phenomena under investigation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Sackmann EK, Fulton AL, Beebe DJ (2014) The present and future role of microfluidics in biomedical research. Nature 507:181–189. https://doi.org/10.1038/nature13118

    Article  Google Scholar 

  2. Reardon S (2015) “Organs-on-chips” go mainstream. Nature 523:266. https://doi.org/10.1038/523266a

    Article  Google Scholar 

  3. Parlato S, De Ninno A, Molfetta R et al (2017) 3D Microfluidic model for evaluating immunotherapy efficacy by tracking dendritic cell behaviour toward tumor cells. Sci Rep 7:1–16. https://doi.org/10.1038/s41598-017-01013-x

    Article  Google Scholar 

  4. Biselli E, Agliari E, Barra A et al (2017) Organs on chip approach: a tool to evaluate cancer-immune cells interactions. Sci Rep 7:1–12. https://doi.org/10.1038/s41598-017-13070-3

    Article  Google Scholar 

  5. Comes MC, Casti P, Mencattini A et al (2019) The influence of spatial and temporal resolutions on the analysis of cell-cell interaction: a systematic study for time-lapse microscopy applications. Sci Rep 9:1–11. https://doi.org/10.1038/s41598-019-42475-5

    Article  Google Scholar 

  6. Beltman JB, Henrickson SE, von Andrian UH et al (2009) Towards estimating the true duration of dendritic cell interactions with T cells. J Immunol Methods 347:54–69. https://doi.org/10.1016/j.jim.2009.05.013

    Article  Google Scholar 

  7. Beltman JB, Marée AFM, De Boer RJ (2009) Analysing immune cell migration. Nat Rev Immunol 9:789–798. https://doi.org/10.1038/nri2638

    Article  Google Scholar 

  8. Harrison JU, Baker RE (2018) The impact of temporal sampling resolution on parameter inference for biological transport models. PLoS Comput Biol 14:1–30. https://doi.org/10.1371/journal.pcbi.1006235

    Article  Google Scholar 

  9. Li S, Karatzoglou A, Gentile C (2016) Collaborative filtering bandits. SIGIR 2016. In: Proceedings of the 39th international ACM SIGIR conference on research and development in information retrieval, pp 539–548. https://doi.org/10.1145/2911451.2911548

  10. Korda N, Szorenyi B, Li S (2016) Distributed clustering of linear bandits in peer to peer networks. In: 33rd International conference on machine learning (ICML 2016) 3:1966–1980

  11. Kar P, Li S, Narasimhan H, et al (2016) Online optimization methods for the quantification problem. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1625–1634

  12. Li S (2016) The art of clustering bandits. PhD thesis. Università degli Studi dell'Insubria

  13. Li S, Chen W, Li S, Leung KS (2019) Improved algorithm on online clustering of bandits. In: IJCAI 2019: international joint conference on artificial intelligence, 2923–2929. https://doi.org/10.24963/ijcai.2019/405

  14. Gentile C, Li S, Kar P, et al. (2017) On context-dependent clustering of bandits. In: 34th International conference on machine learning (ICML 2017), 3:2043–2052

  15. Agliari E, Biselli E, De Ninno A et al (2014) Cancer-driven dynamics of immune cells in a microfluidic environment. Sci Rep 4:11–13. https://doi.org/10.1038/srep06639

    Article  Google Scholar 

  16. Businaro L, De Ninno A, Schiavoni G et al (2013) Cross talk between cancer and immune cells: Exploring complex dynamics in a microfluidic environment. Lab Chip 13:229–239. https://doi.org/10.1039/c2lc40887b

    Article  Google Scholar 

  17. Vacchelli E, Ma Y, Baracco EE et al (2015) Chemotherapy-induced antitumor immunity requires formylpeptide receptor1. Science 350:972–978. https://doi.org/10.1126/science.aad0779o

    Article  Google Scholar 

  18. Montiel D, Cang H, Yang H (2006) Quantitative characterization of changes in dynamical behavior for single-particle tracking studies. J Phys Chem B 110:19763–19770. https://doi.org/10.1021/jp062024j

    Article  Google Scholar 

  19. Dosset P, Rassam P, Fernandez L et al (2016) Automatic detection of diffusion modes within biological membranes using back-propagation neural network. BMC Bioinf 17:1–12. https://doi.org/10.1186/s12859-016-1064-z

    Article  Google Scholar 

  20. Waldchen S, Lehmann J, Klein T et al (2015) Light-induced cell damage in live-cell super-resolution microscopy. Sci Rep 5:1–12. https://doi.org/10.1038/srep15348

    Article  Google Scholar 

  21. Lee H, Yune S, Mansouri M et al (2019) An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets. Nat Biomed Eng 3:173–182. https://doi.org/10.1038/s41551-018-0324-9

    Article  Google Scholar 

  22. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444. https://doi.org/10.1038/nature14539

    Article  Google Scholar 

  23. Wang Z, Li J, Enoh M (2019) Removing ring artifacts in CBCT images via generative adversarial networks with unidirectional relative total variation loss. Neural Comput Appl 31:5147–5158. https://doi.org/10.1007/s00521-018-04007-6

    Article  Google Scholar 

  24. Suresh S, Mohan S (2020) ROI-based feature learning for efficient true positive prediction using convolutional neural network for lung cancer diagnosis. Neural Comput Appl. https://doi.org/10.1007/s00521-020-04787-w

    Article  Google Scholar 

  25. Radford A, Metz L, Chintala S (2016) Unsupervised representation learning with deep convolutional generative adversarial networks. In: 4th International conference on learning representations, ICLR 2016, 1–16

  26. Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2017 January, 5967–5976. https://doi.org/10.1109/CVPR.2017.632

  27. Zhang H, Xu T, Li H, et al. (2017) StackGAN: text to photo-realistic image synthesis with stacked generative adversarial networks. In: Proceedings of IEEE international conference on computer vision, ICCV 2017 October, pp 5908–5916. https://doi.org/10.1109/ICCV.2017.629

  28. Nie D, Trullo R, Lian J et al (2018) Medical image synthesis with deep convolutional adversarial networks. IEEE Trans Biomed Eng 65:2720–2730. https://doi.org/10.1109/TBME.2018.2814538

    Article  Google Scholar 

  29. Xue Y, Xu T, Zhang H, et al. (2017) SegAN: adversarial network with multi-scale L1 loss for medical image segmentation. pp 1–9. https://doi.org/10.1016/B978-012264841-0/50037-8

  30. Mardani M, Gong E, Cheng JY et al (2019) Deep generative adversarial neural networks for compressive sensing MRI. IEEE Trans Med Imaging 38:167–179. https://doi.org/10.1109/TMI.2018.2858752

    Article  Google Scholar 

  31. Mathieu M, Couprie C, LeCun Y (2016) Deep multi-scale video prediction beyond mean square error. In: 4th International conference on learning representations, ICLR 2016—conference track proceedings, pp 1–14

  32. Vondrick C, Pirsiavash H, Torralba A (2016) Generating videos with scene dynamics. Adv Neural Inf Process Syst 613–621

  33. Lee AX, Zhang R, Ebert F, et al. (2018) Stochastic adversarial video prediction. arXiv Preprint arXiv180401523

  34. Xiong W, Luo W, Ma L, et al. (2018) Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 2364–2373. https://doi.org/10.1109/CVPR.2018.00251

  35. Liang X, Lee L, Dai W, Xing EP (2017) Dual motion GAN for future-flow embedded video prediction. In: Proceedings of IEEE international conference on computer vision, pp 1762–1770. https://doi.org/10.1109/ICCV.2017.194

  36. Liu Z, Yeh RA, Tang X, et al. (2017) Video frame synthesis using deep voxel flow. In: Proceedings of IEEE international conference on computer vision 2017 October pp 4473–4481. https://doi.org/10.1109/ICCV.2017.478

  37. Osokin A, Chessel A, Salas REC, Vaggi F (2017) GANs for biological image synthesis. In: Proceedings of IEEE international conference on computer vision 2017 October, pp 2252–2261. https://doi.org/10.1109/ICCV.2017.245

  38. Wang H, Rivenson Y, Jin Y et al (2019) Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat Methods 16:103–110. https://doi.org/10.1038/s41592-018-0239-0

    Article  Google Scholar 

  39. Gupta A, Zou J (2019) Feedback GAN for DNA optimizes protein functions. Nat Mach Intell 1:105–111. https://doi.org/10.1038/s42256-019-0017-4

    Article  Google Scholar 

  40. Ghasemi M, Dehpour AR (2009) Journal of medical ethics and history of medicine ethical considerations in animal studies. J Med Ethics Hist Med 2:2–4

    Google Scholar 

  41. Baumans V (2004) Use of animals in experimental research: an ethical dilemma? Gene Ther 11:S64–S66. https://doi.org/10.1038/sj.gt.3302371

    Article  Google Scholar 

  42. Nguyen M, De Ninno A, Mencattini A et al (2018) Dissecting effects of anti-cancer drugs and cancer-associated fibroblasts by on-chip reconstitution of immunocompetent tumor microenvironments. Cell Rep 25:3884–3893.e3. https://doi.org/10.1016/j.celrep.2018.12.015

    Article  Google Scholar 

  43. Berthold KP, Horn BGS (1981) Determining optical flow. Artif Intell 17:185–203

    Article  Google Scholar 

  44. Revaud J, Weinzaepfel P, Harchaoui Z, Schmid C (2015) EpicFlow: Edge-Preserving Interpolation of Correspondences for Optical Flow. Proc IEEE Conf Comput Vis pattern Recognit

  45. P Weinzaepfel J Revaud Z Harchaoui C Schmid (2013) DeepFlow: large displacement optical flow with deep matching. In: Proceedings of IEEE international conference on computer vision, pp 1385–1392. https://doi.org/10.1109/ICCV.2013.175

  46. Goodfellow IJ, Pouget-Abadie J, Mehdi M, et al (2014) Generative adversarial nets. Adv Neural Inf Process Syst pp 2672–2680

  47. Goodfellow I (2016) NIPS 2016 tutorial: generative adversarial networks. arXiv preprint arXiv:1701.00160

  48. Tulyakov S, Liu MY, Yang X, Kautz J (2018) MoCoGAN: decomposing motion and content for video generation. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp1526–1535. https://doi.org/10.1109/CVPR.2018.00165

  49. Denton E, Szlam A, Fergus R (2015) Deep generative image models using a Laplacian pyramid of adversarial networks emily. PP 1–9

  50. Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. In: Proceedings of IEEE

  51. Gonzalez RC, Woods RE, Eddins SL (2004) Digital image processing using Matlab. Pearson Education India

  52. Kingma DP, Ba JL (2015) Adam: a method for stochastic optimization. In: 3rd International Conference on Learning Representations ICLR 2015—Conference Track Proceedings, pp 1–15

  53. Davies ER (2004) Machine vision: theory, algorithms, practicalities. Elsevier

  54. Munkres J (1957) Algorithms for the assignment and transportation problems. J Soc Ind Appl Math 5:32–38

    Article  MathSciNet  Google Scholar 

  55. Ernst D, Köhler J, Weiss M (2014) Probing the type of anomalous diffusion with single-particle tracking. Phys Chem Chem Phys 16:7686–7691. https://doi.org/10.1039/c4cp00292j

    Article  Google Scholar 

  56. Lawrence I, Kuei L (1989) A Concordance correlation coefficient to evaluate reproducibility. Biomatrics 45:255–268

    Article  Google Scholar 

  57. Lopes RH (2011) Kolmogorov–Smirnov test. International encyclopedia of statistical science. Springer, Berlin, Heidelberg, pp 718–720

    Chapter  Google Scholar 

  58. Selva Castelló J (2018) A comprehensive survey on deep future frame video prediction. Master's thesis. Universitat Politècnica de Catalunya

Download references

Acknowledgements

GK is supported by the Ligue contre le Cancer (équipe labellisée); Agence National de la Recherche (ANR) – Projets blancs; ANR under the frame of E-Rare-2, the ERA-Net for Research on Rare Diseases; Association pour la recherche sur le cancer (ARC); Cancéropôle Ile-de-France; Chancelerie des universités de Paris (Legs Poix), Fondation pour la Recherche Médicale (FRM); a donation by Elior; European Research Area Network on Cardiovascular Diseases (ERA-CVD, MINOTAUR); Gustave Roussy Odyssea, the European Union Horizon 2020 Project Oncobiome; Fondation Carrefour; High-end Foreign Expert Program in China (GDW20171100085 and GDW20181100051), Institut National du Cancer (INCa); Inserm (HTE); Institut Universitaire de France; LeDucq Foundation; the LabEx Immuno-Oncology; the RHU Torino Lumière; the Seerave Foundation; the SIRIC Stratified Oncology Cell DNA Repair and Tumor Immune Elimination (SOCRATE); and the SIRIC Cancer Research and Personalized Medicine (CARPEM).

Author information

Authors and Affiliations

  1. Department of Electronic Engineering, University of Rome Tor Vergata, via del Politecnico 1, 00133, Rome, Italy

    M. C. Comes, J. Filippi, A. Mencattini, P. Casti, D. Di Giuseppe, M. D’Orazio, C. Di Natale & E. Martinelli

  2. Interdisciplinary Center for Advanced Studies on Lab-on-Chip and Organ-on-Chip Applications (ICLOC), University of Rome Tor Vergata, 00133, Rome, Italy

    M. C. Comes, J. Filippi, A. Mencattini, P. Casti, D. Di Giuseppe, M. D’Orazio & E. Martinelli

  3. Equipe labellisée par la Ligue Contre le Cancer, INSERM U1138, Centre de Recherche des Cordeliers, Université Paris Descartes, Université Sorbonne Paris Cité, Université Paris Diderot, Sorbonne Université, Paris, France

    G. Cerrato, A. Sauvat, E. Vacchelli & G. Kroemer

  4. Metabolomics and Cell Biology Platforms, Institut Gustave Roussy, Villejuif, France

    G. Cerrato, A. Sauvat, E. Vacchelli & G. Kroemer

  5. Pôle de Biologie, Hôpital Européen Georges Pompidou, AP-HP, Paris, France

    G. Kroemer

  6. Suzhou Institute for Systems Biology, Chinese Academy of Sciences, Suzhou, China

    G. Kroemer

  7. Department of Women’s and Children’s Health, Karolinska Institute, Karolinska University Hospital, Stockholm, Sweden

    G. Kroemer

  8. Institute for Photonics and Nanotechnologies, Italian National Research Council, Via Cineto Romano 42, 00156, Rome, Italy

    A. De Ninno & L. Businaro

  9. Department of Oncology and Molecular Medicine, Istituto Superiore di Sanità, Viale Regina Elena 299, 00161, Rome, Italy

    F. Mattei & G. Schiavoni

Authors
  1. M. C. Comes
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. J. Filippi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. A. Mencattini
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. P. Casti
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. G. Cerrato
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. A. Sauvat
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. E. Vacchelli
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. A. De Ninno
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. D. Di Giuseppe
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. M. D’Orazio
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. F. Mattei
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. G. Schiavoni
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. L. Businaro
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. C. Di Natale
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. G. Kroemer
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. E. Martinelli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to E. Martinelli.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Fig. 1

Evaluation performance on the single scenario, the negative case of the organ-on-chip experiments. In the upper panel the real histogram of the mean interaction time in comparison with the generated counterpart after subsampling at 0.1667 frames/min. In the lower panel the real histogram of the mean interaction time in comparison with the subsampled counterpart (0.1667 frames/min). P-values for the K–S are also specified. (TIF 110 kb)

Supplementary Fig. 2

Comparison between the generated and subsampled histograms of the mean interaction time for the negative and positive cases on the panels on the top and on the bottom, respectively. Videos are subsampled at 0.1667 frames/min. P-values for the K–S test are also specified. (TIF 109 kb)

Supplementary Video 1 Artificial movie of one out of 100 for the simulated control case. (AVI 21103 kb)

Supplementary Video 2 Subsampled counterpart of Supplementary Video 1 obtained by subsampling at temporal resolution of 0.25 frames/min (every 4 minutes). The red word Start indicates that the temporal subsampling starts from the frame 50, when all immune cells are appeared in the field of view. (AVI 16880 kb)

Supplementary Video 3 Hybrid counterpart of Supplementary Video 1 obtained by alternating sequences of GAN generated frames with original theoretical frames. (AVI 21103 kb)

Supplementary Video 4 Example of extracted Region of Interest (ROI) for the negative case of the Tumor-Immune On Chip real experiment. (AVI 21103 kb)

Supplementary Video 5 Hybrid counterpart of Supplementary Video 3 obtained by alternating sequences of GAN generated frames with original theoretical frames.(AVI 2503 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Comes, M.C., Filippi, J., Mencattini, A. et al. Multi-scale generative adversarial network for improved evaluation of cell–cell interactions observed in organ-on-chip experiments. Neural Comput & Applic 33, 3671–3689 (2021). https://doi.org/10.1007/s00521-020-05226-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-020-05226-6

Keywords

Search

Navigation