Skip to main content
Springer Nature Link
Log in

Self-supervised Orientation-Guided Deep Network for Segmentation of Carbon Nanotubes in SEM Imagery

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 Workshops (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13808))

Included in the following conference series:

Abstract

Electron microscopy images of carbon nanotube (CNT) forests are difficult to segment due to the long and thin nature of the CNTs; density of the CNT forests resulting in CNTs touching, crossing, and occluding each other; and low signal-to-noise ratio electron microscopy imagery. In addition, due to image complexity, it is not feasible to prepare training segmentation masks. In this paper, we propose CNTSegNet, a dual loss, orientation-guided, self-supervised, deep learning network for CNT forest segmentation in scanning electron microscopy (SEM) images. Our training labels consist of weak segmentation labels produced by intensity thresholding of the raw SEM images and self labels produced by estimating orientation distribution of CNTs in these raw images. The proposed network extends a U-net-like encoder-decoder architecture with a novel two-component loss function. The first component is dice loss computed between the predicted segmentation maps and the weak segmentation labels. The second component is mean squared error (MSE) loss measuring the difference between the orientation histogram of the predicted segmentation map and the original raw image. Weighted sum of these two loss functions is used to train the proposed CNTSegNet network. The dice loss forces the network to perform background-foreground segmentation using local intensity features. The MSE loss guides the network with global orientation features and leads to refined segmentation results. The proposed system needs only a few-shot dataset for training. Thanks to it’s self-supervised nature, it can easily be adapted to new datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. OpenCV: K-Means Clustering in OpenCV (Aug 2022). https://docs.opencv.org/3.4/d1/d5c/tutorial_py_kmeans_opencv.html. (Accessed 12 Aug 2022)

  2. Thresholding—skimage v0.13.1 docs (Jun 2022). http://devdoc.net/python/scikit-image-doc-0.13.1/auto_examples/xx_applications/plot_thresholding.html#id4. (Accessed 28 Jun 2022)

  3. Abadi, P.P.S.S., et al.: Reversible tailoring of mechanical properties of carbon nanotube forests by immersing in solvents. Carbon 69, 178–187 (2014)

    Article  Google Scholar 

  4. Aguilar, C., Comer, M., Hanhan, I., Agyei, R., Sangid, M.: 3D fiber segmentation with deep center regression and geometric clustering. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 3741–3749. IEEE (June 2021). https://doi.org/10.1109/CVPRW53098.2021.00415

  5. Bedewy, M., Meshot, E.R., Guo, H., Verploegen, E.A., Lu, W., Hart, A.J.: Collective mechanism for the evolution and self-termination of vertically aligned carbon nanotube growth. J. Phys. Chem. C 113(48), 20576–20582 (2009)

    Article  Google Scholar 

  6. Brandley, E., Greenhalgh, E.S., Shaffer, M.S.P., Li, Q.: Mapping carbon nanotube orientation by fast fourier transform of scanning electron micrographs. Carbon 137, 78–87 (2018). https://doi.org/10.1016/j.carbon.2018.04.063

    Article  Google Scholar 

  7. Brieland-Shoultz, A., Tawfick, S., Park, S.J., Bedewy, M., Maschmann, M.R., Baur, J.W., Hart, A.J.: Scaling the stiffness, strength, and toughness of ceramic-coated nanotube foams into the structural regime. Adv. Func. Mater. 24(36), 5728–5735 (2014)

    Article  Google Scholar 

  8. Cao, A., Dickrell, P.L., Sawyer, W.G., Ghasemi-Nejhad, M.N., Ajayan, P.M.: Super-compressible foamlike carbon nanotube films. Science 310(5752), 1307–1310 (2005)

    Article  Google Scholar 

  9. Carter, R., Davis, B., Oakes, L., Maschmann, M.R., Pint, C.L.: A high areal capacity lithium-sulfur battery cathode prepared by site-selective vapor infiltration of hierarchical carbon nanotube arrays. Nanoscale 9(39), 15018–15026 (2017)

    Article  Google Scholar 

  10. Cola, B.A., Xu, J., Cheng, C., Xu, X., Fisher, T.S., Hu, H.: Photoacoustic characterization of carbon nanotube array thermal interfaces. J. Appl. Phys. 101(5), 054313 (2007)

    Article  Google Scholar 

  11. Cola, B.A., Xu, X., Fisher, T.S.: Increased real contact in thermal interfaces: A carbon nanotube/foil material. Appl. Phys. Lett. 90(9), 093513 (2007)

    Article  Google Scholar 

  12. Davis, B.F., Yan, X., Muralidharan, N., Oakes, L., Pint, C.L., Maschmann, M.R.: Electrically conductive hierarchical carbon nanotube networks with tunable mechanical response. ACS Appli. Mater. Interfaces 8(41), 28004–28011 (2016)

    Article  Google Scholar 

  13. De Volder, M.F.L., Tawfick, S.H., Baughman, R.H., Hart, A.J.: Carbon nanotubes: present and future commercial applications. Science 339(6119), 535–539 (2013). https://doi.org/10.1126/science.1222453

    Article  Google Scholar 

  14. Gommes, C., et al.: Image analysis characterization of multi-walled carbon nanotubes. Carbon 41(13), 2561–2572 (2003). https://doi.org/10.1016/S0008-6223(03)00375-0

    Article  Google Scholar 

  15. Hajilounezhad, T., Bao, R., Palaniappan, K., Bunyak, F., Calyam, P., Maschmann, M.R.: Predicting carbon nanotube forest attributes and mechanical properties using simulated images and deep learning. NPJ Comput. Mater. 7(1), 1–11 (2021)

    Article  Google Scholar 

  16. Hajilounezhad, T., et al.: Exploration of carbon nanotube forest synthesis-structure relationships using physics-based simulation and machine learning. In: IEEE Applied Imagery Pattern Recognition Workshop (AIPR), pp. 1–8 (2019)

    Google Scholar 

  17. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

  18. Hines, R., Hajilounezhad, T., Love-Baker, C., Koerner, G., Maschmann, M.R.: Growth and mechanics of heterogeneous, 3d carbon nanotube forest microstructures formed by sequential selective-area synthesis. ACS Appli. Mater. Interfaces 12(15), 17893–17900 (2020)

    Article  Google Scholar 

  19. Iakubovskii, P.: Segmentation models pytorch. www.github.com/qubvel/segmentation_models.pytorch (2019)

  20. Iijima, S.: Helical microtubules of graphitic carbon. Nature 354(6348), 56–58 (1991). https://doi.org/10.1038/354056a0

    Article  Google Scholar 

  21. Iijima, S.: Carbon nanotubes: past, present, and future. Phys. B 323(1), 1–5 (2002). https://doi.org/10.1016/S0921-4526(02)00869-4

    Article  Google Scholar 

  22. Jing, L., Tian, Y.: Self-supervised visual feature learning with deep neural networks: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 43(11), 4037–4058 (2020). https://doi.org/10.1109/TPAMI.2020.2992393

    Article  Google Scholar 

  23. Jung, Y., Cho, Y.S., Lee, J.W., Oh, J.Y., Park, C.R.: How can we make carbon nanotube yarn stronger? Compos. Sci. Technol. 166, 95–108 (2018)

    Article  Google Scholar 

  24. Kaniyoor, A., Gspann, T.S., Mizen, J.E., Elliott, J.A.: Quantifying alignment in carbon nanotube yarns and similar two-dimensional anisotropic systems. J. Appl. Polym. Sci. 138(37), 50939 (2021). https://doi.org/10.1002/app.50939

    Article  Google Scholar 

  25. Kim, W., Kanezaki, A., Tanaka, M.: Unsupervised learning of image segmentation based on differentiable feature clustering. IEEE Trans. Image Process. 29, 8055–8068 (2020). https://doi.org/10.1109/TIP.2020.3011269

    Article  MATH  Google Scholar 

  26. Koerner, G., Surya, R., Palaniappan, K., Calyam, P., Bunyak, F., Maschmann, M.R.: In-situ scanning electron microscope chemical vapor deposition as a platform for nanomanufacturing insights. In: ASME International Mechanical Engineering Congress and Exposition. vol. 85567, p. V02BT02A052 (2021)

    Google Scholar 

  27. Konopczyński, T., Kröger, T., Zheng, L., Hesser, J.: Instance Segmentation of Fibers from Low Resolution CT Scans via 3D Deep Embedding Learning. arXiv (Jan 2019). 10.48550/arXiv. 1901.01034

    Google Scholar 

  28. Maschmann, M.R.: Integrated simulation of active carbon nanotube forest growth and mechanical compression. Carbon 86, 26–37 (2015)

    Article  Google Scholar 

  29. Maschmann, M.R., Dickinson, B., Ehlert, G.J., Baur, J.W.: Force sensitive carbon nanotube arrays for biologically inspired airflow sensing. Smart Mater. Struct. 21(9), 094024 (2012)

    Article  Google Scholar 

  30. Maschmann, M.R., et al.: Bioinspired carbon nanotube fuzzy fiber hair sensor for air-flow detection. Adv. Mater. 26(20), 3230–3234 (2014)

    Article  Google Scholar 

  31. Maschmann, M.R.: Visualizing strain evolution and coordinated buckling within cnt arrays by in situ digital image correlation. Adv. Func. Mater. 22(22), 4686–4695 (2012)

    Article  Google Scholar 

  32. Maschmann, M.R., Zhang, Q., Du, F., Dai, L., Baur, J.: Length dependent foam-like mechanical response of axially indented vertically oriented carbon nanotube arrays. Carbon 49(2), 386–397 (2011)

    Article  Google Scholar 

  33. Maschmann, M.R., Zhang, Q., Wheeler, R., Du, F., Dai, L., Baur, J.: In situ sem observation of column-like and foam-like cnt array nanoindentation. ACS Appli. Mater. Interfaces 3(3), 648–653 (2011)

    Article  Google Scholar 

  34. Maurer, C.R., Qi, R., Raghavan, V.: A linear time algorithm for computing exact euclidean distance transforms of binary images in arbitrary dimensions. IEEE Trans. Pattern Anal. Mach. Intell. 25(2), 265–270 (2003)

    Article  Google Scholar 

  35. Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9(1), 62–66 (1979). https://doi.org/10.1109/TSMC.1979.4310076

  36. Park, M., et al.: Effects of a carbon nanotube layer on electrical contact resistance between copper substrates. Nanotechnology 17(9), 2294 (2006)

    Article  Google Scholar 

  37. Pathak, S., et al.: Local relative density modulates failure and strength in vertically aligned carbon nanotubes. ACS Nano 7(10), 8593–8604 (2013)

    Article  Google Scholar 

  38. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  39. Tawfick, S., et al.: Mechanics of capillary forming of aligned carbon nanotube assemblies. Langmuir 29(17), 5190–5198 (2013)

    Article  Google Scholar 

  40. Trujillo, M.C.R., Alarcón, T.E., Dalmau, O.S., Zamudio Ojeda, A.: Segmentation of carbon nanotube images through an artificial neural network. Soft. Comput. 21(3), 611–625 (2016). https://doi.org/10.1007/s00500-016-2426-1

    Article  Google Scholar 

  41. Wortmann, T., Fatikow, S.: Carbon nanotube detection by scanning electron microscopy. In: Proceedings of the Eleventh IAPR Conference on Machine Vision Applications, MVA 2009 (2009)

    Google Scholar 

  42. Zbib, A.A., Mesarovic, S.D., Lilleodden, E.T., McClain, D., Jiao, J., Bahr, D.F.: The coordinated buckling of carbon nanotube turfs under uniform compression. Nanotechnology 19(17), 175704 (2008). https://doi.org/10.1088/0957-4484/19/17/175704

  43. Zhang, M., Atkinson, K.R., Baughman, R.H.: Multifunctional carbon nanotube yarns by downsizing an ancient technology. Science 306(5700), 1358–1361 (2004)

    Article  Google Scholar 

Download references

Acknowledgement

This work was partially supported by the National Science Foundation under award number CMMI-2026847. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Author information

Authors and Affiliations

  1. Department of Electrical Engineering and Computer Science, Columbia, USA

    Nguyen P. Nguyen, Prasad Calyam, Kannappan Palaniappan & Filiz Bunyak

  2. Department of Mechanical and Aerospace Engineering, University of Missouri-Columbia, Columbia, MO, USA

    Ramakrishna Surya & Matthew Maschmann

Authors
  1. Nguyen P. Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ramakrishna Surya
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Matthew Maschmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Prasad Calyam
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kannappan Palaniappan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Filiz Bunyak
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Filiz Bunyak .

Editor information

Editors and Affiliations

  1. IBM Research AI and MIT-IBM Watson AI Lab, Haifa, Israel

    Leonid Karlinsky

  2. Technion – Israel Institute of Technology, Haifa, Israel

    Tomer Michaeli

  3. Kyoto University, Kyoto, Japan

    Ko Nishino

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Nguyen, N.P., Surya, R., Maschmann, M., Calyam, P., Palaniappan, K., Bunyak, F. (2023). Self-supervised Orientation-Guided Deep Network for Segmentation of Carbon Nanotubes in SEM Imagery. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13808. Springer, Cham. https://doi.org/10.1007/978-3-031-25085-9_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25085-9_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25084-2

  • Online ISBN: 978-3-031-25085-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics