Skip to main content
SpringerLink
Log in

Rock segmentation visual system for assisting driving in TBM construction

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

The tunnel boring machine (TBM) is a key equipment for excavating long-range tunnels. It is a complex system and hard to be controlled well in practice. In this paper, we propose the rock segmentation visual system to assist TBM driving. Through the system, online size distribution of excavated rocks is automatically analysed and sent back to TBM driver, from which many statistical information can be gathered. The system’s core algorithm is based on semantic segmentation, and the rock detection task is viewed as a rock/background pixel-wise classification problem. Accordingly, the Rock Segmentation Dataset is made with specific annotation strategies, and the goal of the dataset is to pick out large rocks in the images. Many networks are evaluated quantitatively on it, and we select the best suited one. We design two parallel networks to extract rock object and contour mask, such that the connected rock areas in object mask can be split with a mask fusion algorithm. Further network modification is made to boost inference speed that meets the requirement of system design. Experimental results show that the system can effectively detect large rock particles in the images and make necessary statistical analysis. Specifically, the segmentation accuracy achieves 68.3% mIoU, and the inference speed achieves 19.4 FPS under image resolution of \(1600\times 1200\) on one NVIDIA Titan XP GPU. From the viewpoint of statistical analysis, 43.5% rock size IoU and 14.7% error rate of mean rock size are obtained, which is acceptable from the viewpoint of real applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. https://github.com/xuezhen2018/Rock-segmentation-visual-system.

References

  1. Anagnostou, G., Kovári, K.: Face stability conditions with earth-pressure-balanced shields. Tunn. Undergr. Space Technol. 11(2), 165–173 (1996)

    Article  Google Scholar 

  2. Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: a deep convolutional encoder–decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2481–2495 (2017)

    Article  Google Scholar 

  3. Brostow, G.J., Shotton, J., Fauqueur, J., Cipolla, R.: Segmentation and recognition using structure from motion point clouds. ECCV 1, 44–57 (2008)

    Google Scholar 

  4. Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8(6), 679–698 (1986). https://doi.org/10.1109/TPAMI.1986.4767851

    Article  Google Scholar 

  5. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 834–848 (2017)

    Article  Google Scholar 

  6. Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 (2017)

  7. Cho, J.W., Jeon, S., Yu, S.H., Chang, S.H.: Optimum spacing of TBM disc cutters: a numerical simulation using the three-dimensional dynamic fracturing method. Tunn. Undergr. Space Technol. 25(3), 230–244 (2010)

    Article  Google Scholar 

  8. Chollet, F.: Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251–1258 (2017)

  9. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding, pp. 3213–3223 (2016). https://doi.org/10.1109/CVPR.2016.350

  10. Erben, H.: Real-time material analysis and development of a collaboration and trading platform for mineral resources from underground construction projects (2016)

  11. Everingham, M., Gool, L.V., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010)

    Article  Google Scholar 

  12. Farrokh, E., Rostami, J.: Correlation of tunnel convergence with tbm operational parameters and chip size in the Ghomroud tunnel, Iran. Tunn. Undergr. Space Technol. 23(6), 700–710 (2008)

    Article  Google Scholar 

  13. Gong, Q., Zhou, X., Liu, Y., Han, B., Yin, L.: Development of a real-time muck analysis system for assistant intelligence tbm tunnelling. Tunn. Undergr. Space Technol. 107, 103655 (2021)

    Article  Google Scholar 

  14. Gong, Q.M., Zhao, J., Jiao, Y.Y.: Numerical modeling of the effects of joint orientation on rock fragmentation by tbm cutters. Tunn. Undergr. Space Technol. 20(2), 183–191 (2005)

    Article  Google Scholar 

  15. Guyot, O., Monredon, T., Larosa, D., Broussaud, A.: Visiorock, an integrated vision technology for advanced control of aggregate circuits. Miner. Eng. 17(11–12), 1227–1235 (2004)

    Article  Google Scholar 

  16. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp. 2961–2969 (2017)

  17. He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(9), 1904–1916 (2015)

    Article  Google Scholar 

  18. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  19. Hongxin, W., Deming, F.: Theoretical and test studies on balance control of epb shields. China Civ. Eng. J. 40(5), 61–68 (2007)

    Google Scholar 

  20. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)

  21. Kirillov, A., He, K., Girshick, R., Rother, C., Dollár, P.: Panoptic segmentation (2019)

  22. Krähenbühl, P., Koltun, V.: Efficient inference in fully connected crfs with gaussian edge potentials. In: Advances in Neural Information Processing Systems, pp. 109–117 (2011)

  23. Li, G., Yu, Y.: Deep contrast learning for salient object detection. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

  24. Lin, G., Milan, A., Shen, C., Reid, I.: Refinenet: multi-path refinement networks for high-resolution semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1925–1934 (2017)

  25. Liu, W., Rabinovich, A., Berg, A.C.: Parsenet: looking wider to see better. arXiv preprint arXiv:1506.04579 (2015)

  26. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)

  27. Nurzynska, K., Iwaszenko, S.: Application of texture features and machine learning methods to grain segmentation in rock material images. Image Anal. Stereol. 39(2), 73–90 (2020)

    MathSciNet  MATH  Google Scholar 

  28. Outal, S., Beucher, S.: Controlling the ultimate opening residues for a robust delineation of fragmetned rocks. In: The 10th European Congress of Stereology and Image Analysis, Milan (2009)

  29. Outal, S., Jeulin, D., Schleifer, J.: A new method for estimating the 3d size-distribution curve of fragmented rocks out of 2d images. Image Anal. Stereol. (2011)

  30. Paszke, A., Chaurasia, A., Kim, S., Culurciello, E.: Enet: a deep neural network architecture for real-time semantic segmentation. arXiv preprint arXiv:1606.02147 (2016)

  31. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch (2017)

  32. Peng, C., Zhang, X., Yu, G., Luo, G., Sun, J.: Large kernel matters-improve semantic segmentation by global convolutional network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4353–4361 (2017)

  33. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)

  34. Rispoli, A., Ferrero, A.M., Cardu, M., Farinetti, A.: Determining the particle size of debris from a tunnel boring machine through photographic analysis and comparison between excavation performance and rock mass properties. Rock Mech. Rock Eng. 50(10), 2805–2816 (2017)

    Article  Google Scholar 

  35. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241. Springer (2015)

  36. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018)

  37. Sato, T., Takeda, K., Nagamatsu, T., Wakayama, T., Kimura, I., Shinbo, T.: Automatic signal processing of front monitor radar for tunneling machines. IEEE Trans. Geosci. Remote Sens. 35(2), 359 (1997)

    Article  Google Scholar 

  38. Shao, C., Liao, J., Li, X., Su, H.: An adaptive robust control for hard rock tunnel boring machine cutterhead driving system. In: ASME 2015 Dynamic Systems and Control Conference, pp. V003T48A001–V003T48A001. American Society of Mechanical Engineers (2015)

  39. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  40. Smith, B.: Improvements in blast fragmentation using measurement while drilling parameters. Fragblast 6(3/4), 301–310 (2002)

    Article  Google Scholar 

  41. Wang, L., Wei, S., Long, Y., Xu, Y.: Reliability-based performance optimization of tunnel boring machine considering geological uncertainties. IEEE Access 6(99), 19086–19098 (2018)

    Article  Google Scholar 

  42. Xue, Z., Jia, L., Sun, W., Lin, F., Liu, Z., Mao, W.: Multi mask learning of stone segmentation for auto-monitoring system in tbm construction. In: 38th Chinese Control Conference (CCC), pp. 8733–8738 (2019)

  43. Yang, H., Shi, H., Gong, G., Hu, G.: Earth pressure balance control for EPB shield. Sci. China Ser. E: Technol. Sci. 52(10), 2840–2848 (2009)

    Article  Google Scholar 

  44. Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., Sang, N.: Bisenet: bilateral segmentation network for real-time semantic segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 325–341 (2018)

  45. Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., Sang, N.: Learning a discriminative feature network for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1857–1866 (2018)

  46. Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2881–2890 (2017)

  47. Zhao, J., Shi, M., Hu, G., Song, X., Zhang, C., Tao, D., Wu, W.: A data-driven framework for tunnel geological-type prediction based on tbm operating data. IEEE Access (2019). https://doi.org/10.1109/ACCESS.2019.2917756

    Article  Google Scholar 

  48. Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., Torr, P.H.: Conditional random fields as recurrent neural networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1529–1537 (2015)

  49. Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 633–641 (2017)

Download references

Acknowledgements

This work is partially supported by the National Natural Science Foundation of China (Grant Nos. 61633019, No. 61873233), the National Key R&D Program of China (Grant No. 2017YFB1300403) and the Fundamental Research Funds for the Central Universities.

Author information

Authors and Affiliations

  1. State Key Laboratory of Industrial Control Technology, Institute of Cyber-Systems and Control, Zhejiang University, No. 38 Zheda Road, Hangzhou, China

    Zhenfeng Xue, Liang Chen, Zhitao Liu & Weijie Mao

  2. China Railway Engineering Equipment Group Tunnel Equipment Manufacturing Co., LTD, Zhengzhou, China

    Fulong Lin

Authors
  1. Zhenfeng Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhitao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fulong Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Weijie Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Weijie Mao.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xue, Z., Chen, L., Liu, Z. et al. Rock segmentation visual system for assisting driving in TBM construction. Machine Vision and Applications 32, 77 (2021). https://doi.org/10.1007/s00138-021-01203-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-021-01203-8

Keywords

Search

Navigation