Skip to main content
SpringerLink
Log in

ViRFD: a virtual-realistic fused dataset for rock size analysis in TBM construction

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

In TBM (Tunnel Boring Machine) construction process, the rock size analysis system plays an important role in assisting driving. Its core algorithm is based on semantic segmentation, and it brings challenges to dataset acquisition in real applications. To relieve this problem, this paper proposes a virtual-realistic fused dataset, short for ViRFD. The R-part is composed of a realistic dataset from our previous work, and the V-part is simulated by a learning-based method proposed in this paper. Unlike traditional manual methods, we use a virtual engine (Unity) to simulate datasets, since the corresponding ground-truth labels can be automatically extracted by the engine. Specifically, we propose a novel synthetic dataset simulator, named RockSegX. It contains abundant virtual 3D resources to ensure the diversity and fidelity of generated datasets. The main feature of RockSegX lies in its content flexibility, i.e., we are able to control the content of dataset by adjusting the values of several attributes. These attributes are carefully designed for reducing the content difference between V-part and R-part datasets. And we employ a learning-based method to automatically adjust the attributes so that the V-part dataset has the smallest content difference with the R-part. Experimental results show the effectiveness of our method in improving the quality of simulated dataset, and it further boosts the test accuracy for real-world segmentation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Anagnostou G, Kovári K (1996) Face stability conditions with earth-pressure-balanced shields. Tunn Undergr Space Technol 11(2):165–173

    Article  Google Scholar 

  2. Barbosa IB, Cristani M, Caputo B, Rognhaugen A, Theoharis T (2018) Looking beyond appearances: synthetic training data for deep CNNS in re-identification. Comput Vis Image Underst 167:50–62

    Article  Google Scholar 

  3. Brostow GJ, Fauqueur J, Cipolla R (2009) Semantic object classes in video: a high-definition ground truth database. Pattern Recogn Lett 30(2):88–97

    Article  Google Scholar 

  4. Cordts M, Omran M, Ramos S, Rehfeld T, Enzweiler M, Benenson R, Franke U, Roth S, Schiele B (2016) The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3213–3223

  5. Cruz C, Foi A, Katkovnik V, Egiazarian K (2018) Nonlocality-reinforced convolutional neural networks for image denoising. IEEE Signal Process Lett 25(8):1216–1220. https://doi.org/10.1109/LSP.2018.2850222

    Article  Google Scholar 

  6. Devaranjan J, Kar A, Fidler S (2020) Meta-sim2: unsupervised learning of scene structure for synthetic data generation. In: Proceeding of the European conference on computer vision, pp 715–733

  7. Erben H (2016) Real-time material analysis and development of a collaboration and trading platform for mineral resources from underground construction projects. Doctoral thesis in Montanuniversity Leoben

  8. Farrokh E, Rostami J (2008) Correlation of tunnel convergence with tbm operational parameters and chip size in the ghomroud tunnel, iran. Tunn Undergr Space Technol 23(6):700–710

    Article  Google Scholar 

  9. Gaidon A, Wang Q, Cabon Y, Vig E (2016) Virtual worlds as proxy for multi-object tracking analysis. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4340–4349

  10. Geiger A, Lenz P, Stiller C, Urtasun R (2013) Vision meets robotics: the kitti dataset. Int J Robot Res 32(11):1231–1237

    Article  Google Scholar 

  11. Geiger A, Lenz P, Urtasun R (2012) Are we ready for autonomous driving? The kitti vision benchmark suite. In: 2012 IEEE conference on computer vision and pattern recognition, pp 3354–3361. IEEE

  12. Gong Q, Zhou X, Liu Y, Han B, Yin L (2021) Development of a real-time muck analysis system for assistant intelligence tbm tunnelling. Tunn Undergr Space Technol 107:103655

    Article  Google Scholar 

  13. Gretton A, Borgwardt KM, Rasch MJ, Schölkopf B, Smola A (2012) A kernel two-sample test. J Mach Learn Res 13(1):723–773

    MathSciNet  MATH  Google Scholar 

  14. Guyot O, Monredon T, Larosa D, Broussaud A (2004) Visiorock, an integrated vision technology for advanced control of aggregate circuits. Miner Eng 17(11–12):1227–1235

    Article  Google Scholar 

  15. Hattori H, Naresh Boddeti V, Kitani K.M, Kanade T (2015) Learning scene-specific pedestrian detectors without real data. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3819–3827

  16. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778

  17. Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. Adv Neural Inf Process Syst 30

  18. Hongxin W, Deming F (2007) Theoretical and test studies on balance control of epb shields. Chin Civil Eng J 40(5):61–68

    Google Scholar 

  19. Jiang F, Grigorev A, Rho S, Tian Z, Fu Y, Jifara W, Adil K, Liu S (2018) Medical image semantic segmentation based on deep learning. Neural Comput Appl 29:1257–1265

    Article  Google Scholar 

  20. Juliani A, Berges V, Vckay E, Gao Y, Henry H, Mattar M, Lange D (2018) Unity: a general platform for intelligent agents. CoRR arxiv:1809.02627

  21. Kang B, Lee Y, Nguyen TQ (2018) Depth-adaptive deep neural network for semantic segmentation. IEEE Trans Multimedia 20(9):2478–2490. https://doi.org/10.1109/TMM.2018.2798282

    Article  Google Scholar 

  22. Kar A, Prakash A, Liu M.Y, Cameracci E, Yuan J, Rusiniak M, Acuna D, Torralba A, Fidler S (2019) Meta-sim: learning to generate synthetic datasets. In: 2019 IEEE/CVF International conference on computer vision (ICCV), pp 4550–4559

  23. Kipf T.N, Welling M (2016) Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907

  24. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. IEEE Trans Pattern Anal Mach Intell 39(4):640–651

    Google Scholar 

  25. Nie WZ, Jia WW, Li WH, Liu AA, Zhao SC (2021) 3D pose estimation based on reinforce learning for 2D image-based 3D model retrieval. IEEE Trans Multimed 23:1021–1034. https://doi.org/10.1109/TMM.2020.2991532

    Article  Google Scholar 

  26. Nurzynska K, Iwaszenko S (2020) Application of texture features and machine learning methods to grain segmentation in rock material images. Image Anal Stereol 39(2):73–90

  27. Outal S, Beucher S (2009) Controlling the ultimate opening residues for a robust delineation of fragmetned rocks. In: The 10th European Congress of Stereology and Image Analysis, Milan

  28. Outal S, Jeulin D, Schleifer J (2011) A new method for estimating the 3d size-distribution curve of fragmented rocks out of 2d images. Image Anal Stereol 27(2):97–105

  29. Peng C, Zhang X, Yu G, Luo G, Sun J (2017) Large kernel matters–improve semantic segmentation by global convolutional network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4353–4361

  30. Pepik B, Stark M, Gehler P, Schiele B (2012) Teaching 3d geometry to deformable part models. In: 2012 IEEE conference on computer vision and pattern recognition, pp 3362–3369. IEEE

  31. Richter S.R, Vineet V, Roth S, Koltun V (2016) Playing for data: ground truth from computer games. In: European conference on computer vision

  32. Rispoli A, Ferrero AM, Cardu M, Farinetti A (2017) Determining the particle size of debris from a tunnel boring machine through photographic analysis and comparison between excavation performance and rock mass properties. Rock Mech Rock Eng 50(10):2805–2816

    Article  Google Scholar 

  33. Ruiz N, Schulter S, Chandraker M (2019) Learning to simulate. In: International conference on learning representations

  34. Satkin S, Lin J, Hebert M (2012) Data-driven scene understanding from 3d models. In: British machine vision conference, pp 128.1–128.11

  35. Senniappan V, Subramanian J, Papageorgiou EI, Mohan S (2017) Application of fuzzy cognitive maps for crack categorization in columns of reinforced concrete structures. Neural Comput Appl 28:107–117

    Article  Google Scholar 

  36. Shao C, Liao J, Li X, Su H (2015) An adaptive robust control for hard rock tunnel boring machine cutterhead driving system. In: ASME 2015 Dynamic systems and control conference, pp. V003T48A001–V003T48A001. American Society of Mechanical Engineers

  37. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: Proceedings of international conference on learning representations, pp 1–14

  38. Smith B (2002) Improvements in blast fragmentation using measurement while drilling parameters. Fragblast 6(3/4):301–310

    Article  Google Scholar 

  39. Sun X, Zheng L (2019) Dissecting person re-identification from the viewpoint of viewpoint. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 608–617

  40. Tremblay J, Prakash A, Acuna D, Brophy M, Jampani V, Anil C, To T, Cameracci E, Boochoon S, Birchfield S (2018) Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 969–977

  41. Williams RJ (1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach Learn 8(3–4):229–256

    MATH  Google Scholar 

  42. Xue Z, Chen L, Liu Z, Lin F, Mao W (2021) Rock segmentation visual system for assisting driving in tbm construction. Mach Vis Appl 32(4):1–12

  43. Xue Z, Jia L, Sun W, Lin F, Liu Z, Mao W (2019) Multi mask learning of stone segmentation for auto-monitoring system in tbm construction. In: 2019 Chinese Control Conference (CCC), pp 8733–8738. 10.23919/ChiCC.2019.8865323

  44. Xue Z, Mao W, Jiang W (2020) Ehanet: Efficient hybrid attention network towards real-time semantic segmentation. In: 2020 IEEE 6th International Conference on Computer and Communications (ICCC), pp 787–791. 10.1109/ICCC51575.2020.9345050

  45. Xue Z, Mao W, Zheng L (2021) Learning to simulate complex scenes for street scene segmentation. IEEE Transactions on Multimedia p 1. 10.1109/TMM.2021.3062497

  46. Yagiz S (2008) Utilizing rock mass properties for predicting tbm performance in hard rock condition. Tunn Undergr Space Technol 23(3):326–339

    Article  Google Scholar 

  47. Yang H, Shi H, Gong G, Hu G (2009) Earth pressure balance control for epb shield. Sci China Ser E: Technol Sci 52(10):2840–2848

    Article  Google Scholar 

  48. Yao Y, Zheng L, Yang X, Naphade M, Gedeon T (2020) Simulating content consistent vehicle datasets with attribute descent. In: Proceedings of European conference on computer vision, pp 775–791

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant No. 61633019.

Author information

Authors and Affiliations

  1. Research Center for Intelligent Perception and Control, Huzhou Institute of Zhejiang University, No. 819 Xisaishan Road, Huzhou, China

    Zhenfeng Xue & Yong Liu

  2. Institute of Cyber-Systems and Control, Zhejiang University, No. 38 Zheda Road, Hangzhou, China

    Zhenfeng Xue, Liang Chen, Zhitao Liu, Yong Liu & Weijie Mao

Authors
  1. Zhenfeng Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhitao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Weijie Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhenfeng Xue.

Ethics declarations

Conflict of interest

We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

A. Details of attribute definition In RockSegX, we design 12 attributes to efficiently change the content of simulated dataset. These attributes all have great influence on the dataset appearance. Below shows the details of these attributes.

  • Number of small rocks, that changes the numbers of small rocks placed on the conveyor belt.

  • Mean of small rock scales, that changes the mean of small rock scales on the conveyor belt.

  • STD of small rock scales, that changes the standard deviation of small rock scales on the conveyor belt.

  • Number of large rocks, that changes the numbers of large rocks placed on the conveyor belt.

  • Mean of large rock scales, that changes the mean of large rock scales on the conveyor belt.

  • STD of large rock scales, that changes the standard deviation of large rock scales on the conveyor belt.

  • Critical value between bright and dark pixels, that determines the threshold of pixel value to be annotated as rock.

  • Light intensity, that changes the brightness of virtual environments.

  • Light rotation angle along X axis, that changes the light direction along X axis.

  • Light rotation angle along Y axis, that changes the light direction along Y axis.

  • Camera rotation angle along X axis, that changes the camera direction along X axis.

  • Camera rotation angle along Y axis, that changes the Camera direction along Y axis.

It is worth noting that the large rocks are those to be annotated as rock part in the images, and the small rocks are those to be ignored and annotated as background. The critical value between bright and dark pixels is of vital significance in dealing with invisible part in the images.


B. More visualization of simulated datasets The biggest feature of RockSegX lies in its flexibility in controlling the content of simulated datasets. Here, we give more visualization to show the simulated datasets by influence of changing attribute values. The figures are shown in Fig. 10.

Fig. 10
figure 10

More visualization of simulated datasets. Each line shows a group of images that has the same attribute values

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xue, Z., Chen, L., Liu, Z. et al. ViRFD: a virtual-realistic fused dataset for rock size analysis in TBM construction. Neural Comput & Applic 34, 13485–13498 (2022). https://doi.org/10.1007/s00521-022-07179-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07179-4

Keywords

Search

Navigation