skip to main content
10.1145/3383972.3384036acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicmlcConference Proceedingsconference-collections
research-article

Image-based Renovation Progress Inspection with Deep Siamese Networks

Published: 26 May 2020 Publication History

Abstract

Various specialized machine vision systems have been proposed for item inspection, robot supervision and quality control in manufacturing and construction industries. However, construction industries are still lacking solutions for automating the progress inspection in building renovation projects. As smartphones becoming increasingly pervasive among construction workers and use of smartphone photos for documentation getting more popular, in this work, we propose machine learning methods for automatic progress recognition in renovation projects from smartphone images. Renovation progress inspection is formulated as an ordinal classification problem, with every class representing one stage of the renovation process. The baseline solutions inspired by the popular deep learning architectures like VGG19, ResNet, Xception, DenseNet and MobileNet do not benefit from the temporal property of the data. Consecutive stages share a substantial amount of visual features, which makes it difficult to differentiate between them. To cope with that, we design special networks - Ordering Nets which utilize the consecutive property of the classes to predict the order of a photo pair. For the special use case, we train the 1-Step- Net to recognize progress from two subsequent photos. The extracted order information increases classification accuracy and provides more precise temporal prediction. We report our results on a new Reno-2018 dataset - a two-part collection of photos that covers bathroom and kitchen renovation steps. The applicability of our approach is demonstrated on a simulated progress estimation task. Our method significantly improves the temporal accuracy of stage predictions compared to the base deep neural network models.

References

[1]
B. Albertina, M. Watson, C. Holback, R. Jarosz, S. Kirk, Y. Lee, and J. Lemmerman. Radiology data from the cancer genome atlas lung adenocarcinoma [tcga-luad] collection., 2016. The Cancer Imaging Archive.
[2]
F. D. Anton, S. Anton, and T. Borangiu. Integration of Visual Quality Control Services in Manufacturing Lines, pages 329--342. Springer International Publishing, Cham, 2014.
[3]
C. Beckham and C. Pal. A simple squared-error reformulation for ordinal classification. arXiv e-prints, abs/1612.00775, Dec. 2016.
[4]
L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. S. Torr. Fully-convolutional siamese networks for object tracking. In G. Hua and H. Jégou, editors, Computer Vision -- ECCV 2016 Workshops, pages 850--865, Cham, 2016. Springer International Publishing.
[5]
J. Bromley, I. Guyon, Y. LeCun, E. Säckinger, and R. Shah. Signature verification using a "siamese" time delay neural network. In J. D. Cowan, G. Tesauro, and J. Alspector, editors, Advances in Neural Information Processing Systems 6, pages 737--744. Morgan-Kaufmann, 1994.
[6]
M. BÃijgler, G. Ogunmakin, J. Teizer, P. A. Vela, and A. Borrmann. A comprehensive methodology for vision-based progress and activity estimation of excavation processes for productivity assessment. In EG-ICE Workshop on Intelligent Computing in Engineering, Cardiff, Wales, 2014.
[7]
F. Chollet. Xception: Deep learning with depth-wise separable convolutions. CoRR, abs/1610.02357, 2016.
[8]
J. Cohen. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological bulletin, 70(4):213--220, Oct. 1968.
[9]
J. Cuadros and I. Sim. Eyepacs: An open source clinical communication system for eye care. In MedInfo, volume 107 of Studies in Health Technology and Informatics, pages 207--211. IOS Press, 2004
[10]
H. Drucker, C. Cortes, L. D. Jackel, Y. LeCun, and V. Vapnik. Boosting and other ensemble methods. Neural Computation, 6(6):1289--1301, 1994.
[11]
H. Hamledari, B. McCabe, and S. Davari. Automated computer vision-based detection of components of under-construction indoor partitions. Automation in Construction, 74:78--94, 2017.
[12]
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015.
[13]
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770--778, June 2016. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko,
[14]
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, abs/1704.04861, 2017.
[15]
L. Klein, N. Li, and B. Becerik-Gerber. Imaged-based verification of as-built documentation of operational buildings. Automation in Construction, 21:161--171, 2012.
[16]
G. Koch, R. Zemel, and R. Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML Deep Learning workshop, 2015.
[17]
M. Kopsida, I. Brilakis, and P. A. Vela. A review of automated construction progress monitoring and inspection methods. In Proc. of the 32nd CIB W78 Conference 2015, pages 421--431, 2015.
[18]
J. Manyika, S. Ramaswamy, S. Khanna, H. Sarrazin, G. Pinkus, G. Sethupathy, and A. Yaffe. Digital America: a tale of the haves and have-mores. Technical report, McKinsey & Company, 2015.
[19]
N. Mehdiyev, D. Enke, P. Fettke, and P. Loos. Evaluating forecasting methods by considering different accuracy measures. Procedia Computer Science, 95:264--271, 2016. Complex Adaptive Systems Los Angeles, CA November 2-4, 2016.
[20]
I. Melekhov, J. Kannala, and E. Rahtu. Relative camera pose estimation using convolutional neural networks. CoRR, 2017.
[21]
K. Ricanek and T. Tesafaye. Morph: a longitudinal image database of normal adult age-progression. In 7th International Conference on Automatic Face and Gesture Recognition (FGR06), pages 341--345, April 2006.
[22]
S. Roh, Z. Aziz, and F. Peña-Mora. An object-based 3d walk-through model for interior construction progress monitoring. Automation in Construction, 20(1):66--75, 2011.
[23]
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, 2014.
[24]
D. Weimer, B. Scholz-Reiter, and M. Shpitalni. Design of deep convolutional neural network architectures for automated feature extraction in industrial inspection. CIRP Annals - Manufacturing Technology, 65(1):417ÂŰ420, 2016.

Index Terms

  1. Image-based Renovation Progress Inspection with Deep Siamese Networks

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    ICMLC '20: Proceedings of the 2020 12th International Conference on Machine Learning and Computing
    February 2020
    607 pages
    ISBN:9781450376426
    DOI:10.1145/3383972
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    In-Cooperation

    • Shenzhen University: Shenzhen University

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 26 May 2020

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Inspection
    2. deep learning
    3. ordinal classification
    4. progress estimation

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Funding Sources

    • Business Finland

    Conference

    ICMLC 2020

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 134
      Total Downloads
    • Downloads (Last 12 months)11
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 27 Jan 2025

    Other Metrics

    Citations

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media