Abstract
Computer vision is vital for various applications like object tracking for autonomous driving or quality assurance. Hence, assuring that computer vision fulfills given quality criteria is essential and requires sufficient testing. In previous work, authors introduced a testing method relying on image modifications for a photometric stereo application. Image modifications include pixel errors or the rotation of images to be analyzed, revealing a substantial impact on the computed outcome of the photometric stereo application, depending on the applied modification. This paper focuses on whether we can reproduce the impact of image modifications in a real-world setup. In particular, we compare the impact of the rotation of the analyzed sample with the rotation modification applied to the image of the sample. The comparison indicates a similar effect when using rotation, showing that testing based on image modifications is valuable for verifying computer vision applications.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
See https://opencv.org.
References
Beizer, B.: Software Testing Techniques. Van Nostrand Reinhold (1990)
Brosch, N., Ginner, L., Schneider, S., Antensteiner, D., Traxler, L.: Quality inspection of translucent and micro-structured functional surfaces. In: Jalali, B., ichi Kitayama, K. (eds.) AI and Optical Data Sciences III, vol. 12019, pp. 173–184. International Society for Optics and Photonics, SPIE (2022). https://doi.org/10.1117/12.2605273
Eykholt, K., et al.: Robust physical-world attacks on deep learning visual classification. CVPR J. (2018)
Goodfellow, I., McDaniel, P., Papernot, N.: Making machine learning robust against adversarial inputs. Commun. ACM 61(7), 56–66 (2018). https://doi.org/10.1145/3134599
Ikehata, S.: CNN-PS: CNN-based photometric stereo for general non-convex surfaces. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)
Ikehata, S., Wipf, D., Matsushita, Y., Aizawa, K.: Robust photometric stereo using sparse regression. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 318–325 (2012). https://doi.org/10.1109/CVPR.2012.6247691
Klück, F., Wotawa, F., Neubauer, G., Tao, J., Nica, M.: Analysing experimental results obtained when applying search-based testing to verify automated driving functions. In: DSA, pp. 213–219. IEEE (2021)
Leitl, P.A., Wotawa, F., Naughton, J.W., Feichtinger, C., Husen, N.M., Flanschger, A.: Measurements of macroscopic and microscopic riblet defects and their impact on performance. In: Proceedings of the AIAA SCITECH 2022 Forum, vol. 37, San Diego, CA & Virtual (2022). https://doi.org/10.2514/6.2022-0916
Mostadi, M.E., Waeselynck, H., Gabriel, J.M.: Seven technical issues that may ruin your virtual tests for ADAS. In: 2021 IEEE Intelligent Vehicles Symposium (IV), pp. 16–21 (2021). https://doi.org/10.1109/IV48863.2021.9575953
Song, B.W., Liu, Z.Y., Xu, T., Hu, H.B., Huang, M.M.: A method for testing drag-reduction on riblet surfaces based on the Spalding formula. J. Mar. Sci. Appl. 8(4), 333–337 (2009). https://doi.org/10.1007/s11804-009-8073-5
Sotiropoulos, T., Waeselynck, H., Guiochet, J., Ingrand, F.: Can robot navigation bugs be found in simulation? An exploratory study. In: 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS), pp. 150–159 (2017). https://doi.org/10.1109/QRS.2017.25
Tian, Y., Pei, K., Jana, S., Ray, B.: Deeptest: automated testing of deep-neural-network-driven autonomous cars. In: Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, pp. 303–314. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3180155.3180220
Wicker, M., Huang, X., Kwiatkowska, M.: Feature-guided black-box safety testing of deep neural networks. In: International Conference on Tools and Algorithms for Construction and Analysis of Systems (2017)
Wotawa, F.: On the importance of system testing for assuring safety of AI systems. In: AISafety@IJCAI. CEUR Workshop Proceedings, vol. 2419. CEUR-WS.org (2019)
Wotawa, F., Jahaj, L.: Testing photometric stereo applications. In: 2022 9th International Conference on Dependable Systems and Their Applications (DSA) (2022). https://doi.org/10.1109/DSA56465.2022.00029
Wotawa, F., Klampfl, L., Jahaj, L.: A framework for the automation of testing computer vision systems. In: 2021 IEEE/ACM International Conference on Automation of Software Test (AST), pp. 121–124 (2021). https://doi.org/10.1109/AST52587.2021.00023
Wotawa, F., Peischl, B., Klück, F., Nica, M.: Quality assurance methodologies for automated driving. Elektrotech. Informationstechnik 135(4–5), 322–327 (2018)
Acknowledgements
This work is funded by the Austrian Research Promotion Agency (FFG) within the project RiSPECT (874163).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 IFIP International Federation for Information Processing
About this paper
Cite this paper
Wotawa, F., Jehaj, L., Brosch, N. (2023). On the Evaluation of Photometric Stereo Applications Testing Using Image Modifications. In: Bonfanti, S., Gargantini, A., Salvaneschi, P. (eds) Testing Software and Systems. ICTSS 2023. Lecture Notes in Computer Science, vol 14131. Springer, Cham. https://doi.org/10.1007/978-3-031-43240-8_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-43240-8_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43239-2
Online ISBN: 978-3-031-43240-8
eBook Packages: Computer ScienceComputer Science (R0)
