Increasing Trust in Data-Driven Model Validation

Abstract

In recent years, interest in autonomous systems has increased. To observe their environment and interact with it, such systems need to process sensor data including camera images. State-of-the-art methods for object recognition and image segmentation rely on complex data-driven models such as convolutional neural networks. Although no final answer exists yet on how to perform safety evaluation of systems containing such models, such evaluation should comprise at least validation with realistic input data, including settings with suboptimal data quality. Because many test datasets still lack a sufficient number of representative quality deficits, we consider augmenting existing data with quality deficits as necessary. For this purpose, a novel tool framework is presented and illustrated using traffic sign recognition as a use case. The extendable approach distinguishes between augmentation at the object, context, and sensor levels. To provide realistic augmentation and meta-data for existing image datasets, known context information and conditional probabilities are processed. First applications on the GTSRB dataset show promising results. The augmentation of datasets facilitates a more rigorous investigation of how various quality deficits affect the accuracy of a model in its target application scope.