Great advancements in deep-learning-based machine vision are boosting the automotive industry potential to accurately recognize vehicle environment. The need for high quality datasets is of critical importance when training neural networks which are used to detect and classify objects in front of the vehicle. Available open source datasets for semantic segmentation can be used by a wide community of researchers to develop next generation self-driving functions. Those datasets have severe limitations such as class imbalance, unobserved objects, erroneous labelling and limited number of covered scenarios. In this paper, we propose a sequence of steps to combine and manipulate existing open source datasets to maximize the inference performance. Those steps include relabeling with outlier removal, class-driven balancing of validation and training datasets, as well as targeted image manipulation for scarce classes. Our evaluation indicates the improved inference accuracy when compared to the usage of most common open source datasets.