Abstract
The evaluation of driving performance is a vital way to reflect the usability of in-vehicle system. Study the impact of in-vehicle interaction on driving performance can help avoiding hidden driving dangers. Speed of vehicles and distance from the vehicles ahead during driving are important indicators to reflect driving performance. In this study, Speed measurement relies on the GPS module. Conventionally, precise distance detection during driving is mostly based on radar sensors or high-resolution cameras that are both quite expensive. This paper proposed an object distance detection algorithm that relies on ordinary HD binocular camera with relatively low price to detect the distance. A new mismatch elimination method is proposed to improve the performance of the algorithm. At the same time, this paper designed a driving performance evaluation experiment. The distance is measured according to the proposed algorithm. Driving performance with primary tasks (speed maintenance and distance maintenance) and secondary tasks (touch control and voice control) on different driving scenes (straight road and curve road) are evaluated. Experimental results showed that introduction of secondary tasks dose influence the operation of the driver by distracting him. It also affects the driver’s response to external changes. Both speed maintenance task and distance maintenance task have verified this conclusion. The proposed object distance detection method satisfies the accuracy required for driving performance evaluation.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
With the development of industry and progress of society, the number of car ownership has increased significantly. The per capita possession of automobiles is also growing rapidly. Since the traffic scenes are getting more and more complex, requirements for driving system are higher than before.
Considering the advance of automotive industry, the emergence of the intelligent driving system is obviously the future form of transportation. Intelligent driving system can help with many traffic scenes [1]. There will be more communication and collaboration between the driver and the vehicle in the driving system. Human-machine coordinated control system will be the mainstream in the future.
Many factors can influence the driver, including the perception of the environment, the in-vehicle interaction and the psychological and physiological state of the driver.
Driving performance is an essential foundation of the intelligent driving system evaluation. Safe and stable driving of the vehicles is the requirement of the human-machine coordination vehicle system. Detecting vehicles that may affect the driver is an effective way to observe the traffic condition. This also serves as an indicator for driving performance evaluation.
In this article, we focus on the distance from the vehicle ahead while driving. The driver’s mental status and ability of decision-making would affect the performance of distance control. Therefore, the task of keeping distance from the vehicle ahead can be the suitable reference to driving performance. On the other hand, driving speed is another indicator to reflect the driving performance. In this paper, we measured the speed by a GPS module.
Lazar-based methods and visual-based methods are two main methods for distance measurement. Traditional methods, however, cannot avoid the restriction between cost and performance. Considering the dilemma, a visual-based method was proposed in this article.
Traditional binocular visual ranging method including the following steps: calibration of the binocular camera, image rectification, stereo matching, and depth calculation. Stereo matching consumes most of the resource. In the phase of stereo matching, feature extraction and feature matching are main contents. Traditional stereo based on feature points needs a large number of calculations in view of various characteristic such as rotation invariance, scale invariance and invariance of brightness change.
2 The Object Distance Detection Method
2.1 The Visual-Based Ranging Method
Depth information calculated by visual-based ranging methods is based on pictures captured by the cameras. The binocular visual systems include two cameras with similar parameters.
Through the scheme proposed in [3], we can finish the calibration of the two cameras and obtain the internal and external parameters of the binocular camera. Those parameters reflect the hardware information of the cameras and play a role in the following processes.
Selection and Matching of the Feature Points
The selection of feature points is a key step in visual ranging. Feature points matching is the basis of parallax calculation. So far, there have been many feature points extraction methods [2]. Binary Robust Independent Elementary Features (BRIEF) is a kind of feature descriptor that simply uses binary test between pixels in a smoothed image patch [4]. With the consideration of robustness and rotation invariance, Oriented features from accelerated segment test (FAST [5]) and Rotated BRIEF (ORB) [6] were proposed. Besides, the Scale Invariant Feature Transform (SIFT) [7], the Speed Up Robust Features (SURF) [8] and the FAST methods are widely used now.
Considering our project, it is not necessary to maintain these features at the cost of computation and time. In a situation where the target object (vehicle) and the background (road) are very different, we use a calibrated binocular camera to extract feature points and target objects. Then, it is suitable to get reliable result rely on the gradient difference. We can extract feature points according to gradient differences in one-dimensional direction because of the epipolar constraints.
The feature points were selected from the peak points of the gradient in the article. At the same time, a threshold (gradient greater than 50) is set to pick out points for matching process. Each feature point will be compared with feature points on the same polar line.
Mismatching Elimination
After finishing the matching process, most of feature points have the matching point. Besides the feature points without matching points because of the camera view, there are still many mismatched point pairs.
There is no doubt that the mismatched point pairs have a large negative impact on the results. In the past, many studies have proposed solutions to eliminate it. [9, 10] and [11] used the feature of geometric consistency to eliminate feature point pairs whose parallax changes are too significant in small neighborhoods. This feature is based on the fact that the surface of objects is basically smooth in the real world, and the depth value rarely changes suddenly. When the feature points are distributed on the physical margin, it is not surprised that the depth of the feature points in the neighborhood changes suddenly. Another property called matching consistency is adopted in [12,13,14] and [15]. These papers believe that there should be a certain convergence between pairs of matching points in similar areas, which means that matching points of adjacent feature points are distributed in adjacent areas. The parallax change trend reflects the smoothness of the area where the matching points of the feature points are located. It is conceivable that these two characteristics also depend on the smoothness and consistency of the real world.
Further, in [16, 17] and [18], researchers put their eyes on the epipolar constraint. In simple terms, epipolar constrains describe the positional relationship of feature points on the epipolar line. A feature point and its matching point are on the same epipolar line. In the feature point matching process, the spatial ordering of feature points also plays a role. The matching point pairs in the two images of the image pair have the same ordering relationship according to the spatial position. While traversing feature points in the other image to find matching points of feature points, epipolar constraints and order constraints can greatly reduce the workload of traversal and similarity calculations.
In the matching process, mismatched point pairs are inevitable but undesired. Therefore, we screen out the mismatched pairs and discard them. This paper calculates the vector between point pairs from image pairs. In order to magnify the difference between mismatched pairs and well-matched pairs, we put the rectified image pairs in a 2*2 table. The image from the left camera was put on the upper left corner while the image from the right camera in the lower right corner. A two-dimensional Cartesian coordinate system is established. The intersection of the two images is set as the origin. The images shot by the left and right cameras are located in the third quadrant and the first quadrant, respectively. After computing the vectors between matching point pairs in the new coordinate. Mismatched pairs arrive at vectors very different from good matches because of epipolar geometry constraints.
Figure 1 and Figure 2 show the effect of the mismatch rejection. Because of camera calibration and image rectification, the images in a pair can be regarded as two pictures shot by the same camera at the same time. This ideal model also provides the theoretical basis for the mismatch elimination process.
The two points in the matched feature point pair are on the same polar line according to epipolar line constraints. The essential matrix was calculated in the process of calibration. It can be seen as the prior knowledge for our depth calculation. A kind of error function was defined in [11]:
Each feature point multiplied by the same essential matrix shows the same spatial change. Therefore, the calculation of the feature matrix can be skipped. In this paper, the slope between two points in the new coordinate system is proposed as a new feature to match point pairs.
In Eq. (2), k represents the slope, which is a new indicator for matching point pairs. p means the feature point. The upper corner mark indicates which camera the image is coming from, and the lower corner mark is to distinguish the horizontal and vertical coordinates.
Because of the order constraints, the mismatching point pairs have higher slope than the well-matching point pairs. Setting a threshold around the minimum slope can avoid the calculation error. Most mismatched feature point pairs are eliminated in this process.
2.2 Object Detection in 3D Space
With the process described in the previous section, we can get the three-dimensional point clouds mainly distributed on the margins of objects. This is not enough for the analysis of the target vehicle. We need do more work to extract the point clouds belong to the target vehicle.
Point Clouds Clustering
The three views of the point cloud are shown below. The outlines of vehicles were shown in Fig. 3, Fig. 5 and Fig. 7. There are still many points scattered discretely throughout the space. It is necessary to remove those discrete isolated points since they are useless for subsequent processes.
Density-based spatial clustering of applications with noise (DBSCAN) [19], as named, is a density-based clustering method. This clustering method has the advantage of not need to set the clustering method at beginning.
After applying the DBSCAN algorithm to the point cloud shown in Fig. 3, we obtain the point clouds shown in Fig. 4, Fig. 6 and Fig. 8. Different colors of point clouds represent different classes. With the application of DBSCAN, the point clouds are divided into different clusters according to the spatial density distribution. In addition, the noise marking function of DBSCAN in the clustering process also enables it to remove outlier points. By the way, we set the parameters of DBSCAN algorithm including Ɛ (Ɛ = 700 mm) and MinPts (MinPts = 15).
Target Vehicle Extraction
Point clouds located on different objects are divided into several clusters after clustering. To get the target vehicle, we need to re-integrate the clustering result.
The template matching combined with multiple frames point clouds satisfied the requirement of the experiment. It also provide the point clouds of the target vehicle. When a vehicle is running on the road, information such as the road width and the field of vision is utilized to select point clouds within the effective area in advance. Traversing and matching the point clouds classes start from the nearest point cloud class in the effective area. We get the point clouds of target vehicle by matching the combination of point clouds with a preset template.
3 Experiments
3.1 Apparatus
In this experiment, the proposed distance detection method is used to measure the distance between the vehicle and the vehicle ahead. To evaluate the driver’s state during driving, physiological indicators including eye movement and heart rate were collected. Those data will be the indicators for the physical and psychological state of the driver.
Distance changes in the distance maintenance task and speed changes in speed maintenance task are indicators for driving performance. Driving speed and following distance were recorded in our experiments.
To get the above-mentioned data, the following equipment is included in the experiments: a heart rate watch to record the heart rate, an eye tracker to record the driver’s field of vision, and the GPS module to record the driving speed. Of course, a binocular camera (HNY-CV-002, LenaCV) for distance measurement is also included.
The participant wore a heart watch and the eye movement glasses (see Fig. 9). Figure 10 shows appearance of experimental vehicle. The binocular camera was placed on the centerline of the roof to collect images during driving.
3.2 Tasks
Following distance and the driving speed are the dependent variables in the experiment. Images captured by the binocular camera is used for the distance measurement while speed recorded by the GPS module. We set primary tasks including distance maintenance and speed maintenance. Secondary tasks (voice control and touch control) are also set in this experiment.
We asked the participant to execute the secondary tasks while perform the primary tasks. Control groups of the experiment were the primary tasks without secondary tasks. The experimental group is consist of primary tasks with different secondary tasks.
Primary Tasks
Speed Maintenance Task
In this task, we asked the participant to drive the vehicle with a preset speed (20 km/h). The change of speed during the experiment was recorded for further analysis. This task includes driving on the curve road and straight road.
Distance Maintenance Task
The participant drives on the straight road after the target vehicle and keep the following distance at a stable value. We recorded the distance from the target vehicle ahead.
Secondary Tasks
Voice Control
The participant adjusts the temperature of the vehicle air conditioner by the in-vehicle voice control system in this task.
Touch Control
The participant adjusts the temperature of the vehicle air conditioner by touching the main touchable screen in this task.
4 Results
See figures and tables.
5 Discussion
5.1 Speed Maintenance Task
Figure 11 shows the speed changes while driving on the curve. It has a clear gap compared with the straight track without secondary task. There is no doubt that it is harder to control the direction on curve road than driving on the straight road.
After introducing the secondary tasks for air-conditioning adjustment and driving on the same route, the speed curves of driving speed are shown in Fig. 12 and Fig. 13. The former figure presents the speed changes on the curve road, while the latter one is on the straight road.
Comparing the results with and without secondary task, we find that the fluctuations of speed are more and larger with secondary tasks. Standard deviations (see Table 1) provide statistical evidence. The secondary tasks trigged the speed fluctuations. Touch control apparently caused larger and more fluctuations than voice control.
This phenomenon shows that the introduction of the secondary tasks does not only interfere with the driver’s operation, but also affects his judgment of the sudden changes in the external environment. Even if the implement of the secondary tasks and changes of traffic conditions did not happen simultaneously.
Results in Table 1 validates the differences between different experiments. The mean speed is closer to the setting value while driving without secondary tasks. Primary tasks on straight road have better speed control performance than on curve road.
From timestamp 20200112165355 to 20200112165411, the speed curve has the largest change in Fig. 12. We analyze the heart rate and eye movement data during this period. The average heart rate is 80.29 bpm, while the average heart rate during the whole task is 83.53 bpm. According to the pictures captured by binocular camera, the vehicle turned around during this time and was about to go uphill.
With these recorded data, we can speculate that those secondary tasks do affect the primary task by affecting the driving operation. They also affect the driver’s ability to response to changes in the external scene.
5.2 Distance Maintenance Task
Figure 14, Figure 15 and Figure 16 show the changes of the following distance during the distance maintenance task. In this task, we did not set a specific holding distance but asked the participant to maintain a stable distance. The fluctuation frequency and amplitude of the following distance curves reflect the performance of the distance maintenance task.
Standard deviations in Table 3 indicate the performance of different experiments. Since these values are arranged in descending order of voice control, touch control and no secondary tasks. This order is also the same as the order of the driving performance.
From the experimental results, we can find that the introduction of secondary tasks makes the performance of the distance-maintaining task worse. The increase of amount of peaks and troughs of the distance curve verified this conclusion. The amplitudes of distance curve is also larger in Fig. 15 than Fig. 14.
The main factor of peaks and troughs of distance curves is the sudden changes in the speed of the vehicle ahead. The introduction of secondary tasks cost the driver more time to readjust the distance. Touch control has more influence than voice control. We can speculate that the secondary tasks have affected the driver’s intuitive impact on the distance-maintaining task that require continuous attention to external world.
Heart rate of the participant during the distance maintenance task is shown in Fig. 16. These figures are not significantly different between each other. In other words, the driver’s psychological state is not affected too much by the secondary tasks.
Percentage of the time looking outside the window (see Table 2) shows how much the driver was distracted. While adjusting the air conditioner by the touch control, the driver took less time focusing on the road. This also shows that adjusting the air conditioner through touch control is easier to distract the driver than voice control.
6 Conclusions
From the discussion in the last section, secondary task makes negative impact on driving performance. It influences the driver by distracting him. The distraction directly influence the driving operation and makes the driver’s reaction to external changes getting slow. Different in-vehicle interactions has different influence. Touch control has a greater distraction than voice control. With the analysis of the experimental results, it is clearly that distractions significantly affect driving performance. On the other hand, the proposed object distance detection method satisfies the accuracy required for performance evaluation.
References
Bishop, R.: Intelligent vehicle applications worldwide. IEEE Intell. Syst. Appl. 15(1), 78–81 (2000)
Sun, Q.: An improved binocular visual odometry algorithm based on the random sample consensus in visual navigation systems. Ind. Robot Int. J. 44(4), 542–551 (2017)
Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000)
Calonder, M., Lepetit, V., Strecha, C., Fua, P.: BRIEF: binary robust independent elementary features. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 778–792. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15561-1_56
Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 430–443. Springer, Heidelberg (2006). https://doi.org/10.1007/11744023_34
Rublee, E., Rabaud, V., Konolige, K., et al.: ORB: an efficient alternative to SIFT or SURF. In: 2011 International Conference on Computer Vision, pp. 2564–2571 (2011)
Lowe, D.G.: Object recognition from local scale-invariant features. In: Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 2, pp. 1150–1157 (1999)
Bay, H., Tuytelaars, T., Van Gool, L.: SURF: speeded up robust features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 404–417. Springer, Heidelberg (2006). https://doi.org/10.1007/11744023_32
Pizarro, D., Bartoli, A.: Feature-based deformable surface detection with self-occlusion reasoning. Int. J. Comput. Vis. 97(1), 54–70 (2012). https://doi.org/10.1007/s11263-011-0452-0
Lipman, Y., Yagev, S., Poranne, R., et al.: Feature matching with bounded distortion. ACM Trans. Graph. 33(3), 1–14 (2014)
Lin, W.-Y.D., Cheng, M.-M., Lu, J., Yang, H., Do, M.N., Torr, P.: Bilateral functions for global motion modeling. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 341–356. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10593-2_23
Lin, W.-Y., Liu, S., Jiang, N., Do, M.N., Tan, P., Lu, J.: RepMatch: robust feature matching and pose for reconstructing modern cities. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 562–579. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_34
Wang, C., Wang, L., Liu, L.: Density maximization for improving graph matching with its applications. IEEE Trans. Image Process. 24(7), 2110–2123 (2015)
Bian, J., Lin, W., Matsushita, Y., et al.: GMS: grid-based motion statistics for fast, ultra-robust feature correspondence. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2828–2837 (2017)
Maier, J., Humenberger, M., Murschitz, M., Zendel, O., Vincze, M.: Guided matching based on statistical optical flow for fast and robust correspondence analysis. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 101–117. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46478-7_7
Kushnir, M., Shimshoni, I.: Epipolar geometry estimation for urban scenes with repetitive structures. IEEE Trans. Pattern Anal. Mach. Intell. 36(12), 2381–2395 (2014)
Talker, L., Moses, Y., Shimshoni, I.: Estimating the number of correct matches using only spatial order. IEEE Trans. Pattern Anal. Mach. Intell. 41(12), 2846–2860 (2019)
Zhao, P., Ding, D., Wang, Y., et al.: An improved GMS-PROSAC algorithm for image mismatch elimination. Syst. Sci. Control Eng. 6(1), 220–229 (2018)
Ester, M., Kriegel, H.-P., Sander, J., et al.: A density-based algorithm for discovering clusters a density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, pp. 226–231. AAAI Press, Portland (1996)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Gao, Y., Wang, Z., Fu, S. (2020). An Object Distance Detection Method for Driving Performance Evaluation. In: Harris, D., Li, WC. (eds) Engineering Psychology and Cognitive Ergonomics. Cognition and Design. HCII 2020. Lecture Notes in Computer Science(), vol 12187. Springer, Cham. https://doi.org/10.1007/978-3-030-49183-3_23
Download citation
DOI: https://doi.org/10.1007/978-3-030-49183-3_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-49182-6
Online ISBN: 978-3-030-49183-3
eBook Packages: Computer ScienceComputer Science (R0)