This paper presents the development of the modeling strategy of the human driving behavior based on the expression as piecewise linear (PWL) model focusing on the driver's collision avoidance behavior. The driving data are collected by using the three-dimensional driving simulator based on CAVE, which provides stereoscopic immersive vision. In our modeling, the relationship between the driver's sensory information and the operation of the driver such as braking and/or steering amount is expressed by PWL model, which is a class of hybrid dynamical system (HDS). The identification problem for the PWL model is formulated as the mixed integer linear programming (MILP) by transforming the switching conditions into binary variables. From the obtained results, it is found that the driver appropriately switches the 'control low' according to the sensory information. These results enable us to capture not only the physical meaning of the driving skill, but also the decision-making aspect (switching conditions) in the driver's collision avoidance behavior.