This paper presents the development of the modeling of human driving behavior based on an expression as a hybrid dynamical system (HDS) focusing on the driver's vehicle following task. The driving data are collected by using a driving simulator which can provide a stereoscopic immersive vision to the driver. In our modeling, the relationship between the measured sensory information and the output of the driver are expressed by the piecewise ARX (PWARX) model, which is a class of the HDS. As the sensory input, the range between vehicles, range rate, and time derivative of the area of the back of the preceding vehicle (called KdB) are considered. Furthermore, the pedal operation is considered as the output. The identification problem for the PWARX model is solved using the combination of the data clustering and support vector machine. By introducing the PWARX model, it becomes possible to find not only parameters appearing in the motion in each mode but also parameters in the logical switching (decision making) conditions among them from the measured driving data. From the obtained results, it is found that the driver appropriately switches some "control laws" according to the sensory information, in particular, the KdB plays an important role in the decision making, i.e., the switching between modes.