Modeling the driver behavior is expected to play a fundamental role in designing systems of driver monitoring, warning, assist control and training. In this paper, we present an identification method of automobile driver operations based on a hierarchical clustering approach, which leads to a stochastic piecewise affine (PWA) model. The driver behavior can be viewed as an outcome of the hybrid system that consists of (continuous) primitive driving operations and their (discrete) switchings. We describe the driving primitives by PWA models and the switchings by hidden Markov models (HMMs). One significant issue of this hybrid modeling is to extract the distinct states of driving operation from the driver behavior and determine the number of the states. To this problem, we propose a method to estimate the number of states using an idea of hierarchical clustering. We apply our identification method to the accelerator operations of driver, and demonstrate its efficacy through numerical experiments using the real data of four drivers.