In order to reduce traffic accidents caused by the pedestrian, five kinds of dangerous pedestrian abnormal behaviors are studied in the paper. A behavior model between the pedestrian trajectory and the road is built to describe the five kinds of dangerous pedestrian abnormal behaviors: crossing road border, illegal stay, crossing the road, moving along the curb, entering road area. The method contains pedestrian detection, shadow elimination, pedestrian recognition, pedestrian tracking and abnormal behavior detection. Background subtraction method is used to detect moving targets. After shadow elimination, pedestrians are distinguished from vehicles according to the ratio. Then, pedestrian trajectories are gotten by pedestrian tracking. Finally, based on the relation between trajectory and road, the model of five kinds of pedestrian abnormal behaviors is established, and abnormal behaviors are detected according this model. Experiments show that the method can distinguish and detect the pedestrian abnormal behaviors effectively in short time, and it is suitable to use in real time traffic monitoring.