Anomaly detection based on communication behavior is one of difficult problems of industrial control systems for intrusion detection. A normal communication behavior control model is established by using improved one-class SVM and a PSO-OCSVM algorithm based on particle swarm algorithm is designed to optimize parameters in this paper. This method established an intrusion detection model to identify abnormal Modbus TCP traffic according to the normal Modbus function code sequence. And the efficiency, reliability and real-time of the proposed method met the industrial control system for anomaly detection are proved by simulation results.