In recent years, there is an increased need for quality control in the manufacturing sectors. In the steel making, the rolling operation is often the last process that significantly affects the bulk microstructure of the steel. The cost of having defects on rolled steel is high because it takes more than 5000 KW-Hr to produce a ton of steel. Early detection of defects can reduce product damage and manufacturing cost. This paper describes a real-time visual inspection system that uses support vector machine to automatically learn complicated defect patterns. Based on the experimental results generated from over one thousand images, the proposed system is found to be effective in detecting steel surface detects. The speed of the system for feature extraction and defect detection is less than 6 msec per one-megabyte image.