For robot manipulation, it does not only require accuracy but also a fast response if possible. Neural Network has the advantages of high tolerance of error and has the ability of parallelism calculation. When applying to the real time speech recognition system, through one time computation then can get the recognition result immediately, that is different from other methods like VQ, DTW, HMM. So, using Neural Network method to the field for robot speech operation is a good choice. But using Neural Network as the identifier, the dimension of input vector will large, it will occupy more memory storage, and will affect the efficiency of calculation. Therefore, in this paper we raise the concept to combine HMM and BPNN, it can reduce the dimension of input vector to decrease the burden of memory storage; on the other hand, it can also promote the calculating efficiency. For resolving a general BP network problem of slow convergence while training, in this paper we raise the concept of using the recognition rate as a factor to judge whether to stop the training procedure or not, which can save more training time and can also get the required recognition rate.