This paper presents a scalable architecture for realizing real-time speech recognizers based on a word HMM (hidden Markov model). HMM-based recognition algorithms are classified into two acoustic models, i.e., phenome-level model and word-level model. The phenome-level HMM has been widely used in current speech recognition systems which permit large-sized vocabularies. Whereas the word-level HMM has been constrained to small-sized vocabularies because of extremely high computation cost in spite of excellent recognition performance. In order to overcome the shortage, we adopt the scalable architecture focused on the word HMM structure. The proposed architecture can flexibly improve recognition performance and extend word vocabularies. In addition, the computation time is hardly increasing. In order to demonstrate practical solutions, we have designed and evaluated a total system recognizer including speech analysis and noise robustness on a 0.18 /spl mu/m CMOS standard cell library. The recognition time is 35.7 /spl mu/s/word at 128 MHz operating frequency. The recognizer can achieve over middle-sized vocabularies in real-time response.