Discriminative training of hidden Markov models (HMMs) using minimum classification error training (MCE) has been shown to work well for certain speech recognition applications. MCE is, however, somewhat prone to overspecialization. This study investigates various techniques which improve performance and generalization of the MCE algorithm. Improvements of up to 10% in relative error rate on the test set are achieved for the TIMIT dataset.