We describe a new online discriminative learning algorithm that efficiently and effectively recognizes phonemes in a speech sequence. The method builds upon recent work in online learning of a collection of generalized linear models using second order statistics of the model weight vectors. Evaluation on the TIMIT database shows that the algorithm achieves state-of-the-art phoneme recognition error rates compared to many other generative and discriminative models with the same expressive power.