We present a theoretical analysis of Learning Vector Quantization (LVQ) with adaptive distance measures. Specifically, we consider generalized Euclidean distances which are parameterized in terms of a quadratic matrix of adaptive relevance parameters. Winner-takes-all prescriptions based on the heuristic LVQ1 are in the center of our interest. We derive and study stationarity conditions and show, among other results, that stationary prototypes can be written as linear combinations of the training data apart from irrelevant contributions in the null-space of the relevance matrix. The investigation of the metrics updates reveals that relevance matrices become singular with only one or very few non-zero eigenvalues. Implications of this property are discussed and, furthermore, the effect of preventing singularity by introducing an appropriate penalty term is studied. Theoretical findings are confirmed in terms of illustrative example data sets.