The Bayesian mixture of factor analyzers (BMFA), which achieves joint clustering and dimensionality reduction, is with an appealing feature of automatic hyper-parameter learning. In addition to its great success in various unsupervised learning tasks, it exemplifies how the Bayesian statistics can be leveraged to achieve automatic hyper-parameter learning, which is an open problem of modern simultaneous (deep) dimensionality reduction and clustering. Due to the importance of the BMFA, in this paper, its mechanism is carefully investigated, and a robust variant of the BMFA that can mitigate potential outliers is further proposed. Numerical studies are presented to show the remarkable performance of the proposed algorithm in terms of accuracy and robustness.