With the growth of tagged images, researchers are using this highly semantic tag information to assist some vision tasks such as image clustering. However, users may not tag some images at all or some of the images are partially annotated, and this will lead to performance degradation, which is rarely considered by previous works. To alleviate this problem, we propose a new model for image clustering assisted by partially observed tags. Our model enforces sparse representations obtained through sparse coding and latent tag representations learned via matrix factorization to be consistent with the partial image-tag observations. The partition of image database is finally performed using clustering algorithms (e.g., k-means) on the sparse representations. Extensive experiments demonstrate that the proposed model performs better than the state-of-the-art methods.