Matrix factorization is one of the most successful collaborative filtering methods for recommender systems. Traditionally, matrix factorization only uses the observed user-item feedback information, which makes predictions on cold users/items difficult. In many applications, user/item content information are also available and they have been successfully used in content-based methods. In recent years, there are attempts to incorporate content information into matrix factorization. In particular, the Factorization Machine (FM) is one of the most notable examples. However, FM is a general factorization model that models interactions between all features into a latent feature space. In this paper, we propose a novel combination of tree-based feature group learning and matrix co-factorization that extends FM to recommender systems. Experimental results on a number of benchmark data sets show that the proposed algorithm outperforms state-of-the-art methods, particularly for predictions on cold users and cold items.