Nowadays, the generation of energy consumption models in buildings needs to address several issues. Indeed, most existing buildings lack the appropriate equipment to obtain the data required to create such models. Furthermore, the nature of energy consumption data could be correlated with additional information about people in the building that could raise privacy concerns. Based on such aspects, we propose a Federated Transfer Learning (FTL) framework to handle these buildings' data without compromising any private information in which a set of buildings are clustered according to certain characteristics. On the one hand, our works leverage the properties of Federated Learning (FL) to train an energy forecasting model using a small portion of the available buildings respecting their privacy. On the other hand, we transfer the model to the rest of the buildings by using Transfer Learning (TL). We extensively evaluate our approach, and demonstrate it improves the results of alternative scenarios where FL and TL are used separately.