Radio environment map (REM) is a database repository widely adopted for applications such as spectrum sensing, interference management and network planning for wireless networks. However, conventional REM construction requires measurement-capable devices (MCDs) to upload location-based measurements, which exposes the privacy of users. In this paper, we propose a new method to construct the REM utilizing federated learning (FL) to preserve user privacy. In this method, multiple mobile MCDs collect the signal strength measurements in parallel and store the measurements locally. FL is then used to train a shared deep machine learning (DML) model for multiple MCDs collectively. In addition, we pre-process the location information and apply power adjustment to the measurements, either to increase the dynamic range of the input data to the FL or decrease the dynamic range of the measurements. The performance of the proposed FL-based method is evaluated in terms of the colormaps, heatmaps and cumulative distribution function (CDF) of estimation errors. The colormaps show that the power adjustment greatly improves the performance of the FL-based REM. The heatmaps show that smaller estimation errors can be achieved with more participating MCDs. The simulation results also show that the proposed FL-based method achieves better performance than the distance-based method and the nearest-neighbor-based (NN-based) method, in terms of estimation errors.