In this paper, we examine the feasibility of FPGA as a platform for training a convolutional binary-weight neural network. Training a neural network requires more data movement compared to inference. Acceleration of training on an FPGA is, therefore, a challenge because the data movement increases off-chip memory accesses. We try to address this problem by storing most of the data in the on-chip memory and adopting batch renormalization. This allows for training a large network by reducing the required intermediate data and its movement. For the case where all data except the input images can be stored on an FPGA chip, we present an accelerator for training CNNs to classify the CIFAR-10 dataset. Further, we study the impact of network size on performance and energy of FPGA and GPU. Our accelerator mapped in the Arria 10 FPGA chip obtains up-to 9.33X higher energy efficiency compared to the Nvidia Geforce GTX 1080 Ti GPU at similar performance.