Cloudy pixels in satellite images degrade the visibility of captured surface structure. We propose a novel cloudy image synthesis model and develop a cloud removal algorithm using convolutional neural network. We extract the cloud masks from real cloudy satellite images and from real sky images with clouds. Then we investigate the characteristics of real cloudy images and devise a reliable cloudy image synthesis model which considers the background surface color, misalignement of channel images, and blur in clouds. We train a hierarchical cloud removal network using the synthetic cloudy images. Experimental results demonstrate that the proposed algorithm removes the clouds from cloudy satellite images faithfully and outperforms the existing methods.