This paper proposes an algorithm and hardware co-design methodology to accelerate CNNs for pix2pix tasks. An importance map is introduced to train an activation-sparse CNN model, which can effectively reduce the computing cost and external data transmission. Moreover, the model also supports sparse controlling by means of the importance map, making it adaptive for applications with different precision/power requirements. An FPGA-based accelerator with adaptive sparse controlling is designed to support such importance map guided activation sparsity, and demonstrated for super-resolution (SR) application as an example. The accelerator shows advantages in both model accuracy and power consumption. It achieves up to 765 GOPS/W energy efficiency, which is 5.28× batter than the previous FPGA-based SR accelerator.