We propose to address the Inverse Halftone Colorization task, which tries to recover colorful images from black and white halftone prints, and can be treated as the joint problem of inverse halftone and colorization. Although the inverse halftone colorization seems to be achievable via applying inverse halftone followed by colorization, our proposed method advances from two perspectives: (1) we empirically discover that the orders of cascading inverse halftone and colorization (i.e first inverse halftone then colorization versus the reverse order) would lead to results with different properties, hence a fusion scheme is proposed to integrate their results; (2) we introduce several novel losses to encourage the realness, diversity, and the structural coherence of the colorization. Moreover, our model is flexible to support both exemplar-based and random colorization. We conduct extensive experiments to demonstrate the efficacy of our method as well as verify the contributions of our design choices.