As the transistor feature size keeps shrinking, manufacturability has become an urgent issue in semiconductor industry. In order to improve the manufacturability, various resolution enhancement techniques have been proposed, among which layout decomposition and mask optimization have been considered as the most powerful solutions in advanced technology nodes. Different from many previous survey papers that categorize literatures by type of manufacturing process, we argue that different manufacturing scenarios can share similar mathematical models. This paper carefully summarizes a series of methodologies that have been successfully applied to VLSI layout decomposition and mask optimization problems.