Classifiers Guided Controllable Text Generation for Discrete Diffusion Language Models

Abstract

The classifier-guided method is a simple yet effective technique for controllable text generation. However, its application to the discrete diffusion model, a promising non-autoregressive language model, remains unexplored. In this paper, controllable generation is achieved through multiple iterations of conditional backward diffusion, each iteration comprising two key steps: conditional prediction and masking. The conditional prediction distribution is approximated by the product of classifier probabilities and language model probabilities. Two distinct algorithms, Largest-M and gradient search, are introduced to sample from this joint distribution. Largest-M operates within the discrete token sequence space, while gradient search functions within the continuous hidden state space. Both algorithms aim to identify texts that maximize the joint probability. Experiments across four fine-grained controllable tasks demonstrate the effectiveness of the proposed algorithms, achieving success rates up to 99% on certain tasks. Furthermore, the inherent multi-modality problem in discrete diffusion models is mitigated by framing multi-modality suppression as a controllable generation task. The experiment result indicates that integrating multi-modality classifiers effectively enhances the performance of discrete diffusion models.