Conditional random fields (CRFs) have been successful in many sequence labeling tasks, which conventionally rely on a hand-craft feature representation of input data. However, a powerful data representation could be another determining factor of the performance, which has not attracted enough attention yet. We describe a novel sequence labeling framework that builds a supervised CRF and an unsuper-vised dynamic model on a shared nonlinear feature transformation neural network. The model could be used for transfer learning by jointly optimizing two learning tasks together. We demonstrate the effectiveness of the proposed modeling framework using synthetic data. We also show that this model yields a significant improvement of recognition accuracy over conventional CRFs on gesture recognition tasks.