Abstract
The performance of many efficient Deep semi-supervised learning(SSL) is severely degraded when the distribution of unlabeled and labeled data does not match. Some recent approaches have chosen to weaken or even remove out-of-distribution (OOD) data, which can lose the potential value of OOD data. We propose TextSMatch to solve this issue, a simple, safe and effective SSL method for text classification, which recycles the OOD data near the labeled domain to make full use of the information in OOD data. Specifically, adversarial domain adaptation is applied to the OOD data to project it into the space of ID and labeled data, and its recoverability is assessed using the use of migration probabilities. Moreover, TextSMatch unifies the mainstream methods. In addition to consistency regularization training of class probabilities for unlabeled data and its augmented data, we also normalized the structure of embedding with contrastive learning based on pseudo-labeled. TextSMatch performs significantly better than other baseline methods on AG News and Yelp datasets in scenarios such as class mismatch and different amounts of labeled data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Jia, L., Zhang, Z., Wang, L., Jiang, W., Zhao, M.: Adaptive neighborhood propagation by joint l2, 1-norm regularized sparse coding for representation and classification. In: 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 201–210. IEEE (2016)
Zhang, H., Zhang, Z., Zhao, M., Ye, Q., Zhang, M., Wang, M.: Robust triple-matrix-recovery-based auto-weighted label propagation for classification. IEEE Trans. Neural Networks Learn. Syst. 31(11), 4538–4552 (2020)
Zhang, Z., Li, F., Jia, L., Qin, J., Zhang, L., Yan, S.: Robust adaptive embedded label propagation with weight learning for inductive classification. IEEE Trans. Neural Networks Learn. Syst. 29(8), 3388–3403 (2017)
Sajjadi, M., Javanmardi, M., Tasdizen, T.: Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In: NIPS, pp. 1163–1171 (2016)
Xie, Q., Dai, Z., Hovy, E., Luong, M.-T., Le, Q.V.: Unsupervised data augmentation for consistency training. In: NIPS (2020)
Lee, D.-H., et al.: Pseudo-label: the simple and efficient semi-supervised learning method for deep neural networks. In: Workshop on Challenges in Representation Learning, ICML, volume 3 (2013)
Berthelot, D., et al.: Mixmatch: a holistic approach to semi-supervised learning. In: NIPS, pp. 5050–5060 (2019)
Berthelot, D., et al.: Semi-supervised learning with distribution alignment and augmentation anchoring. In: ICLR, Remixmatch (2020)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607. PMLR (2020)
He, K., Fan, H., Wu,, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738 (2020)
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. In: NIPS (2020)
Chen, J., Yang, Z., Yang, D.: Mixtext: linguistically-informed interpolation of hidden space for semi-supervised text classification. In: ACL, pp. 2147–2157 (2020)
Tarvainen, A., Valpola, H.: Mean teachers are better role models: weight-averaged consistency targets improve semi-supervised deep learning results. In: ICLR, Workshop Track Proceedings (2017)
Miyato, T., Maeda, S., Koyama, M., Ishii, S.: Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell. 41(8), 1979–1993 (2018)
Oliver, A., Odena, A., Raffel, C., Cubuk, E.D., Goodfellow, I.J.: Realistic evaluation of deep semi-supervised learning algorithms. arXiv preprint arXiv:1804.09170 (2018)
Chen, Y., Zhu, X., Li, W., Gong, S.: Semi-supervised learning under class distribution mismatch. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 3569–3576 (2020)
Guo, L.-Z., Zhang, Z.-Y., Jiang, Y., Li, Y.-F., Zhou, Z.-H.: Safe deep semi-supervised learning for unseen-class unlabeled data. In: International Conference on Machine Learning, pp. 3897–3906. PMLR (2020)
Yang, X., Song, Z., King, I., Xu, Z.: A survey on deep semi-supervised learning. arXiv preprint arXiv:2103.00550 (2021)
Cubuk, E.D., Zoph, B., Shlens, J., Le, Q.V.: Randaugment: practical data augmentation with no separate search. arXiv preprint arXiv:1909.13719, 2(4), 7 (2019)
Edunov, S., Ott, M., Auli, M., Grangier, D.: Understanding back-translation at scale. arXiv preprint arXiv:1808.09381 (2018)
Laine, S., Aila, T.: Temporal ensembling for semi-supervised learning. In: ICLR (2017)
Sohn, K., et al.: Simplifying semi-supervised learning with consistency and confidence. In: NIPS, Fixmatch (2020)
Grandvalet, Y., Bengio, Y., et al.: Semi-supervised learning by entropy minimization. In: CAP, pp. 281–296 (2005)
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018)
Peters, M.E., et al.: Deep contextualized word representations. In: NAACL-HLT, pages 2227–2237 (2018)
Liu, Y., et al.: Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
Behrmann, N., Fayyaz, M., Gall, J., Noroozi, M.: Long short view feature decomposition via contrastive video representation learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9244–9253 (2021)
Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136 (2016)
Nguyen, A., Yosinski, J., Clune, J.: Deep neural networks are easily fooled: high confidence predictions for unrecognizable images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 427–436. IEEE Computer Society (2015)
Liang, S., Sun, R., Li, Y., Srikant, R.: Understanding the loss surface of neural networks for binary classification. In: Dy, J.G., Krause, A. (eds.) International Conference on Machine Learning, pp. 2835–2843. PMLR, PMLR (2018)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
Liang, S., Li, Y., Srikant, R.: Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.02690 (2017)
Lotfollahi, M., Naghipourfar, M., Theis, F.J., Alexander Wolf, F.: Conditional out-of-distribution generation for unpaired data using transfer vae. Bioinformatics 36(Supplement_2), i610–i617 (2020)
Cai, Z., Ravichandran, A., Maji, S., Fowlkes, C., Tu, Z., Soatto, S.: Exponential moving average normalization for self-supervised and semi-supervised learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 194–203 (2021)
Ganin, Y., et al.: Domain-adversarial training of neural networks. J. Mach. Learn. Res. 17(1), 2030–2096 (2016)
Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: IEEE, pp. 1125–1134 (2017)
Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. In: International Conference on Machine Learning, pp. 1180–1189. PMLR (2015)
Yao, Y., Deng, J., Chen, X., Gong, C., Wu, J., Yang, J.: Deep discriminative CNN with temporal ensembling for ambiguously-labeled image classification. In: AAAI, vol. 34, pp. 12669–12676 (2020)
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT, Bert (2019)
Funding
The publication of this paper is funded by NSFC (No. U1711266) and Key-Area Research and Development Program of Guangdong Province (No. 2019B010153001).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Xu, Y., Lin, G., Zeng, N., Qu, Y., Zeng, K. (2022). TextSMatch: Safe Semi-supervised Text Classification with Domain Adaption. In: Zhang, H., et al. Neural Computing for Advanced Applications. NCAA 2022. Communications in Computer and Information Science, vol 1637. Springer, Singapore. https://doi.org/10.1007/978-981-19-6142-7_33
Download citation
DOI: https://doi.org/10.1007/978-981-19-6142-7_33
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-19-6141-0
Online ISBN: 978-981-19-6142-7
eBook Packages: Computer ScienceComputer Science (R0)