Abstract
The detection of fake news on social networks is highly desirable and socially beneficial. In real scenarios, there are few labeled news articles and a large number of unlabeled articles. One prominent way is to consider fake news detection as a few-shot learning task. However, existing few-shot learning methods suffer from two limitations when they are used to detect fake news. First, they may not directly learn linguistic knowledge from both labeled and unlabeled news articles. Second, few training articles usually lead to over-fitting of the model. In this paper, we propose a novel Few-Shot Knowledge Distillation (FSKD) model which is based on the student-teacher framework. BERT is fine-tuned on a few labeled news articles and used as the teacher model which provides soft labels of unlabeled articles. Then a few labeled news articles and a lot of unlabeled articles with soft labels are used to train a student model via knowledge distillation. An optimization algorithm is proposed to take advantage of the large-scale news articles and alleviate over-fitting simultaneously. Experimental results on real-world public datasets demonstrate that the proposed model can achieve superior performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ba, J., Caruana, R.: Do deep nets really need to be deep? In: NeurIPS, pp. 2654–2662 (2014)
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT, pp. 4171–4186 (2019)
Douze, M., Szlam, A., Hariharan, B., Jégou, H.: Low-shot learning with large-scale diffusion. In: CVPR, pp. 3349–3358 (2018)
Geng, R., Li, B., Li, Y., Zhu, X., Jian, P., Sun, J.: Induction networks for few-shot text classification. In: EMNLP-IJCNLP, pp. 3902–3911 (2019)
Gou, J., Yu, B., Maybank, S.J., Tao, D.: Knowledge distillation: a survey. Int. J. Comput. Vis. 129(6), 1789–1819 (2021)
Gu, J., Wang, Y., Chen, Y., Li, V.O.K., Cho, K.: Meta-learning for low-resource neural machine translation. In: EMNLP, pp. 3622–3631 (2018)
Hinton, G.E., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. CoRR abs/1503.02531 (2015)
Jiao, X., et al.: Tinybert: Distilling BERT for natural language understanding. In: EMNLP, pp. 4163–4174 (2020)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2015)
Kwon, S., Cha, M., Jung, K., Chen, W., Wang, Y.: Prominent features of rumor propagation in online social media. In: ICDM, pp. 1103–1108 (2013)
Liu, Y., Wu, Y.B.: Early detection of fake news on social media through propagation path classification with recurrent and convolutional networks. In: AAAI pp. 354–361 (2018)
Ma, J., et al.: Detecting rumors from microblogs with recurrent neural networks. In: IJCAI, pp. 3818–3824 (2016)
Ma, J., Gao, W., Wong, K.: Rumor detection on twitter with tree-structured recursive neural networks. In: ACL, pp. 1980–1989 (2018)
Miyato, T., Maeda, S., Koyama, M., Ishii, S.: Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell. 41(8), 1979–1993 (2019)
Ren, X., Shi, R., Li, F.: Distill BERT to traditional models in Chinese machine reading comprehension (student abstract). In: AAAI, pp. 13901–13902 (2020)
Rios, A., Kavuluru, R.: Few-shot and zero-shot multi-label learning for structured label spaces. In: EMNLP, pp. 3132–3142 (2018)
Ruchansky, N., Seo, S., Liu, Y.: CSI: A hybrid deep model for fake news detection. In: CIKM, pp. 797–806 (2017)
Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR abs/1910.01108 (2019)
Shu, K., Mahudeswaran, D., Wang, S., Lee, D., Liu, H.: Fakenewsnet: a data repository with news content, social context and dynamic information for studying fake news on social media. CoRR abs/1809.01286 (2018)
Shu, K., Sliva, A., Wang, S., Tang, J., Liu, H.: Fake news detection on social media: a data mining perspective. SIGKDD Explor. 19(1), 22–36 (2017)
Sun, S., Cheng, Y., Gan, Z., Liu, J.: Patient knowledge distillation for BERT model compression. In: EMNLP-IJCNLP, pp. 4322–4331 (2019)
Sung, F., Yang, Y., Zhang, L., Xiang, T., Torr, P.H.S., Hospedales, T.M.: Learning to compare: relation network for few-shot learning. In: CVPR, pp. 1199–1208 (2018)
Wang, W.Y.: "liar, liar pants on fire": a new benchmark dataset for fake news detection. In: ACL, pp. 422–426 (2017)
Wang, Y., Yao, Q., Kwok, J.T., Ni, L.M.: Generalizing from a few examples: a survey on few-shot learning. ACM Comput. Surv. 53(3), 63:1–63:34 (2020)
Wu, Y., Lin, Y., Dong, X., Yan, Y., Ouyang, W., Yang, Y.: Exploit the unknown gradually: one-shot video-based person re-identification by stepwise learning. In: CVPR, pp. 5177–5186 (2018)
Yang, F., Liu, Y., Yu, X., Yang, M.: Automatic detection of rumor on sina weibo. In: SIGKDD Workshop (2012)
Yang, S., Shu, K., Wang, S., Gu, R., Wu, F., Liu, H.: Unsupervised fake news detection on social media: A generative approach. In: AAAI, pp. 5644–5651 (2019)
Yu, M., et al.: Diverse few-shot text classification with multiple metrics. In: NAACL-HLT, pp. 1206–1215 (2018)
Acknowledgement
This work is partially supported by NFSC-General Technology Joint Fund for Basic Research (No.U1936206) and the National Natural Science Foundation of China (No. 62172237, 62077031.). We thank the AC, SPC, PC and reviewers for their insightful comments on this paper.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Yuan, J., Chen, C., Hou, C., Yuan, X. (2023). FSKD: Detecting Fake News with Few-Shot Knowledge Distillation. In: Yang, X., et al. Advanced Data Mining and Applications. ADMA 2023. Lecture Notes in Computer Science(), vol 14180. Springer, Cham. https://doi.org/10.1007/978-3-031-46677-9_29
Download citation
DOI: https://doi.org/10.1007/978-3-031-46677-9_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-46676-2
Online ISBN: 978-3-031-46677-9
eBook Packages: Computer ScienceComputer Science (R0)