Abstract
The accessibility to the Big Data platform today has raised hopes to analyse the large volume of online documents and to make sense of them quickly, which has provided further impetus for automated unsupervised text summarisation. In this regards, Latent Dirichlet Allocation (LDA) is a popular topic model based text summarisation method. However, the generated LDA topic word summaries contain many redundant words, i.e., duplicates and morphological variants. We hypothesise that duplicate words do not improve the content quality of summary, but for good reasons, the morphological variants do. The work sets out to investigate this hypothesis. Consequently, a unique LDA summary of significant topic words is constituted from the LDA summary by removing the duplicate words, but retaining the distinctive morphological variants. The divergence probability of the unique LDA summary is compared against the LDA baseline summary of the same size. Short summaries of 0.67% and 2.0% of the full text size of the input documents are evaluated. Our findings show that the content quality of unique LDA summary is no better than its corresponding LDA baseline summary. However, if the duplicate words are removed from the baseline summary, producing a compressed version of itself with unique words, i.e., a unique LDA baseline summary; and, if the compression ratio is taken into consideration, it will appear that the content quality of a LDA summary constituted using unique significant words have indeed improved.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Blei, D., Ng, A., Jordan, M.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)
Lee, S., Song, J., Kim, Y.: An empirical comparison of four text mining methods. J. Comput. Inf. 51(1), 1–10 (2010)
Annamalai, M., Mukhlis, S.F.N.: Content quality of clustered latent Dirichlet allocation short summaries. In: Jaafar, A., Mohamad Ali, N., Mohd Noah, S.A., Smeaton, A.F., Bruza, P., Bakar, Z.A., Jamil, N., Sembok, T.M.T. (eds.) AIRS 2014. LNCS, vol. 8870, pp. 494–504. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-12844-3_42
Dopichaj, P., Harder, T.: Conflation method and spelling mistakes – a sensitivity analysis in information retrieval. Citeseer (2004)
Sirsat, S.R., Chavan, V., Mahalle, H.S.: Strength and accuracy analysis of affix removal stemming algorithms. Int. J. Comput. Sci. Inf. Technol.gies 4(2), 265–269 (2013)
Hobson, S.F.: Text summarization evaluation: correlation human performance on an extrinsic task with automatic intrinsic metrics. Dissertation, University of Maryland, College Park (2007)
Kullback, S.: The Kullback-Leibler distance. Am. Stat. 41(4), 340–341 (1987)
McCallum A.K.: MALLET: a machine learning for language toolkit (2002). http://mallet.cs.umass.edu
Saranyamol, C., Sindhu, L.: A survey on automatic text summarisation. Int. J. Comput. Sci. Inf. Technol. 5, 7889–7893 (2014)
Hassel, M.: Evaluation of automatic text summarization: a practical implementation. Licentiate Thesis, University of Stockholm, Sweden (2004)
Steinberger, J., Jezek, K.: Evaluation measures for text summarization. Comput. Inform. 28, 1001–1026 (2007)
Lloret, E., Palomar, M.: Text summarisation in progress: a literature review. Artif. Intell. Rev. 37, 1–41 (2011). Springer Science+Business Media B.V.
Louis, A., Nenkova, A.: Automatically evaluating content selection in summarization without human models. In: Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 306–314 (2009)
Manning, C.D., Surdeanu, M., Bauer, J.F., Jenny, B., Steven, J., McClosky, D.: The Stanford CoreNLP natural language processing toolkit. In: 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 55–60 (2014)
Acknowledgement
The authors thankfully acknowledge Universiti Teknologi MARA (UiTM) for support of this work, which was funded under the Malaysian Ministry of Higher Education’s Fundamental Research Grant Scheme (ref. no FRGS/1/2016/ICT01/UITM/02/3).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Annamalai, M., Narawi, A.M. (2017). Content Quality of Latent Dirichlet Allocation Summaries Constituted Using Unique Significant Words. In: Mohamed, A., Berry, M., Yap, B. (eds) Soft Computing in Data Science. SCDS 2017. Communications in Computer and Information Science, vol 788. Springer, Singapore. https://doi.org/10.1007/978-981-10-7242-0_26
Download citation
DOI: https://doi.org/10.1007/978-981-10-7242-0_26
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-7241-3
Online ISBN: 978-981-10-7242-0
eBook Packages: Computer ScienceComputer Science (R0)