Abstract
Document relational network has been effective in retrieving and evaluating papers. Despite their effectiveness, relational measures, including co-citation, are far from ideal and need improvements. The assumption underlying the co-citation relation is the content relevance and opinion relatedness of cited and citing papers. This may imply existence of some kind of co-opinionatedness between co-cited papers which may be effective in improving the measure. Therefore, the present study tries to test the existence of this phenomenon and its role in improving information retrieval. To do so, based on CITREC, a medical test collection was developed consisting of 30 queries (seed documents) and 4823 of their co-cited papers. Using NLP techniques, the co-citances of the queries and their co-cited papers were analyzed and their similarities were computed by 4 g similarity measure. Opinion scores were extracted from co-citances using SentiWordnet. Also, nDCG values were calculated and then compared in terms of the citation proximity index (CPI) and co-citedness measures before and after being normalized by the co-opinionatedness measure. The reliability of the test collection was measured by generalizability theory. The findings suggested that a majority of the co-citations exhibited a high level of co-opinionatedness in that they were mostly similar either in their opinion strengths or in their polarities. Although anti-polar co-citations were not trivial in their number, a significantly higher number of the co-citations were co-polar, with a majority being positive. The evaluation of the normalization of the CPI and co-citedness by the co-opinionatedness indicated a generally significant improvement in retrieval effectiveness. While anti-polar similarity reduced the effectiveness of the measure, the co-polar similarity proved to be effective in improving the co-citedness. Consequently, the co-opinionatedness can be presented as a new document relation and used as a normalization factor to improve retrieval performance and research evaluation.




Similar content being viewed by others
References
Abu-Jbara, A., Ezra, J., & Radev, D. R. (2013). Purpose and polarity of citation: Towards NLP-based bibliometrics. In HLT-NAACL (pp. 596–606).
Agarwal, S., Choubey, L., & Yu, H. (2010). Automatically classifying the role of citations in biomedical articles. In Proceedings of American Medical Informatics Association fall symposium (AMIA), Washington, DC (pp. 11–15).
Amadi, U. P. (2014). Exploiting the role of polarity in citation analysis. Baltimore County: University of Maryland.
Andrejko, A., & Bieliková, M. (2012). Comparing instances of ontological concepts for personalized recommendation in large information spaces. Computing and Informatics, 28(4), 429–452.
Athar, A. (2011). Sentiment analysis of citations using sentence structure-based features. In Proceedings of the ACL 2011 student session (pp. 81–87). Association for Computational Linguistics.
Athar, A. (2014). Sentiment analysis of scientific citations. Technical Report, University of Cambridge, Computer Laboratory,(UCAM-CL-TR-856).
Athar, A., & Teufel, S. (2012). Context-enhanced citation sentiment detection. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 597–601). Association for Computational Linguistics.
Badran, O. M. (1984). An alternative search strategy to improve information retrieval. In Proceedings of the 47th ASIS annual meeting (pp. 137–140).
Bichteler, J., & Eaton, E. A. (1980). The combined use of bibliographic coupling and cocitation for document retrieval. Journal of the American Society for Information Science, 31(4), 278.
Bodoff, D., & Li, P. (2007, July). Test theory for assessing IR test collections. In Proceedings of the 30th annual international ACM SIGIR conference on research and development in information retrieval (pp. 367–374). ACM.
Bornmann, L., & Daniel, H. D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45–80.
Boyack, K. W., Small, H., & Klavans, R. (2013). Improving the accuracy of co-citation clustering using full text. Journal of the American Society for Information Science and Technology, 64(9), 1759–1767.
Brooks, T. A. (1985). Private acts and public objects: An investigation of citer motivations. Journal of the American Society for Information Science, 36(4), 223–229.
Callahan, A., Hockema, S., & Eysenbach, G. (2010). Contextual cocitation: Augmenting cocitation analysis and its applications. Journal of the American Society for Information Science and Technology, 61(6), 1130–1143.
Cavalcanti, D. C., Prudêncio, R. B., Pradhan, S. S., Shah, J. Y., & Pietrobon, R. S. (2011). Good to be bad? Distinguishing between positive and negative citations in scientific impact. In 2011 23rd IEEE international conference on tools with artificial intelligence (ICTAI) (pp. 156–162). IEEE.
Chubin, D. E., & Moitra, S. D. (1975). Content analysis of references: Adjunct or alternative to citation counting? Social Studies of Science, 5(4), 423–441.
Dabrowska, A., & Larsen, B. (2015). Exploiting citation contexts for physics retrieval. In Second workshop on bibliometric-enhanced information retrieval (pp. 14–21).
Ding, Y., Zhang, G., Chambers, T., Song, M., Wang, X., & Zhai, C. (2014). Content-based citation analysis: The next generation of citation analysis. Journal of the association for information science and technology, 65(9), 1820–1833.
Dong, C., & Schäfer, U. (2011). Ensemble-style self-training on citation classification. In IJCNLP (pp. 623–631).
Doslu, M., & Bingol, H. O. (2016). Context sensitive article ranking with citation context analysis. Scientometrics, 108, 653–671.
Egghe, L., & Rousseau, R. (1990). Introduction to informetrics: Quantitative methods in library, documentation and information science. Amsterdam: Elsevier.
Elkiss, A., et al. (2008). Blind men and elephants: What do citation summaries tell us about a research article? Journal of the American Society for Information Science and Technology, 59(1), 51–62.
Esuli, A., & Sebastiani, F. (2007). SentiWordNet: A high-coverage lexical resource for opinion mining. Technical Report ISTI-PP-002/2007, Institute of Information Science and Technologies (ISTI) of the Italian National Research Council (CNR). http://nmis.isti.cnr.it/sebastiani/Publications/2007TR02.pdf.
Eto, M. (2012). Spread co-citation relationship as a measure for document retrieval. In Proceedings of the fifth ACM workshop on research advances in large digital book repositories and complementary media (pp. 7–8). ACM.
Eto, M. (2013). Evaluations of context-based co-citation searching. Scientometrics, 94(2), 651–673.
Eto, M. (2014). Document retrieval method using random walk with restart on weighted co-citation network. Proceedings of the American Society for Information Science and Technology, 51(1), 1–4.
Eto, M. (2015). Combination effects of word-based and extended co-citation search algorithms. In Proceedings of the 15th ACM/IEEE-CS joint conference on digital libraries (pp. 245–246). ACM.
Fujiwara, T., & Yamamoto, Y. (2015). Colil: A database and search service for citation contexts in the life sciences domain. Journal of biomedical semantics, 6(1), 38.
Gipp, B., & Beel, J. (2009). Citation proximity analysis (CPA)-A new approach for identifying related work based on co-citation analysis. In Proceedings of the 12th international conference on scientometrics and informetrics (ISSI’09) (Vol. 2, pp. 571–575). Rio de Janeiro (Brazil): International Society for Scientometrics and Informetrics.
Hamedani, M. R., Kim, S. W., & Kim, D. J. (2016). SimCC: A novel method to consider both content and citations for computing similarity of scientific papers. Information Sciences, 334, 273–292.
Hanney, S., Grant, J., Jones, T., & Buxton, M. (2005). Categorising citations to trace research impact. In Proceedings of the 10th international conference of the international society for scientometrics and informetrics. Stockholm: Karolinska University Press.
Hasanain, M., Suwaileh, R., Elsayed, T., Kutlu, M., & Almerekhi, H. (2018). EveTAR: Building a large-scale multi-task test collection over Arabic tweets. Information Retrieval Journal, 21(4), 307–336.
Hernández-Alvarez, M. Y. R. I. A. M., & Gomez, J. M. (2016). Survey about citation context analysis: Tasks, techniques, and resources. Natural Language Engineering, 22(03), 327–349.
Janssens, A. C. J., & Gwinn, M. (2015). Novel citation-based search method for scientific literature: Application to meta-analyses. BMC Medical Research Methodology, 15(1), 1.
Järvelin, K., & Kekäläinen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS), 20(4), 422–446.
Jeong, Y. K., Song, M., & Ding, Y. (2014). Content-based author co-citation analysis. Journal of Informetrics, 8(1), 197–211.
Jia, M. (2018). Citation function and polarity classification in biomedical papers. Electronic Thesis and Dissertation Repository, 5367.
Jochim, C., & Schütze, H. (2012). Towards a generic and flexible citation classifier based on a faceted classification scheme. In Proceedings of COLING 2012 (pp. 1343–1358).
Kanoulas, E., & Aslam, J. A. (2009). Empirical justification of the gain and discount function for nDCG. In Proceedings of the 18th ACM conference on Information and knowledge management (pp. 611–620). ACM.
Kekäläinen, J. (2005). Binary and graded relevance in IR evaluations—Comparison of the effects on ranking of IR systems. Information Processing and Management, 41(5), 1019–1033.
Leung, P. T., Macdonald, E. M., Stanbrook, M. B., Dhalla, I. A., & Juurlink, D. N. (2017). A 1980 letter on the risk of opioid addiction. New England Journal of Medicine, 376(22), 2194–2195.
Lipetz, B. A. (1965). Improvement of the selectivity of citation indexes to science literature through inclusion of citation relationship indicators. Journal of the Association for Information Science and Technology, 16(2), 81–90.
Liu, B. (2012). Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, 5(1), 1–167.
Liu, B., & Zhang, L. (2012). A survey of opinion mining and sentiment analysis. In Mining text data (pp. 415–463). Boston, MA: Springer.
Liu, S., Chen, C., Ding, K., Wang, B., Xu, K., & Lin, Y. (2014). Literature retrieval based on citation context. Scientometrics, 101(2), 1293–1307.
MacRoberts, M. H., & MacRoberts, B. R. (1984). The negational reference: Or the art of dissembling. Social Studies of Science, 14(1), 91–94.
MacRoberts, M. H., & MacRoberts, B. R. (1989). Problems of citation analysis: A critical review. Journal of the American Society for information Science, 40(5), 342–349.
Mahalakshmi, G. S., Siva, R., & Sendhilkumar, S. (2015). Context based retrieval of scientific publications via reader lens. In Computational intelligence in data mining (Vol. 3, pp. 583–596). Springer India.
Martyn, J. (1964). Bibliographic coupling. Journal of Documentation, 20(4), 236.
Matosin, N., Frank, E., Engel, M., Lum, J. S., & Newell, K. A. (2014). Negativity towards negative results: A discussion of the disconnect between scientific worth and scientific culture. Disease Models & Mechanisms, 7:171–173.
Moravcsik, M. J., & Murugesan, P. (1975). Some results on the function and quality of citations. Social Studies of Science, 5(1), 86–92.
Nakov, P. I., Schwartz, A. S., & Hearst, M. (2004). Citances: Citation sentences for semantic analysis of bioscience text. In Proceedings of the SIGIR’04 workshop on search and discovery in bioinformatics (pp. 81–88).
Parthasarathy, G., & Tomar, D. C. (2014). Sentiment analyzer: Analysis of journal citations from citation databases. In 2014 5th international conference- confluence the next generation information technology summit (confluence) (pp. 923–928). IEEE.
Parthasarathy, G., & Tomar, D. C. (2015). A survey of sentiment analysis for journal citation. Indian Journal of Science and Technology. https://doi.org/10.17485/ijst/2015/v8i35/55134.
Piao, S., Ananiadou, S., Tsuruoka, Y., Sasaki, Y., & McNaught, J. (2007). Mining opinion polarity relations of citations. In International workshop on computational semantics (IWCS) (pp. 366–371).
Ritchie, A., Robertson, S., & Teufel, S. (2008). Comparing citation contexts for information retrieval. In Proceedings of the 17th ACM conference on Information and knowledge management (pp. 213–222). ACM.
Saraçoğlu, R., Tütüncü, K., & Allahverdi, N. (2007). A fuzzy clustering approach for finding similar documents using a novel similarity measure. Expert Systems with Applications, 33(3), 600–605.
Schafer, U., & Spurk, C. (2010). TAKE scientist’s workbench: semantic search and citation-based visual navigation in scholar papers. In 2010 IEEE fourth international conference on semantic computing (ICSC) (pp. 317–324). IEEE.
Segaran, T. (2007). Programming collective intelligence: Building smart web 2.0 applications. Beijing: O’Reilly Media, Inc.
Sendhilkumar, S., Elakkiya, E., & Mahalakshmi, G. S. (2013). Citation semantic based approaches to identify article quality. In Proceedings of international conference ICCSEA (pp. 411–420).
Shuy, R. W. (2003). 22 Discourse analysis in the legal context. The Handbook of Discourse Analysis, 18, 437.
Small, H. (1973). Co-citation in the scientific literature: A new measure of the relationship between two documents. Journal of the American Society for information Science, 24(4), 265–269.
Small, H. (1982). Citation context analysis. Progress in Communication Sciences, 3, 287–310.
Small, H. (2011). Interpreting maps of science using citation context sentiments: A preliminary investigation. Scientometrics, 87(2), 373–388.
Smith, L. C. (1981). Citation analysis. Library Trends, 30(1), 83–106.
Su, M. C., & Chou, C. H. (2001). A modified version of the K-means algorithm with a distance based on cluster symmetry. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(6), 674–680.
Teufel, S., Siddharthan, A., & Tidhar, D. (2006). Automatic classification of citation function. In Proceedings of the 2006 conference on empirical methods in natural language processing (pp. 103–110). Association for Computational Linguistics.
Teufel, S., Siddharthan, A., & Tidhar, D. (2009). An annotation scheme for citation function. In Proceedings of the 7th SIGdial workshop on discourse and dialogue (pp. 80-87). Association for Computational Linguistics.
Urbano, J., Marrero, M., & Martín, D. (2013). On the measurement of test collection reliability. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval (pp. 393–402). ACM.
Wang, Y., Wang, L., Li, Y., He, D., Chen, W., & Liu, T. Y. (2013, April). A theoretical analysis of NDCG ranking measures. In Proceedings of the 26th annual conference on learning theory (COLT 2013) (Vol. 8).
White, H. D. (2016). Bag of works retrieval: TF* IDF weighting of co-cited works. In BIR@ ECIR (pp. 63–72).
Yoon, S. H., Kim, S. W., & Park, S. (2016). C-Rank: A link-based similarity measure for scientific literature databases. Information Sciences, 326, 25–40.
Yu, B. (2013). Automated citation sentiment analysis: What can we learn from biomedical researchers. Proceedings of the American Society for Information Science and Technology, 50(1), 1–9.
Zhao, H. (2014). Sharding for literature search via cutting citation graphs. In 2014 IEEE international conference on Big Data (Big Data) (pp. 77–79). IEEE.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Yaghtin, M., Sotudeh, H., Mirzabeigi, M. et al. In quest of new document relations: evaluating co-opinion relations between co-citations and its impact on Information retrieval effectiveness. Scientometrics 119, 987–1008 (2019). https://doi.org/10.1007/s11192-019-03058-3
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-019-03058-3