Skip to main content
Springer Nature Link
Log in

In quest of new document relations: evaluating co-opinion relations between co-citations and its impact on Information retrieval effectiveness

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Document relational network has been effective in retrieving and evaluating papers. Despite their effectiveness, relational measures, including co-citation, are far from ideal and need improvements. The assumption underlying the co-citation relation is the content relevance and opinion relatedness of cited and citing papers. This may imply existence of some kind of co-opinionatedness between co-cited papers which may be effective in improving the measure. Therefore, the present study tries to test the existence of this phenomenon and its role in improving information retrieval. To do so, based on CITREC, a medical test collection was developed consisting of 30 queries (seed documents) and 4823 of their co-cited papers. Using NLP techniques, the co-citances of the queries and their co-cited papers were analyzed and their similarities were computed by 4 g similarity measure. Opinion scores were extracted from co-citances using SentiWordnet. Also, nDCG values were calculated and then compared in terms of the citation proximity index (CPI) and co-citedness measures before and after being normalized by the co-opinionatedness measure. The reliability of the test collection was measured by generalizability theory. The findings suggested that a majority of the co-citations exhibited a high level of co-opinionatedness in that they were mostly similar either in their opinion strengths or in their polarities. Although anti-polar co-citations were not trivial in their number, a significantly higher number of the co-citations were co-polar, with a majority being positive. The evaluation of the normalization of the CPI and co-citedness by the co-opinionatedness indicated a generally significant improvement in retrieval effectiveness. While anti-polar similarity reduced the effectiveness of the measure, the co-polar similarity proved to be effective in improving the co-citedness. Consequently, the co-opinionatedness can be presented as a new document relation and used as a normalization factor to improve retrieval performance and research evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. https://www.isg.uni-konstanz.de/projects/citrec/.

  2. http://colil.dbcls.jp/browse/papers/.

  3. http://sentiwordnet.isti.cnr.it/.

  4. https://rdrr.io/cran/gt4ireval/src/R/gstudy.R.

References

  • Abu-Jbara, A., Ezra, J., & Radev, D. R. (2013). Purpose and polarity of citation: Towards NLP-based bibliometrics. In HLT-NAACL (pp. 596–606).

  • Agarwal, S., Choubey, L., & Yu, H. (2010). Automatically classifying the role of citations in biomedical articles. In Proceedings of American Medical Informatics Association fall symposium (AMIA), Washington, DC (pp. 11–15).

  • Amadi, U. P. (2014). Exploiting the role of polarity in citation analysis. Baltimore County: University of Maryland.

    Google Scholar 

  • Andrejko, A., & Bieliková, M. (2012). Comparing instances of ontological concepts for personalized recommendation in large information spaces. Computing and Informatics, 28(4), 429–452.

    Google Scholar 

  • Athar, A. (2011). Sentiment analysis of citations using sentence structure-based features. In Proceedings of the ACL 2011 student session (pp. 81–87). Association for Computational Linguistics.

  • Athar, A. (2014). Sentiment analysis of scientific citations. Technical Report, University of Cambridge, Computer Laboratory,(UCAM-CL-TR-856).

  • Athar, A., & Teufel, S. (2012). Context-enhanced citation sentiment detection. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 597–601). Association for Computational Linguistics.

  • Badran, O. M. (1984). An alternative search strategy to improve information retrieval. In Proceedings of the 47th ASIS annual meeting (pp. 137–140).

  • Bichteler, J., & Eaton, E. A. (1980). The combined use of bibliographic coupling and cocitation for document retrieval. Journal of the American Society for Information Science, 31(4), 278.

    Article  Google Scholar 

  • Bodoff, D., & Li, P. (2007, July). Test theory for assessing IR test collections. In Proceedings of the 30th annual international ACM SIGIR conference on research and development in information retrieval (pp. 367–374). ACM.

  • Bornmann, L., & Daniel, H. D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45–80.

    Article  Google Scholar 

  • Boyack, K. W., Small, H., & Klavans, R. (2013). Improving the accuracy of co-citation clustering using full text. Journal of the American Society for Information Science and Technology, 64(9), 1759–1767.

    Article  Google Scholar 

  • Brooks, T. A. (1985). Private acts and public objects: An investigation of citer motivations. Journal of the American Society for Information Science, 36(4), 223–229.

    Article  Google Scholar 

  • Callahan, A., Hockema, S., & Eysenbach, G. (2010). Contextual cocitation: Augmenting cocitation analysis and its applications. Journal of the American Society for Information Science and Technology, 61(6), 1130–1143.

    Google Scholar 

  • Cavalcanti, D. C., Prudêncio, R. B., Pradhan, S. S., Shah, J. Y., & Pietrobon, R. S. (2011). Good to be bad? Distinguishing between positive and negative citations in scientific impact. In 2011 23rd IEEE international conference on tools with artificial intelligence (ICTAI) (pp. 156–162). IEEE.

  • Chubin, D. E., & Moitra, S. D. (1975). Content analysis of references: Adjunct or alternative to citation counting? Social Studies of Science, 5(4), 423–441.

    Article  Google Scholar 

  • Dabrowska, A., & Larsen, B. (2015). Exploiting citation contexts for physics retrieval. In Second workshop on bibliometric-enhanced information retrieval (pp. 14–21).

  • Ding, Y., Zhang, G., Chambers, T., Song, M., Wang, X., & Zhai, C. (2014). Content-based citation analysis: The next generation of citation analysis. Journal of the association for information science and technology, 65(9), 1820–1833.

    Article  Google Scholar 

  • Dong, C., & Schäfer, U. (2011). Ensemble-style self-training on citation classification. In IJCNLP (pp. 623–631).

  • Doslu, M., & Bingol, H. O. (2016). Context sensitive article ranking with citation context analysis. Scientometrics, 108, 653–671.

    Article  Google Scholar 

  • Egghe, L., & Rousseau, R. (1990). Introduction to informetrics: Quantitative methods in library, documentation and information science. Amsterdam: Elsevier.

    Google Scholar 

  • Elkiss, A., et al. (2008). Blind men and elephants: What do citation summaries tell us about a research article? Journal of the American Society for Information Science and Technology, 59(1), 51–62.

    Article  Google Scholar 

  • Esuli, A., & Sebastiani, F. (2007). SentiWordNet: A high-coverage lexical resource for opinion mining. Technical Report ISTI-PP-002/2007, Institute of Information Science and Technologies (ISTI) of the Italian National Research Council (CNR). http://nmis.isti.cnr.it/sebastiani/Publications/2007TR02.pdf.

  • Eto, M. (2012). Spread co-citation relationship as a measure for document retrieval. In Proceedings of the fifth ACM workshop on research advances in large digital book repositories and complementary media (pp. 7–8). ACM.

  • Eto, M. (2013). Evaluations of context-based co-citation searching. Scientometrics, 94(2), 651–673.

    Article  Google Scholar 

  • Eto, M. (2014). Document retrieval method using random walk with restart on weighted co-citation network. Proceedings of the American Society for Information Science and Technology, 51(1), 1–4.

    Article  MathSciNet  Google Scholar 

  • Eto, M. (2015). Combination effects of word-based and extended co-citation search algorithms. In Proceedings of the 15th ACM/IEEE-CS joint conference on digital libraries (pp. 245–246). ACM.

  • Fujiwara, T., & Yamamoto, Y. (2015). Colil: A database and search service for citation contexts in the life sciences domain. Journal of biomedical semantics, 6(1), 38.

    Article  Google Scholar 

  • Gipp, B., & Beel, J. (2009). Citation proximity analysis (CPA)-A new approach for identifying related work based on co-citation analysis. In Proceedings of the 12th international conference on scientometrics and informetrics (ISSI’09) (Vol. 2, pp. 571–575). Rio de Janeiro (Brazil): International Society for Scientometrics and Informetrics.

  • Hamedani, M. R., Kim, S. W., & Kim, D. J. (2016). SimCC: A novel method to consider both content and citations for computing similarity of scientific papers. Information Sciences, 334, 273–292.

    Article  Google Scholar 

  • Hanney, S., Grant, J., Jones, T., & Buxton, M. (2005). Categorising citations to trace research impact. In Proceedings of the 10th international conference of the international society for scientometrics and informetrics. Stockholm: Karolinska University Press.

  • Hasanain, M., Suwaileh, R., Elsayed, T., Kutlu, M., & Almerekhi, H. (2018). EveTAR: Building a large-scale multi-task test collection over Arabic tweets. Information Retrieval Journal, 21(4), 307–336.

    Article  Google Scholar 

  • Hernández-Alvarez, M. Y. R. I. A. M., & Gomez, J. M. (2016). Survey about citation context analysis: Tasks, techniques, and resources. Natural Language Engineering, 22(03), 327–349.

    Article  Google Scholar 

  • Janssens, A. C. J., & Gwinn, M. (2015). Novel citation-based search method for scientific literature: Application to meta-analyses. BMC Medical Research Methodology, 15(1), 1.

    Article  Google Scholar 

  • Järvelin, K., & Kekäläinen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS), 20(4), 422–446.

    Article  Google Scholar 

  • Jeong, Y. K., Song, M., & Ding, Y. (2014). Content-based author co-citation analysis. Journal of Informetrics, 8(1), 197–211.

    Article  Google Scholar 

  • Jia, M. (2018). Citation function and polarity classification in biomedical papers. Electronic Thesis and Dissertation Repository, 5367.

  • Jochim, C., & Schütze, H. (2012). Towards a generic and flexible citation classifier based on a faceted classification scheme. In Proceedings of COLING 2012 (pp. 1343–1358).

  • Kanoulas, E., & Aslam, J. A. (2009). Empirical justification of the gain and discount function for nDCG. In Proceedings of the 18th ACM conference on Information and knowledge management (pp. 611–620). ACM.

  • Kekäläinen, J. (2005). Binary and graded relevance in IR evaluations—Comparison of the effects on ranking of IR systems. Information Processing and Management, 41(5), 1019–1033.

    Article  Google Scholar 

  • Leung, P. T., Macdonald, E. M., Stanbrook, M. B., Dhalla, I. A., & Juurlink, D. N. (2017). A 1980 letter on the risk of opioid addiction. New England Journal of Medicine, 376(22), 2194–2195.

    Article  Google Scholar 

  • Lipetz, B. A. (1965). Improvement of the selectivity of citation indexes to science literature through inclusion of citation relationship indicators. Journal of the Association for Information Science and Technology, 16(2), 81–90.

    Google Scholar 

  • Liu, B. (2012). Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, 5(1), 1–167.

    Article  Google Scholar 

  • Liu, B., & Zhang, L. (2012). A survey of opinion mining and sentiment analysis. In Mining text data (pp. 415–463). Boston, MA: Springer.

  • Liu, S., Chen, C., Ding, K., Wang, B., Xu, K., & Lin, Y. (2014). Literature retrieval based on citation context. Scientometrics, 101(2), 1293–1307.

    Article  Google Scholar 

  • MacRoberts, M. H., & MacRoberts, B. R. (1984). The negational reference: Or the art of dissembling. Social Studies of Science, 14(1), 91–94.

    Article  Google Scholar 

  • MacRoberts, M. H., & MacRoberts, B. R. (1989). Problems of citation analysis: A critical review. Journal of the American Society for information Science, 40(5), 342–349.

    Article  Google Scholar 

  • Mahalakshmi, G. S., Siva, R., & Sendhilkumar, S. (2015). Context based retrieval of scientific publications via reader lens. In Computational intelligence in data mining (Vol. 3, pp. 583–596). Springer India.

  • Martyn, J. (1964). Bibliographic coupling. Journal of Documentation, 20(4), 236.

    Article  Google Scholar 

  • Matosin, N., Frank, E., Engel, M., Lum, J. S., & Newell, K. A. (2014). Negativity towards negative results: A discussion of the disconnect between scientific worth and scientific culture. Disease Models & Mechanisms, 7:171–173.

    Article  Google Scholar 

  • Moravcsik, M. J., & Murugesan, P. (1975). Some results on the function and quality of citations. Social Studies of Science, 5(1), 86–92.

    Article  Google Scholar 

  • Nakov, P. I., Schwartz, A. S., & Hearst, M. (2004). Citances: Citation sentences for semantic analysis of bioscience text. In Proceedings of the SIGIR’04 workshop on search and discovery in bioinformatics (pp. 81–88).

  • Parthasarathy, G., & Tomar, D. C. (2014). Sentiment analyzer: Analysis of journal citations from citation databases. In 2014 5th international conference- confluence the next generation information technology summit (confluence) (pp. 923–928). IEEE.

  • Parthasarathy, G., & Tomar, D. C. (2015). A survey of sentiment analysis for journal citation. Indian Journal of Science and Technology. https://doi.org/10.17485/ijst/2015/v8i35/55134.

    Article  Google Scholar 

  • Piao, S., Ananiadou, S., Tsuruoka, Y., Sasaki, Y., & McNaught, J. (2007). Mining opinion polarity relations of citations. In International workshop on computational semantics (IWCS) (pp. 366–371).

  • Ritchie, A., Robertson, S., & Teufel, S. (2008). Comparing citation contexts for information retrieval. In Proceedings of the 17th ACM conference on Information and knowledge management (pp. 213–222). ACM.

  • Saraçoğlu, R., Tütüncü, K., & Allahverdi, N. (2007). A fuzzy clustering approach for finding similar documents using a novel similarity measure. Expert Systems with Applications, 33(3), 600–605.

    Article  Google Scholar 

  • Schafer, U., & Spurk, C. (2010). TAKE scientist’s workbench: semantic search and citation-based visual navigation in scholar papers. In 2010 IEEE fourth international conference on semantic computing (ICSC) (pp. 317–324). IEEE.

  • Segaran, T. (2007). Programming collective intelligence: Building smart web 2.0 applications. Beijing: O’Reilly Media, Inc.

    Google Scholar 

  • Sendhilkumar, S., Elakkiya, E., & Mahalakshmi, G. S. (2013). Citation semantic based approaches to identify article quality. In Proceedings of international conference ICCSEA (pp. 411–420).

  • Shuy, R. W. (2003). 22 Discourse analysis in the legal context. The Handbook of Discourse Analysis, 18, 437.

    Google Scholar 

  • Small, H. (1973). Co-citation in the scientific literature: A new measure of the relationship between two documents. Journal of the American Society for information Science, 24(4), 265–269.

    Article  MathSciNet  Google Scholar 

  • Small, H. (1982). Citation context analysis. Progress in Communication Sciences, 3, 287–310.

    Google Scholar 

  • Small, H. (2011). Interpreting maps of science using citation context sentiments: A preliminary investigation. Scientometrics, 87(2), 373–388.

    Article  Google Scholar 

  • Smith, L. C. (1981). Citation analysis. Library Trends, 30(1), 83–106.

    Google Scholar 

  • Su, M. C., & Chou, C. H. (2001). A modified version of the K-means algorithm with a distance based on cluster symmetry. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(6), 674–680.

    Article  Google Scholar 

  • Teufel, S., Siddharthan, A., & Tidhar, D. (2006). Automatic classification of citation function. In Proceedings of the 2006 conference on empirical methods in natural language processing (pp. 103–110). Association for Computational Linguistics.

  • Teufel, S., Siddharthan, A., & Tidhar, D. (2009). An annotation scheme for citation function. In Proceedings of the 7th SIGdial workshop on discourse and dialogue (pp. 80-87). Association for Computational Linguistics.

  • Urbano, J., Marrero, M., & Martín, D. (2013). On the measurement of test collection reliability. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval (pp. 393–402). ACM.

  • Wang, Y., Wang, L., Li, Y., He, D., Chen, W., & Liu, T. Y. (2013, April). A theoretical analysis of NDCG ranking measures. In Proceedings of the 26th annual conference on learning theory (COLT 2013) (Vol. 8).

  • White, H. D. (2016). Bag of works retrieval: TF* IDF weighting of co-cited works. In BIR@ ECIR (pp. 63–72).‏

  • Yoon, S. H., Kim, S. W., & Park, S. (2016). C-Rank: A link-based similarity measure for scientific literature databases. Information Sciences, 326, 25–40.

    Article  Google Scholar 

  • Yu, B. (2013). Automated citation sentiment analysis: What can we learn from biomedical researchers. Proceedings of the American Society for Information Science and Technology, 50(1), 1–9.

    Article  Google Scholar 

  • Zhao, H. (2014). Sharding for literature search via cutting citation graphs. In 2014 IEEE international conference on Big Data (Big Data) (pp. 77–79). IEEE.

Download references

Author information

Authors and Affiliations

  1. Department of Knowledge and Information Sciences, Faculty of Education and Psychology, Eram Campus, Shiraz University, Shiraz, Iran

    Maryam Yaghtin, Hajar Sotudeh & Mahdieh Mirzabeigi

  2. Department of Computer Science and Engineering, School of Electrical and Computer Engineering, Shiraz University, Shiraz, Iran

    Seyed Mostafa Fakhrahmad

  3. Department of Educational Management and Planning, Faculty of Education and Psychology, Eram Campus, Shiraz University, Shiraz, Iran

    Mehdi Mohammadi

Authors
  1. Maryam Yaghtin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hajar Sotudeh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mahdieh Mirzabeigi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Seyed Mostafa Fakhrahmad
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mehdi Mohammadi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hajar Sotudeh.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yaghtin, M., Sotudeh, H., Mirzabeigi, M. et al. In quest of new document relations: evaluating co-opinion relations between co-citations and its impact on Information retrieval effectiveness. Scientometrics 119, 987–1008 (2019). https://doi.org/10.1007/s11192-019-03058-3

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-019-03058-3

Keywords

Search

Navigation