Large-scale evaluation framework for local influence theories in Twitter

https://doi.org/10.1016/j.ipm.2014.06.002Get rights and content

Highlights

  • We propose a formalized evaluation framework for topic-specific influence theories specialized in Twitter.

  • We introduce a novel evaluation metric that assesses the aggregate sentiment of a group of user.

  • We introduce a two-dimensional taxonomy for classifying influence theories.

  • We use the framework to evaluate five existing theories from literature and assess their performance.

Abstract

Influence theories constitute formal models that identify those individuals that are able to affect and guide their peers through their activity. There is a large body of work on developing such theories, as they have important applications in viral marketing, recommendations, as well as information retrieval. Influence theories are typically evaluated through a manual process that cannot scale to data voluminous enough to draw safe, representative conclusions. To overcome this issue, we introduce in this paper a formalized framework for large-scale, automatic evaluation of topic-specific influence theories that are specialized in Twitter. Basically, it consists of five conjunctive conditions that are indicative of real influence exertion: the first three determine which influence theories are compatible with our framework, while the other two estimate their relative effectiveness. At the core of these two conditions lies a novel metric that assesses the aggregate sentiment of a group of users and allows for estimating how close the behavior of influencers is to that of the entire community. We put our framework into practice using a large-scale test-bed with real data from 75 Twitter communities. In order to select the theories that can be employed in our analysis, we introduce a generic, two-dimensional taxonomy that elucidates their functionality. With its help, we ended up with five established topic-specific theories that are applicable to our settings. The outcomes of our analysis reveal significant differences in their performance. To explain them, we introduce a novel methodology for delving into the internal dynamics of the groups of influencers they define. We use it to analyze the implications of the selected theories and, based on the resulting evidence, we propose a novel partition of influence theories in three major categories with divergent performance.

Introduction

In the context of a social network, influencers are prominent individuals with special characteristics that enable them to affect a disproportionately large number of their peers with their actions. Their special characteristics are related to their individual activity and social background as well as to their position in the network (i.e., their connections with the other members). These influencers typically play a crucial role in a variety of scientific and business domains (Bakshy, Hofman, Mason, & Watts, 2011). For example, marketing campaigns could gain value from this process, since individual customers are prone to imitate their highly influential peers with respect to product adoption (Keller, Fay, & Berry, 2007). Instead of overwhelming the entire customer base with massive, but blind advertisements, marketing campaigns could target a small number of influential people. This cost-effective alternative is called viral marketing and is capable of achieving similar levels of product diffusion with traditional approaches (Brown & Hayes, 2008).

To facilitate such applications, a lot of research has focused on the identification of influencers. This effort led to the development of influence theories,1 which constitute formal models that estimate for every member of a social network the influence she exerts on her peers according to one or more criteria. Some theories study influence on a global scale, considering the activity and the user base of the entire social network. Most commonly, though, an individual’s influence is local: she may be considered expert in a specific domain, but her opinion usually holds little weight outside this particular area. Based on this principle, local influence theories aim at identifying influencers among the members of individual communities, which are usually formed around a particular topic. Typically, the local theories are more accurate and efficient than the global ones, as they exclusively consider the activity and the dynamics inside the individual communities.

A major issue in the study of local influence theories is their evaluation. Over the recent years, On-line Social Networks (OSNs) have provided researchers with powerful tools for studying the dynamics of influence diffusion. They contain a vast amount of user-generated content as well as explicit connections among their members, thus allowing for the analysis of social influence on an unprecedented scale. Still, influence constitutes a subjective concept and, as such, it is very hard to measure and track. Most works actually lack a formal methodology for evaluating the results produced by their theories. Instead, they typically resort to selecting a small sample of the top ranked users in order to assess their authority in the real world (e.g., their fame or the quality of their content) (Cha, Haddadi, Benevenuto, & Gummadi, 2010; Weng, Lim, Jiang, & He, 2010). This manual procedure, however, cannot scale to large volumes of data and, thus, is incapable of yielding representative, reproducible and generalizable results.

In this work, we aim to overcome this shortcoming, by establishing a principled framework that is capable of evaluating local influence theories for OSNs on a large-scale. It receives as input the groups of influencers they define – called prominent groups in the following – along with the rest of the community and the corresponding activity. The goal of our framework is to estimate the relative accuracy of influence theories in predicting activity patterns that denote an imitation by the rest of the community. Internally, our framework encompasses five conditions that should be satisfied by a prominent group with real influence over the other members of the community. These conditions can be summarized as follows:

  • 1.

    real influencers comprise a small subset of the community,

  • 2.

    they are able to affect their fellow members with limited cost, i.e., by accounting for a limited portion of the community’s overall activity,

  • 3.

    their activity is highly correlated with that of the remaining community with respect to an objectively measured metric,

  • 4.

    their activity that is relevant to this metric chronologically precedes that of their peer community members, and

  • 5.

    the volume of their activity that is relevant to this metric corresponds to a mere fraction of the overall activity this metric takes into account.

Conditions 1, 2 and 5 actually correspond to the pre-processing requirements of our framework. Their goal is to ensure that a prominent group is compatible with it, accounting for a limited portion of the activity and the user base of the underlying community. These are fundamental prerequisites for drawing safe conclusions from the analysis performed by our framework. The remaining two conditions encapsulate the real functionality of our framework. At their core lies an objectively measurable metric that correlates the activity of a prominent group with the rest of the community. To elucidate its functionality, consider a metric that assesses the aggregate sentiment of a group of users; a high correlation between the prominent group and the rest of the community members indicates that the stance of the former coincides with the overall “mood” of the latter. Failure with respect to either of these conditions indicates a theory that is inadequate in identifying real influencers. In contrast, an influence theory is effective if the individuals it marks as influencers satisfy both conditions. The stronger these conditions hold for them, the more effective the theory is. For instance, among two theories with similar performance, the one that exhibits higher correlation with the rest of the community is preferred.

Given that all five conditions rely on objectively measurable metrics, our framework allows for comparing the performance of local influence theories on a large-scale, without manual intervention. To put it into practice, we form a large-scale benchmark dataset that consists of real-world data. We actually draw our data from Twitter,2 which was selected for several reasons (Bakshy et al., 2011; Cha et al., 2010; Weng et al., 2010): it is one of the most popular OSNs in the field, it conveys ad hoc, yet clear and manageable rules for social interaction among its members, it abounds in dynamic topic communities and finally, it provides easy access to large volumes of user-generated content. In total, our test-bed comprises 75 topic communities from Twitter with more than 600,000 highly active users, who have posted over 6 million messages during a time period of 7 months. Hence, it is suitable for performing large-scale qualitative and quantitative analyses with our framework.

Our experimental study also includes several local influence theories from the literature. To facilitate the understanding of their functionality, we introduce a two-dimensional taxonomy that classifies them with respect to their scope and the metric they use for assessing influence. The former criterion partitions influence theories into global, local (i.e., topic-specific) and hybrid ones, while the latter criterion distinguishes the evidence they consider into textual, graphical and hybrid information. We then map the main influence theories for Twitter to our taxonomy and explain which ones are applicable to our settings. Our analysis results in the selection of five established and representative influence theories that have been widely used in the literature.

The outcomes of our thorough evaluation indicated significant differences in the performance of these influence theories. To explain the resulting performance patterns, we introduce a novel methodology for delving into the internal functionality of every theory in order to examine the dynamics of its group of influencers. In essence, it comprises a series of statistical analyses that reveal three aspects of each prominent group:

  • 1.

    the levels of homophily among its members,

  • 2.

    the versatility of their activity, and

  • 3.

    the affinity between them, in terms of the frequency of their pairwise interactions.

The outcomes of this methodology advocate a tripartite categorization of influence theories: (i) those that form groups of influencers with strong ties among them, (ii) those that select unrelated, but individually powerful influencers, and (iii) those that mark as influencers ordinary users, who lack any sense of team spirit and exhibit low levels of collaboration. In practice, the last category yields poor performance with respect to our evaluation framework, while the first one identifies highly effective influencers, who coordinate with each other in order to spread their impact to the entire community. Similar effectiveness is achieved by the influencers of the second category, despite the limited collaboration between them, because they benefit from their individual merits.

On the whole, the main contributions of our work are the following:

  • We formalize the problem of evaluating the performance of local influence theories on a large scale. We actually reduce it to checking five objectively-measurable conditions that provide strong indications of real influence exertion in the context of any social network.

  • We put our evaluation framework into practice, testing five established local influence theories over a large dataset that comprises 75 Twitter communities with more than 600,000 users and 6 million tweets.

  • We analyze the performance of the selected influence theories through a novel methodology that provides insights into their functionality and the dynamics of the prominent groups they define.

  • We further analyze the performance of the selected influence theories, by introducing a two-dimensional taxonomy that classifies influence theories according to their scope and the influence metric(s) they employ. We apply it to the main influence theories for Twitter, but it is general enough to accommodate theories for any other social network.

The rest of the paper is structured as follows: in Section 2, we present the most important works in the field and organize them according to our two-dimensional taxonomy. Section 3 formalizes the notions that lie at the core of Twitter and based on them, it introduces our evaluation framework. In Section 4, we analyze the performance of selected influence theories with respect to the five conditions of our framework and in Section 5, we introduce a novel methodology for analyzing the internal dynamics of prominent groups. Finally, Section 6 concludes the paper, providing directions for future research.

Section snippets

Related work

Influence diffusion in real-world social networks has been the subject of various studies over the past few decades – see (Katz, Lazarsfeld, & Roper, 2005) for more details. Recently, it raised new interest among researchers, largely due to the popularity of OSNs, such as Facebook3 and Twitter. The user activity recorded by these systems actually allows researchers to study real-world social influence on an unprecedented scale. To the best of our knowledge, though, no

Problem formulation

In this section, we formally define our evaluation framework along with the fundamental notions that are related with it. We begin with the basic parts of Twitter that lie at the core of local influence theories, we continue with the formalization of the theories that participate in the evaluation and conclude with the definition of the five conditions that comprise our framework.

Evaluation of local influence theories

In this section, we put our evaluation framework into practice, comparing the influence theories of the previous section on the basis of Problem 1. First, we document the process we followed in the creation of our large-scale, real-world benchmark dataset. Then, we examine every condition individually, in the order it should be applied. The only exception is the Size Condition: in its place, we consider the fixed set of prominent sizes k  {10, 20, 50, 100}. We selected this particular set of k

Internal analysis of local influence theories

This section introduces a methodology for the internal analysis of local influence theories, i.e., for examining the dynamics that occur inside the prominent groups they define. This methodology considers three aspects of the relations among prominent users:

  • (i)

    their homophily, as inferred from the reciprocal links among them,

  • (ii)

    their affinity, in terms of the frequency of their communication, and

  • (iii)

    the versatility of their activity, as indicated by the overlap patterns among the prominent groups of

Conclusion

In this paper, we proposed an evaluation framework for assessing the relative effectiveness of topic-specific influence theories. Its goal is to examine whether these theories identify as influencers small groups of users who are able to affect certain, objectively measurable aspects of a community with minimal effort. Two assumptions lie at the core of our framework:

  • i.

    Each topic community is defined by the same hashtag throughout its lifetime and, thus, it involves members that are not

References (42)

  • D. Gayo-Avello

    Nepotistic relationships in Twitter and their impact on rank prestige algorithms

    Information Processing & Management

    (2013)
  • J. Yang et al.

    Patterns of temporal variation in online media

    (2011)
  • M. Abbasi et al.

    Real-world behavior analysis through a social media lens

    (2012)
  • A. Anagnostopoulos et al.

    Influence and correlation in social networks

    (2008)
  • S. Aral et al.

    Distinguishing influence-based contagion from homophily-driven diffusion in dynamic networks

    Proceedings of the National Academy of Sciences

    (2009)
  • E. Bakshy et al.

    Everyone’s an influencer: Quantifying influence on Twitter

    (2011)
  • Barbosa, L., & Feng, J. (2010). Robust sentiment detection on Twitter from biased and noisy data (pp. 36–44). Beijing,...
  • F. Bodendorf et al.

    Detecting opinion leaders and trends in online social networks

    (2009)
  • D. Brown et al.

    Influencer marketing: Who really influences your customers?

    (2008)
  • Bruns, A., & Burgess, J. (2011). The use of Twitter hashtags in the formation of ad hoc publics. In European consortium...
  • M. Cha et al.

    Measuring user influence in Twitter: The million follower fallacy

    (2010)
  • W. Chen et al.

    Scalable influence maximization for prevalent viral marketing in large-scale social networks

    (2010)
  • Choudhury, M. D., et al. (2010). “Birds of a Feather”: Does user homophily impact information diffusion in social...
  • D. Cosley

    Sequential influence models in social networks

    (2010)
  • A. Cui

    Discover breaking events with popular hashtags in Twitter

    (2012)
  • P. Domingos et al.

    Mining the network value of customers

    Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining (KDD′01)

    (2001)
  • Ernesto, D. -A., et al. (2012). What is happening right now ... that interests me? Online topic discovery and...
  • T.L. Fond et al.

    Randomization tests for distinguishing social influence and homophily effects

    Proceedings of the 19th international conference on World wide web (WWW′10)

    (2010)
  • G. Giannakopoulos

    Representation models for text classification: A comparative analysis over three web document types

    (2012)
  • A. Go et al.

    Twitter sentiment classification using distant supervision

    Processing

    (2009)
  • A. Goyal et al.

    Learning influence probabilities in social networks

    (2010)
  • Cited by (22)

    • FLOWER: Fusing global and local associations towards personalized social recommendation

      2018, Future Generation Computer Systems
      Citation Excerpt :

      Therefore, we denote such methods as local methods. Although both global and local approaches have achieved their success in social recommendations, adopting local correlations alone may disregard the propagation of connections [56] while utilizing global associations singly may disregard the essential ‘triadic closure’ rule. Based on the understandings, the objectives of the study are to answer the following questions:

    • Measuring user influence on Twitter: A survey

      2016, Information Processing and Management
      Citation Excerpt :

      Somehow, topical-sensitive measures are a kind of local measures, not in a topological sense but in a semantic level. According to Kardara et al. (2015), these measures are more effective and functional than global measures, i.e., those that are not restricted to a topical domain. These measures are usually used in offline analysis.

    • Influencers Analysis from Social Media Data

      2022, International Conference on Artificial Intelligence and Data Engineering, AIDE 2022
    • Social Network Influence: Making the Case for Semantic Analysis

      2022, International Conference on Electrical, Computer, and Energy Technologies, ICECET 2022
    View all citing articles on Scopus
    View full text