Keywords

1 Introduction

Increasing response rate to surveys has been a goal of researchers since surveys were created. One simple method of increasing a participant’s willingness to complete a survey is through personalization. For example, dressing similarly to the participant to appear as an in-group member or appearing as a member of a group the participant has a high opinion of are ways to increase response rate when conducting surveys in person [8, 13]. While these studies on personalization have traditionally been applied to in-person surveys, the internet has provided researchers with the option to use online survey designs [4, 17].

Today, online surveys are common, convenient, inexpensive, and increasingly accessible in our progressively globalized and internet-connected world. The benefits of online surveys include cross-cultural and expanded geographical reach as well as easily modified formats for greater flexibility (e.g., [7]). With an increasing use of online surveys for data collection, and with an increasingly diverse pool of participants who have access to such surveys, it is imperative that researchers become aware of how best to recruit accurate samples of populations and how to develop user-centered personalized surveys.

Currently, with surveys designed by the majority for the majority, “low-incidence” or specialized subsets of the population are being passed over for survey research or are not responding when recruited [3, 20]. Additionally, there is an overall trend of declining surrey response rate across all modes, be it in-person or online [1]. Online surveys that are distributed via email have had a significantly decreasing response rate since 1986, according to one review [21]. This decline affects the ecological validity of the final survey dataset. Reduced response rate disrupts the accuracy of the final sample, as minority groups of identities may be less motivated to respond, allowing an oversampling of majority group members or specific kinds of groups (i.e., the volunteer bias, which explains people with certain personality traits are more likely to participate in research [12]).

Presently, the literature focuses on how researchers can display in-group social identities to increase face-to-face survey response rate, but whether these same factors affect online surveys is yet to be investigated. Research examining how personalization and in-group/out-group social identities affects online survey responses is lacking. The purpose of this study is to determine if incorporating social identities will improve online survey response rate, much the way traditional pen-and-paper surveys automatically include the researcher’s identity due to the in-person nature of the research. Increasing personalization through developing user-centered design surveys could contribute to recruiting a more representative sample of the population as well as an overall increase in response rate. As Cook, Heath, and Thompson [4] state, “Election polls make clear that the representativeness of our samples is much more important than the response rate we obtain. A sample of fewer than 1% of the population can be more representative, indeed much more representative, than a sample of 50% or 60% of the population” (p. 821). Findings from this study have implications for designing user-centered online surveys that reach more diverse samples of participants, allowing for a more representative, ecologically valid dataset.

1.1 Increasing In-Person Survey Response

Surveys have been one of the main methods of collecting data for research for decades (see Converse, 2017 for a history on survey research in the United States of America). Originally, recruitment for surveys was in-person as a face-to-face interaction. Groves, Cialdini and Couper [13] have shown in these face-to-face interactions “similarity leads to liking”; that is, if the person requesting information for the in-person survey is similar to the participant, the participant is more likely to participate in the survey (p. 488). They give examples of researchers driving older cars when conducting door-to-door surveys in poorer neighborhoods and dressing up in suits when in richer neighborhoods.

Dell, Vaidyanathan, Medhi, Cutrell and Thies [8] support expanding diversity in HCI research populations. In their study, the researcher interviewed participants and asked if they would be interested in a new technology they were selling. The interviewer was either Indian, like the participants, or a white Westerner. If the in-person interviewer was perceived as a foreigner, then the Indian participants preferred the interviewer’s technology, even if it was objectively inferior, due to a heightened opinion of western technology. They concluded that social identities have an impact on participants’ responses and willingness to participate.

1.2 Increasing Online Survey Response

Studies on online survey design have shown that including pictures and visual elements beyond simple text facilitates response rate. Historically, Dillman, Tortora, Conradt and Bowker [11] found that pictures reduce participant responses, but this is due to HTML requiring more data and being slower to load in the nineties compared to today’s highspeed internet. However, even back then, Dillman and colleagues did recommend graphics as a way to keep participants’ attention, such as giving them a visual progress indicator [10].

Couper, Tourangeau, Conrad and Crawford [5] found online surveys that show participants visible options have a better response rate. Much like Dillman et al. [10], this continues a theme of using images to help to engage participant attention. Deutskens, De Ruyter, Wetzels and Oosterveld [9] found that the inclusion of visual elements, such as product images, improved the response rate of online surveys. Daley, McDermott, McCormack Brown and Kittleson [6] reported that their focus group of undergraduates recommended clipart be used within online surveys and a photo of the primary investigator be displayed on the site hosting the survey as a way to add a personal touch to an otherwise impersonal questionnaire. This echoes earlier studies on how the personal interaction in face-to-face surveys can improve response rate [13]. Sauermann and Roach [19] found that personalization via addressing the participant’s first name increases emailed survey response rate by as much as 48%, in a study of 24,651 survey respondents. Cook, Heath and Thompson [4] conducted a meta-analysis of online survey response rate and found personalized contacts to be a major factor associated with a high response rate in online surveys.

1.3 Knowledge Gap

It is clear from the reviewed literature that personalization is important to survey participants, and that considering their identities and how they may perceive the person giving the survey has effects on response rate [4, 13, 19]. Many of the reviewed studies on visual elements and personalization of survey design are dated (late nineties to early aughts) or focus on in-person surveys only. Very little research has investigated how the “human element” in face-to-face surveys translates to online surveys in this digital age. This is further expanded upon by Schlesinger, Edwards and Grinter [20] in their call to action to begin including more diverse participant groups and controlling for the multitude of identities an individual can have in HCI (human-computer interaction) studies. Thus, personalization and social identities (i.e., foreigner or native, same income bracket or different) have real effects and should be considered in the context of the respondents’ identities interacting with the survey design. With increased awareness of the role of diversity in surveys [2, 20], attention should be given to how online surveys can be personalized on a level beyond simply using the person’s name, especially since respondents are not experiencing a face-to-face interaction with survey researchers.

How can survey response rates be increased in online surveys? Does Groves et al.’s [13] “similarity leads to liking” in in-person surveys translate to an online format? Does the presence of faces, either as in-group members or out-group members, affect online survey responses the same way it does in-person surveys? The answers to these questions have the potential to inform researchers of how personalized, user-centered online surveys could increase the response rate and representation of target populations, as well as improve the experience of survey participants.

2 Current Study

This study investigated if participants’ identities (i.e., university affiliation) affects responses to different survey designs in an online format the same way these identities affect face-to-face survey responses [8, 13]. The experimental design tested if inclusion of faces with differing social group identities had an effect on participants’ willingness to continue answering questions. Participants saw the face of someone they believe attends their university, the face of someone they believe attends a rival university, or a simple text-based survey with no picture. Participants were then presented with two questions that gave them the option to continue participating at no additional benefit to them.

2.1 Research Questions

The following research questions guided the direction of this study.

  • RQ-1: Does seeing an in-group member versus an out-group member (based on university affiliation) have different effects on participants’ responses to online surveys, similar to the effects seen in in-person surveys?

  • RQ-2: Does the mere presence of a photo of a person, regardless of group membership, impact responses differently than a text-only, photo-less design in an online format?

2.2 Assumptions and Hypotheses

The assumptions that underlie the hypotheses to be tested in this study are that in-group members (members from the participant’s own university) have a positive influence on participants’ willingness to participate in a focus group and answer an additional survey when asked [13].

Conversely, out-group members (members of a rival university to the participant’s university) have a negative influence on participants’ willingness to participate in a focus group and to answer an additional survey when asked [8].

Another assumption is that there will be an overall effect of increased positive influence on participants’ willingness to participate in a focus group and answer an additional survey when comparing conditions with a photo to the condition with text-only [6, 9, 11]. Based on these assumptions, the following hypotheses were tested:

  • H1: Participants are more likely to continue to answer survey questions if they view a picture of someone they believe is from their own university (The University of Tennessee, Knoxville).

  • H2: Participants are less likely to continue to answer questions if they view a picture of someone they believe is from a rival university (Alabama or Florida).

  • H3: Participants are more likely to continue to answer questions if they view a picture of a person, regardless of university, than if they view only text with no picture.

2.3 Theoretical Framework

This study is grounded in the overarching theory of user-centered design, which gives attention to user characteristics and the environment in designing a product and the perspective of how this product will be understood and used by a user. User-centered design takes into consideration user beliefs, attitudes, and behaviors as they connect to specific tasks the user aims to achieve [16, 22]. Personalization is an important factor of user-centered design [15, 22].

In this study, the design of the survey will be tailored based on two assumptions: 1. participants are more willing to answer additional questions in a survey if they believe the person asking is a member of their “in-group,” and 2. university affiliation can be considered an in-group or out-group status. Groves, Cialdini and Couper [13] is a prime example of using in-group status to garner responses by intentionally changing their appearance to appeal to the participants’ demographic groups they were surveying. They did so by wearing a suit or by driving a cheaper car depending on whether the potential participants were of a higher or lower income class, respectively.

Within postal surveys, university affiliation has shown to be a marker of in-group status [21]. In a study on own-race bias, participants were more likely to remember the faces of those they believed belonged to their own university than the faces of those they believed belonged to a rival university, regardless of race [14]. Based on existing literature (see [18] for a review), one would have predicted that race would have a stronger effect on face recognition; however, university affiliation was what determined recognition accuracy instead.

2.4 Methodology

This study employed a quantitative method to gather data about participants’ perceptions of researcher in-group status and how these perceptions affected responses to online survey questions asking for additional information to be provided. A survey questionnaire was developed by the researchers to collect the data. There were two instances where a participant could agree or disagree to contribute more data. These responses were quantified by coding participants’ willingness to answer more questions as a binary response.

Participants.

The Psychology Department at the University of Tennessee, Knoxville (UT) has a SONA (Sona Systems Ltd.) database of undergraduate students enrolled in introductory psychology courses, maintained by the head of the experimental psychology department. Undergraduates taking introductory psychology courses at UT are required to enroll in the SONA database and complete studies that researchers post for course credit. Students must complete 5.5 credits worth of studies by the end of the semester, and completing this survey earned the student 0.5 credits. There are approximately one thousand undergraduate students actively enrolled in the database each semester.

Following the IRB approval (granted on November 1st, 2018), all students in this database were recruited to participate in the study. An invitation and a distribution link to the survey was posted in the SONA database, which allowed students to self-select to participate based on interest in a football themed survey; a total of 400 students completed the survey.

Instrumentation.

Qualtrics survey software (Qualtrics, Provo, UT) was used to develop survey questions and to collect data from participants. The survey questionnaire consisted of 20 closed questions and 11 open-ended questions, followed by 2 closed questions asking about contributing more data, making the survey 33 questions long at its shortest length. However, if students agree to answer additional survey questions, there was a second set of questions about personality traits that included 2 matrix questions, 3 closed questions, and 1 open-ended question. Students were expected to take fifteen to thirty minutes to complete the survey, depending on whether they agreed to the second set of questions. Estimated completion times were calculated via Qualtrics in-house completion time estimate.

Conditions.

The first page of the survey randomly displayed one of five conditions to the participants: a stock image of an actor with UT school logos, the same actor with UT football logos (the 2 in-group conditions, which included a “Go Vols!” in the text); the same actor with Florida University logos, the same actor with Alabama University logos (rival universities, the 2 out-group conditions, which included a “Go Gata!” or “Roll Tide!” respectively), or a plain text box with no images or references to a college football team (a control for picture effects).

Procedure.

All undergraduates enrolled in the SONA database as of the fall semester 2018 (approximately 1000 students) were recruited to participate in this study. An invitation with a link to the survey was posted in this database. Students browsing research studies and who were interested in the survey titled “College Football Fan Survey” took the survey. Thus, participants self-selected to participate depending on their interest in the study. This helped ensure that the students were indeed fans of college football and held strong beliefs about their favorite team.

Four hundred students (N = 400) completed the survey. However, only data where students identified the University of Tennessee, Knoxville as their favorite football team and who filled out all open-ended questions were included for analysis (N = 358). Data was excluded from analysis if the student identified a different team than UT as the favorite (N = 38), or if the student did not fill out the open-ended questions (N = 4). Filling out the open-ended questions was used as an indicator of whether the students were paying attention and genuinely answering the questions.

Before filling out any of the questionnaires, participants viewed the experimental condition picture or text-box and were asked to read and electronically sign an informed consent form on the first page of the survey, indicating they agreed to participate. Participants first completed the thirty-one-question survey about college football, which asked questions such as, which college football team is your favorite, and how often do you attend games.

Next, participants were presented with the experimental condition (picture or text-box) again with a question asking if they would be willing to participate in an in-person or over the phone focus group session at a later date. They were also informed that their answer to this question would not increase nor decrease their participation credit of 0.5 credits. Afterwards, participants were asked if they would be willing to answer 6 additional questions for an unrelated survey about fantasy characters and personality traits, again with no change to their participation grade regardless of their answer. There was no focus group and the additional optional survey was used as “filler,” both explained in the debriefing. These questions were to test if participants were willing to continue participating in the research at no additional credit depending on if they thought an in-group member (someone from their own university, UT) or an out-group member (someone from a rival football university) was asking.

After completing the survey, the participants were provided a debriefing explaining that the study was not about football, but rather about in-group statuses of researchers and their effect on responses. This is required by UT’s IRB in cases where the experiment uses deception, which was the circumstance in this study. Then, they were asked if they consented to submitting their data for analysis given that they were now aware of the true nature of the study. All participants whose responses were included in the data analysis (N = 358) agreed to the debriefing.

2.5 Data Analysis

Responses to the survey were filtered to only include those who agreed to both the informed consent and the debriefing. Of those responses, data was analyzed using those who listed a variation of UT as their favorite football team and who also answered all other text-box questions (e.g., If you could choose to go back in time to any college football game of your choosing, which game would you want to see in person and why?). Answers to the text-box questions were not analyzed; they were simply used as a marker to tell if the participant was completing the survey fully or was skipping questions and leaving them blank. This was to ensure that the data from those who listed UT as their favorite team were paying attention to the survey.

Responses were grouped based on the five experimental conditions. Participants either saw a picture of a man with University of Tennessee school logos (UTstudent, N = 74), University of Tennessee football logos (UTsports, N = 67) (the two in-group conditions), University of Florida school logos (FLstudent, N = 76), University of Alabama school logos (ALstudent, N = 65) (the two out-group conditions as these are rival football teams to UT), or a simple text box with no images or references to a college team or university (Text-Only, N = 76) (the control condition). The picture was a stock image of a young white man, which is the same for all four groups featuring an image. Only the logos on the picture changed between groups. The participants viewed this picture on the first page where they agreed to the informed consent and again before they saw the two questions asking them to contribute more data.

Of question responses, answers to “Would you be willing to participate in a focus group?” and “Would you be willing to answer another questionnaire?” were dummy coded to list affirmative answers (“yes I’d be willing to”) as 2, and negative answers (“no I don’t want to participate”) as 1.

In addition to the five groups based on what picture (or lack of picture) the participants saw, a group coefficient was created based on the group status of the picture. These three groups were created by combining UTstudent and UTsports (“In-group member” group, N = 141), ALstudent and FLstudent (“Out-group member” group, N = 141), and the text-only condition (“Control”, N = 76). While the five separate groups allow for possible differences to be seen based on the specific picture or university, combining the UT groups and the rival university groups into “in” or “out” status helped delineate any overarching effect the in-group/out-group status had on responses. Additionally, participants were grouped by whether they saw a photo (N = 282) or text only (N = 76) to determine if there was an effect of seeing an image of a face, regardless of group membership of the actor.

Using SPSS statistical software, contingency tables were set up and a chi-square test of independence was performed to examine the relationship between condition and answers to each survey question requesting additional information. This study utilized two questions, one about participating in a focus group and one about answering an additional survey. The chi-square tests were used to determine if answers to those two questions were statistically dependent on condition. The answers to these two questions were analyzed separately by condition. The answers to the questions were also tested to see if they were correlated.

3 Results

3.1 RQ-1

Does seeing an in-group member versus an out-group member (based on university affiliation) have different effects on participants’ responses to online surveys, similar to the effects seen in in-person surveys?

The relationship between condition (UT sports, UT student, AL student, FL student, Text-only) and answer was insignificant for both questions, X2 (4, N = 358) = 0.89, p = 0.93 for the question about participating in the focus group and X2 (4, N = 358) = 3.85, p = 0.43 for the question about completing the additional survey (see Table 1).

Table 1. Contingency Table comparing Answers * Condition.
Full size table

The relationship between group status (in-group, out-group, or control) and answer was insignificant for both questions, X2 (2, N = 358) = 0.15, p = 0.93 for the question about participating in the focus group and X2 (2, N = 358) = 1.03, p = 0.60 for the question about completing the additional survey (see Table 2). Since the control group was much smaller than the in-group and out-group, an additional Chi-square test was run only on in-group and out-group effects on both questions. Results were insignificant for both the focus group question, X2 (1, N = 282) = 0.20, p = 0.89, and the additional survey question, X2 (1, N = 282) = 0.02, p = 0.90.

Table 2. Contingency Table comparing Answers * Group.
Full size table

3.2 RQ-2

Does the mere presence of a photo of a person, regardless of group membership, impact responses differently than a text-only, photo-less design in an online format?

The relationship between presence/absence of a photo (photo vs. text-only) and answer was insignificant for both questions, X2 (1, N = 358) = 0.13, p = 0.72 for the question about participating in the focus group and X2 (1, N = 358) = 1.02, p = 0.31 for the question about completing the additional survey (See Table 3).

Table 3. Contingency Table comparing Answers * Photo.
Full size table

Given that no condition had an effect on responses, a phi-coefficient analysis was used as a measure of association between the two binary variables to determine if answers to one question were correlated with the other across all conditions. There was a significant moderately positive correlation, φ = 0.45, p < .01, between a participant’s answers to the first and second question. If a participant said “yes” to the first question, the participant was likely to say “yes” to the second question; likewise, if the participant said “no” to the first question, the participant was likely to say “no” to the second question, regardless of experimental condition (See Table 4).

Table 4. Contingency Table comparing Focus Group Answers * Extra Survey Answers.
Full size table

4 Discussion

While there have been studies that have investigated how a researcher’s social identity and the survey respondent’s social identity interact in face-to-face scenarios [8, 13], and how personalization in online surveys, especially personalization based on university affiliation, can increase response rate [4, 19, 21], no study thus far has investigated the effect of seeing pictures of in-group or out-group university member faces on responses to an online survey.

This study employed a photo of a young white man with university logos around him as a proxy digital representation of the “researcher.” Participants who indicated the University of Tennessee, Knoxville was their favorite college team were placed in one of five conditions when taking the survey: two conditions had them view the actor with UT logos, two of the conditions had them view the actor with a rival football university’s logos, and the last condition had them view a simple text box instead of an image.

Results showed that the photo the participants saw before the survey and before answering questions about contributing more data did not have an effect on their answers. Similarly, whether or not the participants saw a photo of a person with the instructions did not influence their answers. Instead, it was found that a participant’s first answer was moderately correlated with their second answer, regardless of what condition they were in. This may imply that once a participant has agreed to participate, conditions within the survey surrounding identity do not have a large impact on responses.

The most important factor about this study to note was that it was conducted with participants who had agreed to answer the football survey; per the IRB, only data from those who agreed to the informed consent and debriefing were included in the original N = 400. While other studies have investigated survey response rate by analyzing whether a participant agreed or disagreed to participate after initial contact, this study looked at those who had already agreed to participate and tested how the experimental conditions affected their responses within the survey (see [1] for a meta-analysis of quantitative data regarding response rates to surveys). Based on the way this experiment was set up, participants saw the informed consent page with the photo of the actor, and only data from those who agreed to the informed consent after seeing the experimental condition could be analyzed. Thus, whether a participant agreed or disagreed to the informed consent (therefore agreeing/disagreeing to participate in the survey) based on experimental conditions could not analyzed. A future direction for research would be analyzing whether participants agree to take the survey at all if they view a picture of an in-group or out-group member, instead of focusing on answers within the survey of those who already agreed to participate.

5 Implications

These findings have methodological and practical implications. As to the methodology, results indicate that, in analyses of participants who agree to participate in a survey, individual differences between participants, such as different personality traits, may have more of an effect than the perceived social identity of the researcher. If the participant is someone who would be willing to participate in a focus group and answer additional survey questions, the participant is likely to do this regardless of who is asking if they have already agreed to participate. Likewise, if the participant is someone who is unwilling to be a part of a focus group or answer an additional survey, the participant is likely to be unwilling even after they have agreed to participate, regardless of who they believe is asking. Thus, future research should investigate whether a participant agrees or disagrees to begin a survey based on who they believe is asking, instead of looking at which participants continue to participate after they have already agreed to take the survey. Future studies should collect participant demographic data and design surveys that include pictures representing diverse racial or cultural identities to determine if there is an interaction effect of these identities on responses.

A practical implication for these findings is that increasing response rate from participants with diverse backgrounds creates a more representative dataset [3]. This could improve knowledge gained from survey-based experiments about both general and specific subsets of the population. With increased use of online surveys as a method for research, attention should be given to how personalization in the context of user-centered design can be implemented to enhance response rate and quality of responses. By personalizing surveys based on user-centered design principles, participants may be more willing to begin surveys, even if the personalization does not have an effect once they agree to complete the survey.

6 Limitations

One of the main limitations to consider in this study is the population. As data was collected from mostly freshman students, it is possible they did not yet feel strong ties to their university as their college football team, despite indicating UT as their favorite. Hence, seeing a rival university was not as salient or negative. Data was also collected during the latter half of the semester, and since students were taking this survey for a course grade, it is possibly the high number of negative responses was due to a lack of time on the participants’ part. Had there been no course grade requirement incentive, responses may have been different.

Additionally, the survey design used a singular white man as an in-group or out-group member. Relevant literature has shown other demographic information to be equally as salient as university affiliation, such as the race of the participant and the researcher. Similarly, studies on whether a person is shown alone or in a group of people has been shown to have an effect on how a participant perceives them [14, 23, 24]. It is possible that showing the participant a picture of a person who is from the same university but is of a different race from the participant may have different effects on responses than seeing someone of the same race and same university [24]. It could also be possible that showing a group of researchers who are all from a university as the “research team” would have a different effect than showing one person as a representative of a university [14].

7 Conclusions

There are many factors to consider when designing a survey. With the decline in online response rates, researchers should employ the best practices and methods to increase responses overall [1, 17]. While some social identities may not affect participants’ willingness to respond, other factors may. In this study, use of university affiliation appeared to have no effect on answers to questions within the survey. Previous studies have revealed an effect of university affiliation on participants’ willingness to participate in postal surveys [21], and on participants’ opinions of whether a person is an in-group or out-group member [14, 24]. Our findings are mainly attributed to the design of the survey in that participants agreed or disagreed to informed consent while viewing the photo of the in-group/out-group member, and those who did not consent to participate were not analyzed. Researchers should include the image of the actor (the “researcher” proxy) on the first page of the survey and analyze whether participants agree to take the survey at all if they see an in-group or out-group member; analyzing those who choose not to participate based on group status of the actor may provide additional insights. These images should be diverse and more representative of participants’ backgrounds and cultures.

Future directions should explore the social identities of participants and how this factor interacts, both positively and negatively, with the perceived identities of the researchers. This could include testing the effects of group size as well as other demographic manipulations. Including questions to collect demographic data would allow for matching a participants’ race, gender, and university affiliation, among other information, to determine the effect of each of these variables on responses. Researchers should consider using a mixed research method to collect not only quantitative, but also qualitative data. For example, surveys could include questions to explore participants’ feelings toward their own university and to gather their opinions about university affiliation and its impact on their responses. Alternatively, such data could be obtained through individual or focus group interviews following the survey.

User-centered survey design is a new and unexplored theory in terms of how personalization can interact with participants’ willingness to participate. As online surveys become a more mainstream research approach, studies should focus on effective interface design to motivate participants to complete surveys. Increasing responses from marginalized social identities will provide a more accurate representation of the population in the final sampling, which could lead to enhancing the ecological validity and generalizability of collected data, in addition to improving the experience of the participants.