Keywords

1 Introduction

The top principle of human-centered design from a Cognitive Load Theoretical perspective is that the design will never take users even a millisecond to think of “how to use” [1]. One of the persistent challenges in interface design is how to accommodate diversity in cultures, human abilities, background, and personalities, among other factors [2]. In fact, the current trend in globalization calls for universal design of interfaces and tools to meet the needs of all users. Additionally, globalization has become one of the strategic goals of academic institutions in the US [3].

The primary pathway for attracting international diverse students to US institutions of higher education is through the recruitment (admissions) websites [4]. Nonetheless, studies have revealed that these websites require substantial improvements for accommodating international students’ information seeking and needs [5]. While previous studies have examined international graduate students’ information behavior and needs (see for example, [6, 7, 8]), scarce research has addressed international doctoral students’ interactions with university recruitment web pages. The present study should fill this gap.

The purposes of this study are to: (1) examine international doctoral students’ interactions with university recruitment websites, and (2) explore their perceptions of the usability of the websites based on their own experiences.

Findings from this study should provide a basis for designing user-centered university recruitment websites that seek to attract and meet the needs of international doctoral students. The fact that these students may share similar information needs and cultural background, among others, with other international students, the findings could also serve as benchmarks for providing highly usable websites that are supportive of diverse international graduate students.

2 Related Literature

2.1 University Website Evaluation Through Usability

Compared with recruiting domestic students in the US, the ways to attract international students tend to be more limited to word-of-mouth and information on university programs’ websites [4]. From the marketing perspective, the usability of recruitment websites of an organization can impact the user’s perception of that organization [9]. Researchers have investigated the usability of university websites using various approaches. One of these is administering a standard evaluation scale, such as the Website Analysis and MeasureMent Inventory (WAMMI). This scale consists of five main factors: Attractiveness, controllability, efficiency, helpfulness, and learnability. Mentes and Turan [10], for example, applied this scale in a study that examined 339 students’ perceptions of the Namik Kemal University website in Turkey. They found that all factors, except for controllability, are positively associated with the students’ perceptions of the usability of the website.

Rahman and Ahmed [11] adapted 34 items from four different usability scales (Questionnaire for User Interface Satisfaction, JISC checklist for academic websites, QUIS, and WAMMI) to evaluate the usability of the University of Dhaka’s website in Bangladesh. They categorized five factors to examine usability: (1) navigation, searching and interface attractiveness, (2) interactivity and functionality, (3) accuracy, currency and authority of information, (4) accessibility, understandability, learnability, and operability, and (5) efficiency and reliability. Students were moderately satisfied with the usability of the website (average rating = 4/7).

Hasan [12] gathered students’ perceptions of the usability of nine different university websites in Jordan using five usability factors: (1) navigation, (2) architecture, (3) easy of use and communication, (4) visual design, and (5) content. The top three important factors to the students were content, navigation, and ease of use. Poor visual design (e.g., colors and fonts) and inconsistency in visuals across web pages within a specific website surfaced as problems.

Tuzun, Akinci, Kurtoglu, Atal, and Pala [13] investigated ten students’ information seeking of the Hacettepe University Registrar’s Office web page in Turkey using eye tracking and retrospective think aloud protocol. Students performed ten assigned search tasks. Eye-tracking data analysis revealed that students frequently scrolled down and up the main page of the website and scanned looking for information on the search tasks. Students’ attention was distracted by the blank area of the page. Interview data revealed that the webpage small font size, labeling of menu items, and disorganization of contents (e.g., relevant links spread across different parts of a page) were problematic.

One recent study [5] evaluated 42 US doctoral programs websites in special education from prospective students’ (master’s) point of view using a survey instrument with eight components (Navigation, program description, an application element, homepage links, doctoral page elements, homepage organization, faculty information, and overall experiences). The researchers collected data in 2006, 2008, 2011 and 2014. Forty-two students evaluated two websites each time. Key findings revealed that over the 8-year study period (i.e., 2006–2014), there was no significant improvement in website usability. Students rated most questions as “somewhat agree.”

The reviewed studies revealed usability issues in university websites. In these studies, students were native speakers of the languages of the websites. It can be assumed that non-native speakers (e.g., international students) are likely to face more or different usability issues when they interact with websites that are not written in their native languages. For example, Huang and Bilal [14] interviewed 33 international doctoral students and explored their information needs, information seeking behaviors, feelings, and overall experiences during the process of applying for doctoral study in the US. They found that students had two primary information needs (where to apply and how to apply) and used two categories of resources to meet those needs, Experiential Resources (EXR) (e.g., peers, social networks, and social media), and Authorized Resources (AUR) (unofficial and official websites). The many issues and challenges students faced in using university recruitment websites resulted in negative feelings about the overall application process. Students made several suggestions for enhancing the usability of the websites.

In [14], Huang and Bilal employed individual interviews. Given that students were asked to recall their experiences with the application process from memory, and due to the nature of interviews that rely on subjective experiences, it is possible that the students’ accounts are prone to inaccuracies. As a follow-up, we conducted a case study (reported in this paper) using empirical observations in conjunction with exit interviews and examined international doctoral students interactions with the University of Tennessee recruitment websites and associated web pages.

3 Research Questions

The following research questions guided this study:

  • RQ 1: How do international doctoral students interact with the University of Tennessee recruitment websites and associated web pages?

  • RQ2: How do international doctoral students evaluate the usability of the University of Tennessee recruitment websites and associated web pages based on their own experiences?

4 Research Method

4.1 Mixed Method

This study is exploratory in nature. We employed both quantitative and qualitative methods for data elicitation. For the quantitative method, we used eye tracking in Tobii X2-60 Eye Tracker to record the students’ (hereafter, participants) eye movements, and Tobii Studio to capture their interaction behaviors with recruitment websites/pages. For the qualitative method, we utilized exit interviews to gather insights about the students’ experiences with the websites/pages. Eye tracking generates quantitative data allowing for identifying Areas of Interests (AOIs) on a web page, eye movements, and eye fixations, among other factors [15, 16]. Qualitative data using interviews provide understanding of data generated from the quantitative method, while allowing for exploring the students’ experiences with and assessment of the usability of websites/pages.

4.2 Population and Sample

In phase one of this study [14], we recruited 33 international doctoral students from the University of Tennessee, Knoxville (UTK), and accounting for 5% of the total enrollment of doctoral international students [17]. In the present study (phase two), we recruited five international doctoral students in the College of Communication and Information (CCI) at UTK, as of the writing of this paper. Given that around 80% of usability issues will be detected by the first five participants in a usability study [18], we believe that the sample size is somewhat adequate for reporting the preliminary findings of this study.

4.3 Instruments

Questionnaire.

We developed both a brief questionnaire to gather participants’ demographic and background information (e.g., age, gender, country of origin, department, and years in the doctoral program), and an interview instrument based on the WAI Site Usability Testing Questions [19] for user-centered design.

Search Tasks.

We assigned the participants six search tasks (Table 1) and provided a scenario to contextualize the tasks. These tasks varied in terms of requirements. A search task may be simple and fact-finding in that it requires finding a target answer (e.g., T-1), while another task may be somewhat complex in that it requires finding relevant information for more than one facet or component (e.g., T-4).

Table 1. Search tasks.

Scenario.

A friend from your home country is interested in applying to your doctoral program. Your friend asked for specific information about the program, which is reflected in the six search tasks. Find the information that your friend has requested. All search tasks were contextualized in this scenario.

4.4 Search Task Outcome Measures

Search task outcomes were measured in terms of (a) success, (b) dwell time, and (c) time-based efficiency.

(a) Success.

We used a three-point success scale (1 = success, 0.5 = partial success, and 0 = no success). Success means that a participant found the correct answer or relevant information for a search task. Partial success means that a participant found the correct answer or relevant information for some parts of a search task. No success indicates that a participant did not find any information on a search task, or that the information found was incorrect. For tasks with sub-tasks (T-2, T-4, T-5), success was assigned a“1” (success = 1), which we divided equally across the subtasks. For example, T4-a (success = 0.33), T4-b (success = 0.33), and T4-c (success = 0.33).

(b) Dwell Time.

We measured the viewing time of a web page in seconds, from the time a page is loaded after being clicked by a participant to the time the participant leaves the page by closing the browser.

(c) Time-Based Efficiency.

We measured time-based efficiency by accounting for both success and dwell time on all six search tasks across all five participants. The measure uses the following equation [20].

$$ \frac{{\sum\nolimits_{j = 1}^{R} {\sum\nolimits_{i = 1}^{N} {\frac{{t_{ij} }}{{n_{ij} }}} } }}{NR}\left( {{{\text{Sec.}} \mathord{\left/ {\vphantom {{\text{Sec.}} {\text{task}}}} \right. \kern-0pt} {\text{task}}}} \right) $$

Where:

  • N = total number of tasks (6)

  • R = number of participants (5)

  • \( n_{ij} \) = success rate on task i by participant j

  • \( t_{ij} \) = time spent by participant j to complete task i

4.5 Setting and Procedures

The data collection took place in the College of Communication and Information (CCI) User eXperience Laboratory. The Lab is equipped with Tobii Eye-Tracker X2-60 and Tobii Studio. We recruited five participants through personal networks (doctoral students in CCI) to participate in this study (as of the writing of this paper). Following the study approval by UTK Institutional Review Board (IRB), we invited the five participants to take part in this study. We scheduled the participants’ visits to the Lab. Prior to the visits, we integrated the six assigned search tasks and respective scenario into Tobii Studio along with instructions. The search tasks appeared on the computer screen in sequence.

During each participant’s visit, we introduced the purposes of the study, provided an introduction to the Lab and its equipment, and asked the participant to read and sign an informed consent form. Next, the participant was seated in front of the desktop that has the Tobii Eye-Tracker and Studio. The participant’s eyes were calibrated by the Eye-Tracker. Each participant was assigned an ID under which we saved all eye tracking and interaction activities as well as the recorded exit interview.

Each participant’s experimental session was recorded in the Eye Tracker and Studio, saved, and exported for analysis. Upon completing the six search tasks, the participant was interviewed to elicit data about his/her experiences in using the recruitment websites/pages. The interview was audio-recorded using a digital recorder. Recorded interviews were transcribed, coded, and analyzed using NVivo 12 software.

4.6 Data Analysis

We employed descriptive statistics (e.g., Mean value) to analyze the data gathered from the demographic questionnaire. We coded the interview data using open coding to generate themes [21]. As to the recorded interaction activities, we replayed each participant’s recorded task session and examined the texts and URLs the participant highlighted as the answer or relevant information, and judged task success accordingly (see Sect. 4.4). Additionally, we calculated each participant’s dwell time on each search task. As to recorded eye-tracking data, we used heatmaps and AOIs generated by Tobii Studio. Heatmaps provide visualizations that show the general distribution of a participant’s gaze points and the elements on a page that receives more attention than others. AOIs are applied to investigate where the participants looked first (time to the first fixation), or elements on a page that first attract a participant’ attention. We also generated fixation counts, that is, the number of times a participant looked in an AOI on a page. Typically, an area that receives higher fixation count than other areas on a page could indicate difficulty in locating information [22].

5 Results

We present the results in the context of the two research questions that guided this study: (1) How do international doctoral students interact with the University of Tennessee recruitment websites/pages?, and (2) How do international doctoral students evaluate the usability of the University of Tennessee recruitment websites/pages based on their own experiences? We present the participants’ demographic and background information first, followed by the results that answered the research questions.

5.1 Participants’ Demographics

Five international doctoral students participated in this study, one male and four females, aged 28 to 43 years. Three participants are from Asia and two are from Europe.

5.2 Participants’ Interactions with Recruitment Websites

Participants performed six assigned search tasks. Their task outcomes were measured based on success, dwell time, and time-based efficiency. Their interactions with recruitment websites/ pages were analyzed using heatmaps and AOIs.

Success on Search Tasks.

Each participant completed six search tasks, totaling 30 across the five participants. The average success rate on tasks one to six (T-1 to T-6) was 0.7, 1.0, 0.6, 1.0, 1.0, and 1.0, respectively; and the grand average of the participants’ success across the six tasks is 0.88 (Table 2).

Table 2. Task success, dwell time, and time-based efficiency

Dwell Time.

Participants took the shortest time viewing pages for T-1 and the longest time viewing pages for T-5 (93.60 s versus 216.20 s respectively). The average dwell time on T-1 to T-6 across the five participants is 140.73 s (Table 2).

Time-Based Efficiency.

As shown in Table 2, the average time-based efficiency per task ranged from 94.8 s (T-6) to 216.2 s (T-5). The grand average of time-based efficiency across the six search tasks the five participants performed is 159.32 s (Table 2).

Eye-Tracking Data.

We analyzed eye movement and fixations on three web pages that all five participants used in performing the six search tasks: (1) UTK Graduate School Admissions (https://gradschool.utk.edu/admissions/), (2) College of Communication and Information (CCI) homepage (https://www.cci.utk.edu/), and (3) CCI PhD Admissions page (https://www.cci.utk.edu/phdadmissions).

UTK Graduate School Admissions.

Participants used this page to complete T-1 and T-4. Figure 1 shows the two sections (in yellow and red) that received the participants’ most attention. The first section is the middle column of the page titled, Office of Graduate Admissions. On this page, the top paragraph in this column has a description of the goal of the Office of Graduate Admissions and provides links to a “wide range of graduate degree offerings, how and when to apply, and what to expect upon admission”, among others. The second section is the right side menu with links to the Office of Graduate Admissions, Applying to Graduate School, Application Status, Application Deadlines, Graduate Programs, and After Admission. Participants also gazed at the left side menu, but paid more attention to information in the middle column of the page and the left right menu items.

Fig. 1.
figure 1

Heatmap of the UTK Graduate School Admission web page (Color figure online)

We generated AOIs around eight areas that received eye fixation on the UTK Graduate School Admissions page. As seen in Fig. 2, these areas are: the search bar (A), left side menu (B), cover image (C), first paragraph of text in the middle column (D), other paragraphs on the page (E), short description referring to international students (F), right side menu (G), and contact information (H).

Fig. 2.
figure 2

AOIs on the UTK Graduate School Admission web page

Figure 3 shows that the participants’ mean time to first fixation on the page is longest on (F) (Mean time = 84.88 s), followed by (A) (Mean time = 70.71 s), and (E) (Mean time = 11.76 s), while it is shortest on H, D, C, B, and E, respectively. As to eye fixation count, participants fixated most on area (G) (Mean count = 41.14 times), followed by (D) (Mean count = 39 times), and (B) (Mean count = 31.57 times), while they fixated the least on areas E, C, H, A, and F, respectively (Fig. 3).

Fig. 3.
figure 3

AOIs (Time to First Fixation Mean and Fixation Count Mean) on the UTK Graduate School Admissions web page

CCI Homepage.

Participants viewed the CCI homepage as a starting point for T-2, T-3, T-5, and T-6. They seemed to browse the whole page to find information on these tasks (Fig. 4). Three parts received participants’ highest attention (in red) on this page: (1) the middle part of left side menu, which includes the labeled links: Graduate, Schools, People, Research, and Alumni & Friends; (2) the left and right bottom part of the image (located on the left side of the middle column), and (3) the top part of the main content that includes two news headlines. We generated AOIs (Fig. 5) around eight parts of the page that received eye fixation. These are: the search bar (A), left side menu (B), blank area on the left side of the page (C), cover image at the top page (D), cover image caption (E), main content-News (F), Upcoming Events (G), and More News (H).

Fig. 4.
figure 4

Heatmap of the CCI Homepage (Color figure online)

Fig. 5.
figure 5

AOIs on the CCI Homepage

Figure 6 shows that the participants’ mean time to first fixation on the page is longest on G (Mean time = 16.15 s), while it is shortest on areas D, H, B, C, A, F, and E, respectively. As to eye fixation count, participants fixated most on area (F) (Mean count = 84.4 times), followed by (G) (Mean count = 36 times), (B) (Mean count = 34 times), and D (Mean count = 26.8 times); while they fixated the least on areas C, E, H, and A, respectively.

Fig. 6.
figure 6

AOIs (Time to First Fixation Mean and Fixation Count Mean) on the CCI Homepage

CCI PhD Admissions Web Page.

Participants used this page to find the answers for search task (T-5) (i.e., application requirements, application deadline, and application link). Figure 7 shows that participants spent the longest time from first fixation on three parts of the page (in red and yellow): Application link, Application deadline, and Financial Aid. We generated AOIs (Fig. 8) around eight areas that received eye fixations on the page: the search bar (A), left side menu (B), and five parts of the main content of the page, including the blank area on the left side of the page (C), requirements (D), application link (E), requirements in another part of the page (F), deadline (G) and financial aid and other information (H).

Fig. 7.
figure 7

Heatmap of the CCI PhD Admission web page (Color figure online)

Fig. 8.
figure 8

AOIs on the CCI PhD Admission web page

Figure 9 illustrates that participants mean time to first fixation on the page is longest on area (C) (Mean time = 45.44 s), followed by (B) (Mean time = 36.24 s), (G) (Mean time = 25.35 s), and (H) (Mean time = 23.96 s), while it is shortest on areas F, E, and D, respectively. As to eye fixation count, participants fixated most on area (G) (Mean count = 43.75 times), followed by (D) (Mean count = 30.25 times), (E) (Mean count = 26.26 times), (F) (Mean count = 26.25 times), and (H) (Mean count = 25.25 times). It is noteworthy that area (A) or the search bar received zero count, indicating that participants browsed and did not perform any searches in the three pages to find information on search task five (T-5).

Fig. 9.
figure 9

AOIs (Time to First Fixation Mean and Fixation Count Mean) on the CCI PhD Admission web page

5.3 Perceptions of Websites Usability

We interviewed the participants upon completion of the five search tasks and gathered insights about their experiences, including their assessment of the usability of the UTK and CCI websites/pages. We categorized their responses into six themes: navigation, content, organization, visual design, learnability, and user-centered design.

Navigation.

Participants experienced issues related to navigation, including misleading headings, difficulty locating information, and spending more time than expected in finding information. For example, one participant said: I know there is a lot of information, but I’m not sure how to navigate through the website. (P05). Another participant preferred using Google to find the information instead of navigating the Graduate School’s website: I probably need 30 s to find the answer, but it will be much faster by just searching Google with a few words (P02).

Content.

Participants commented on dated information they encountered on the sites/pages and two mentioned dead links on the CCI faculty web pages. These two issues caused frustration. One participant noted: They need to update a little bit more frequently. You can see the most updated information was last year actually in November (P02). Another participant indicated: I couldn’t click on the link and get inside to see their research interests and stuff because it was not working. So I felt really uncertain and kind of frustrated (P01).

Organization.

Two participants commented on redundant information across different web pages. One participant said: It seems to be some information is repeating on some of the pages, but some information is so hard to find because it’s too general (P03). The other participant suggested change to the organization of websites across units (e.g., College, Schools/Departments, and the Graduate School): There are 20 bars. It’s just too much. I’ll say narrow it down or summarize it better, like to Apply, Go [Click] here. (P05).

Visual Design.

Participants suggested improving the visual design of UTK and CCI websites/pages, including augmenting pictures and using different colors to highlight important or key information. One participant commented: Probably they can highlight some information, use highlight in different color to make people know this is important (P02). Another participant suggested augmenting the CCI page with photos of students and the College and improving the quality of posted photos: …Like some alumni who got achievement recently, we have a photo on the home page of our program. I think this is a good promotion information (P05).

Learnability.

Two participants found the interfaces of the websites and pages hard to learn. One participant mentioned: I wouldn’t be able to figure out it all on my own where to go and where to find it starting from let’s say the calendar, I needed to email someone (P03). The other participant noted that at the time of applying to the doctoral program, he/she needed physical human assistance to get the information from the website…The information is out there, just I couldn’t get to it (P05).

User-Centered Design.

All participants perceived no differences between international students and native speakers in terms of how the websites/pages are designed, organized, and written. They indicated that if the university or academic programs aim to attract international applicants, their websites should provide a specific section or even a page designed for international students. For example, one participant mentioned: Probably they want to have a separate button just for international applicants where there’ll be all information just for international applicants (P03). Another participant commented on feeling welcomed as an international student: I think probably you can take a photo of the international students and put it on the home page to let applicants from other countries to know that they are not alone … and [that] you are welcome here (P02).

6 Discussion

6.1 Search Task Outcomes

Success.

Participants completed four of the six search tasks successfully (success rate = 66%). The two tasks that participants did not complete successfully are T-1 and T-3. For T-1, participants had to find a page that best provide background information about UTK. One participant, for example, chose the CCI PhD Admission page while the other participant viewed the UTK International Student and Scholar Services. Although the latter is specifically designed for international students, it provides information about what to do after being admitted to UTK.

As to T-3, participants had to find possible career development information after graduating from UTK. The two participants who did not complete this task successfully indicated that such information is not available on the site. Eye-tracking data shows that these two participants browsed the CCI page and, although they fixated their eyes on the CCI Doctoral Students & Alumni link, they did not click on it. Instead, one participant clicked on links at bottom of the left side menu including, Diversity & Communication in the Workplace, and the other participant visited the CCI Doctoral Students & Alumni page that has a link to the PhD Alumni Spotlights but did not click on it. It could be that the labeling of this link was not meaningful to that participant. It is noteworthy that while the PhD Alumni Spotlights provides information about featured alumni and their careers, the CCI website does not have a separate category or page on careers. In a previous study [14], we found that international doctoral students were interested in finding information on future careers on the university and program websites that they visited. The study by [23] echoes these findings. Accordingly, we suggest that doctoral program recruitment websites not only include the achievements of featured alumni, but also provide a page dedicated to careers.

Dwell Time and Time-Based Efficiency.

As was anticipated, participants spent the shortest time viewing pages on T-1, which is fact-based, and longest time on T-5. The latter task is somewhat complex as it has three requirements or facets (a, b, and c) combining findings target answers and relevant information. As was found in a previous study, the more complex a search task is in terms of its requirements or facets, the longer it takes to complete [18].

By combining dwell time and success, participants’ time-based efficiency across the six search tasks is surprising (Mean = 159.32 s or nearly 2.5 min). This could be due to failure on two of the six search tasks. Though inconclusive, analysis of additional data from additional participants (in progress) should provide better understanding of participants’ time-based efficiency.

Eye Fixation.

The findings show that the dark background in both the UTK Graduate School Admissions page received the longest first fixation time. Long fixation time is negatively associated with search efficiency [22, 24]. To be noted, the text with the dark blue background in this page is specifically intended for international applicants and contains a link to Admissions for International Students page. Surprisingly, no participant clicked on that link. The location of this text toward the bottom of the page could have minimized the participants’ perception of its importance. Additionally, the visual design may not have supported the participants’ cognitive and physical affordances [25]. Accordingly, we suggest that website designers avoid placing key information on dark backgrounds or in locations that are not prominent on a web page.

6.2 Usability Evaluation

Participants thought that while they were able to use the UTK Graduate Admissions and CCI recruitment web pages, the faced issues pertaining to the content, organization, visual design, and learnability. Lack of user-centered design exacerbated these problems. This finding aligns with Sundeen et al. [5] whose study revealed that students need to find key information easily on websites and without resorting to others (e.g., staff) for assistance. As the usability of recruitment websites and pages influence prospective students’ perceptions of a university [9], they should be designed from the students’ perspectives to support their information needs and interaction behaviors.

6.3 Implications

The findings from this study are preliminary. Nevertheless, they have implications for improving the usability of the UTK Graduate School Admissions and College of Communication and Information websites/pages to support international doctoral students’ information needs and interaction behaviors.

6.4 Limitations

One of the main limitations of this study resides in the small sample size. However, as was mentioned earlier, this study reports the preliminary findings based on five international doctoral students. Although the sample is small, the findings provide a strong basis for recruiting additional participants to further evaluate the usability of the UTK recruitment websites and pages.

Another limitation resides in the nature of interviews used in this study. Interviews are based on participants’ self-reporting and, therefore, tend to be prone to inaccuracies.

7 Conclusion

The case study reported in this paper was conducted at the University of Tennessee, Knoxville campus. It explored international doctoral students’ interactions with the recruitment websites/pages of the Graduate School Admissions and College of Communication and Information using eye tracking. It employed exit interviews and gathered data from participants about their experiences with these websites and pages. Participants performed six assigned search tasks. Their eye movements and fixations were calculated to detect AOIs on these pages. Combining eye-tracking with exit interviews provided understanding of the participants’ interaction behaviors and unveiled usability issues in these websites/pages from their perspectives. Interestingly, participants seemed to prefer browsing through the websites/pages to searching, especially on the three web pages for which we calculated fixation times and counts. Although inconclusive, this could be due to the nature of the search tasks they performed. Future studies should elicit participants’ reasons for browsing instead of searching. Although the findings are based on five participants, they convey the need for additional research in this area of study. The six search tasks the participants performed in this study are based on international doctoral students’ information needs expressed in a previous study [14]. These tasks should be effective for use in future work.