Abstract
The economic census, conducted by the U.S. Census Bureau every five years, collects operational and performance data from roughly 4 million U.S. business establishments in non-farming industries. The level of detail requested, along with wide variation in the complexity of businesses, requires a highly sophisticated and flexible software design for online data collection. Thus, for the 2017 Economic Census, the Census Bureau redesigned and expanded its Web survey instrument, providing respondents with several new, more convenient features to help them complete the economic census.
Based on usability testing, debriefing phone calls, discussions with subject matter experts, and analysis of help calls received by the Census Bureau, we identified several features of the re-engineered website that were potentially problematic for users. Ten ‘how-to’ videos were created to assist respondents in navigating and utilizing these features to complete the Economic Census. Video topics included a general overview of the main screen, as well as specific action-items such as how to print, review data, and download spreadsheets.
Researchers conducted cognitive interviews with business respondents in order to evaluate and refine the how-to videos. The goal was to identify the best content for the videos that would address the most frequent user issues. During the interviews, participants completed several tasks designed to emulate the actions they would take when completing the actual survey. Participants then viewed the how-to videos, and provided feedback for each one.
Testing revealed strong support for the implementation of the videos. These videos helped orient respondents to the overall site and its key features. Ultimately, adding effective and readily accessible tutorial videos to the survey website may reduce help center workload and decrease the likelihood of respondent breakoff.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
The economic census is a survey produced by the Economic Directorate of the U.S Census Bureau that collects information on U.S. businesses. This survey collects data from more than 4 million U.S establishments in over 1000 industries. Respondents to this survey answer questions regarding their company’s employment records, payroll, revenue and sales receipts among other information. The economic census is conducted every 5 years. Data gathered from this program contributes to key economic indicators such as the gross domestic product (GDP) and is an effective means of understanding the state of the American economy.
For the 2017 Economic Census, the Census Bureau updated its existing method of data collection. Up until the mid-2000’s, data for the economic census was collected via a software program that respondents would upload onto their computer and enter their data into.
For the 2017 Economic Census survey, Census Bureau programmers developed an entirely new web-based electronic reporting system to host the survey. The web instrument titled ‘Centurion’ is a complex site that offers participants numerous ways to interact with their company’s data and report for it. Respondents must navigate through several necessary steps built into the site in order to submit their data.
Due to the feature-heavy nature of the reporting platform, usability testing was conducted to ensure that the site was functional for novice users, and that all the necessary features were easily accessible and usable. This usability testing, as well as multiple debriefing phone calls and discussions with subject matter experts revealed several confusing features on the Centurion site that may be effectively addressed with short how-to videos.
Ten videos were designed to help orient respondents to the overall site and its’ key features. Each video lasted no more than two minutes and detailed a feature on the site, while a narrator described the actions being taken on the screen. The short video topics included a walkthrough of the login process, a general overview of the site’s main dashboard, as well as specific directions to complete different tasks that were problematic for respondents.
Areas of the site that were difficult for respondents to navigate were determined through usability testing, respondent debriefings and through analysis of the records of the calls received by Centurion customer service.
Preliminary testing was conducted to assess the effectiveness of these new tutorial videos, and to identify the optimal location within the website to place them. The study sessions were designed so that participant’s feedback and actions would assist in identifying what improvements could be made to the videos once they are revised and finalized.
The goal of this study was to identify the best content for the videos that would address the most frequent user issues, as well as the best location for the videos. Effective and readily accessible tutorial videos may reduce call center burden, and decrease the likelihood of respondent breakoff.
2 Methods
Researchers conducted cognitive interviews with 12 respondents in order to evaluate and refine the new tutorial videos for the Centurion site. Testing was conducted during the winter months of 2017. Researchers traveled locally within the DC, Maryland and northern Virginia regions to interview respondents at their place of business. Participants were recruited via several Census Survey respondent files including the Economic Census and Annual Survey of Manufactures/Report of Organization (ASM/COS).
Attempts were made to recruit participants from businesses of varying sizes. In total, twelve businesses were recruited: four single-establishment businesses, six medium-sized (20–2499 employees) businesses and two small (1–19 employees) multi-establishment businesses.
Participants held a wide range of positions in their company, a plurality being associated with an accounting position. On average, participants were in their respective positions for 8.6 years. Participants for this study fell into a wide range of ages, with an average age of 38 years. Participants were well educated; all but one participant had earned a Bachelor’s degree. See Table 1 for all participant characteristics.
During the interviews, participants first filled out the consent form and several basic demographic questions. Every session was recorded with a screen-capturing program after getting consent. Following this, participants were brought to the Centurion website and asked to complete six tasks. These tasks were in line with the actions they would take while completing the actual economic census survey. All participants completed the same tasks.
The tasks outlined for participants were to (a) login to the site; (b) print the survey questions; (c) start the survey; (d) add a new location (e) download/upload Spreadsheet template; and finally to (f) find assistance on the site. These tasks were simplified versions of those that were used during usability testing of the Economic Census instrument in May and June of 2017.
After finishing these tasks, participants completed the second half of the study which involved viewing 5 how-to videos. Each video was less than two minutes long. There was a total of 10 how-to videos produced, but the number of videos the participants viewed was narrowed down to 5 due to time constraints. The topic of the videos that the participants viewed were predetermined to ensure that each video would be viewed approximately the same amount of times.
After viewing, participants were asked for feedback on the characteristics and content of the videos by answering on a Likert scale ranging from 1–7. The questions asked after each video included (a) how effective the video was in addressing the topic or target issue, (b) how appropriate the length of the video was, (c) how satisfied overall they were with it, (d) how likely they would be to recommend that someone who is having issues using the site watch the how-to videos, and finally participants were provided with a write-in box to describe any other thoughts or comments about the video.
After the viewing portion of the session was complete, respondents were asked several debriefing questions including what other ways the site can offer assistance, and how developers can best direct users to the how-to videos. They were also asked for their opinions regarding the ideal location for the videos on the website.
3 Results
Testing revealed several important aspects of the how-to videos that can be improved upon, as well as some potential improvements that can be made to the Centurion site to create a more user-friendly experience.
During the testing session, participants completed 6 tasks on the Centurion site that emulated the actions they would take when filling out the actual economic census. There was a 100% success rate for those that partook (technical difficulties prevented 1 participant from finishing the tasks). Although the tasks were relatively simple in nature, had there been substantial issues on the site, they likely would have been illuminated by participants attempting these tasks.
Participant feedback offered several findings and suggestions that can be generalized across all the videos. These included improving the sound quality, and improving full screen resolution. It was also recommended that the finalized videos include closed captioning. Overall ratings of the videos on a 7 point scale were very high. The average rating across all videos was 6.2. See Table 2 for the results of the video satisfaction questionnaire.
After viewing the videos, participants completed a questionnaire designed to ascertain experience with internet and computer use. For this project, we amended the questions to include items regarding one’s use of how-to videos in their daily life. The participants recruited for this study were all savvy internet/computer users. Their actions regarding use of how-to videos were more variable. This being said, all but one participant reported examples of how-to videos they’ve watched recently, and a majority, (n = 7) reported having viewed 1–2 how-to videos in the last month. Some examples of these video topics provided referred to handy do-it-yourself projects, and videos regarding complex software walk-throughs. See Tables 3 and 4 for several highlights of the how-to video use questionnaire.
Finally, at the end of the session participants answered several debriefing questions, some suggestions and responses resulting included bringing the link to the videos closer to the main reporting dashboard (as opposed to the uppermost banner of the site); offering a live chat as another means of assistance; and including a prompt in the overview of the site directing users to the videos.
4 Conclusion
The pretesting research offered valuable feedback on refining the content of the videos, and determining the best location for them within the website. The usability findings were overall positive with regard to both the functionality of the Centurion site, and the general effectiveness of the how-to videos. Hopefully, the implementation of how-to videos to the survey platform will be a valuable asset to respondents who are experiencing any difficulties during the reporting process.
Following the mail-out of the 2017 Economic Census, researchers will be able to examine the paradata generated by respondents, to analyze user behaviors such as how many times the how-to videos link has been clicked, and furthermore, how many views each videos has. Analysis of the paradata may also lead to informative debriefing calls with respondents who utilized them. Future paradata analysis will also reveal if there is a significant overall decrease in call-center inquiries and respondent breakoff.
This study revealed numerous lessons for any future videos that may be created. Some guidelines for video development include putting substantial effort into the creation of the video scripts that the narrator reads in each video. For this study, drafts were produced and then edited and reviewed by several subject-matter experts in various departments of the Census Bureau. Further work on the frontend of the project included putting extensive work into determining the best topic for each video. These decisions were informed by usability testing of the web-instrument, debriefing calls with past respondents, and discussions with help-center staff. Finally, if resources allow, having access to high-quality recording equipment and a quiet space to record the narration appears to go a long way in convincing respondents of the legitimacy and overall quality of the videos, as several participants noted that background noise was a distraction to them as they tested out the pilot videos created for this study.
Ultimately, no amount of guesswork can be a proxy for observing a novice user’s interactions with an interface. Testing the survey platform and videos with participants was a critical component of understanding what improvements could be made. By employing multiple methods of analysis such as examining the accuracy of the tasks the participants completed, various Likert scales, and verbal feedback, researchers were able to use qualitative and quantitative means to assess the effectiveness of the videos and overall platform. Results of this study suggest future Census Bureau products may benefit from the implementation of how-to videos, particularly on websites and platforms that are newly developed and/or have numerous features built into the instrument.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 This is a U.S. government work and its text is not subject to copyright protection in the United States; however, its text may be subject to foreign copyright protection
About this paper
Cite this paper
Keegan, R. (2018). Developing Instructional Videos to Direct Business Respondents Through a Feature-Heavy Online Survey Instrument. In: Stephanidis, C. (eds) HCI International 2018 – Posters' Extended Abstracts. HCI 2018. Communications in Computer and Information Science, vol 852. Springer, Cham. https://doi.org/10.1007/978-3-319-92285-0_40
Download citation
DOI: https://doi.org/10.1007/978-3-319-92285-0_40
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-92284-3
Online ISBN: 978-3-319-92285-0
eBook Packages: Computer ScienceComputer Science (R0)