skip to main content
10.1145/2442882.2442891acmotherconferencesArticle/Chapter ViewAbstractPublication Pagesacm-devConference Proceedingsconference-collections
research-article

Man versus machine: evaluating IVR versus a live operator for phone surveys in India

Published: 11 January 2013 Publication History

Abstract

Many organizations in the developing world need to conduct phone surveys to collect data from low-income respondents. Such organizations generally have two options: employ a live operator, or utilize interactive voice response (IVR). Despite the relevance of this question, we are unaware of any work that rigorously compares the accuracy, speed, and cost of an IVR survey relative to a live operator.
In this paper, we address these questions by giving two identical interviews -- one using IVR, and one using a live operator -- to 31 low-income job seekers in India. The IVR interview included a brief introduction by a live operator, to provide context for the call. Out of the 20 people who completed both surveys, we found that IVR incurs a 4.0% error rate (95% C. I. 2.5% -- 6.1%) and requires 2.5 times longer for users. We summarize our experience as a set of recommendations for practitioners in this space.

References

[1]
Gram Vaani: http://www.gramvaani.org.
[2]
S. K. Agarwal, A. Kumar, A. A. Nanavati, and N. Rajput. User-generated content creation and dissemination in rural areas. ITID, 6(2), 2010.
[3]
K. Buckstaff, D. McLain, and T. Szybalski. Benchmarking customer service. In Public Power, June 2008.
[4]
B. Clark and B. Burrell. Freedom fone: dial-up information service. In ICTD, 2009.
[5]
R. Corkrey and L. Parkinson. Interactive voice response: Review of studies 1989--2000. Behavior Researc Meth., 2002.
[6]
W. H. Curioso, B. T. Karras, P. E. Campos, C. Buendia, K. K. Holmes, and A. M. Kimball. Design and Implementation of Cell-PREVEN: A Real-Time Surveillance System for Adverse Events Using Cell Phones in Peru. AMIA Annu Symp Proc, 2005.
[7]
D. A. Dillman, G. Phelps, R. Tortora, K. Swift, J. Kohrell, J. Berck, and B. L. Messer. Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet. Social Science Research, 38(1):1--18, 2009.
[8]
D. Gardner-Bonneau, H. E. Blanchard, and B. Suhm. Ivr usability engineering using guidelines and analyses of end-to-end calls. In Human Factors and Voice Interactive Systems, Signals and Communication Technology. Springer, 2008.
[9]
A. S. Grover, K. Calteaux, E. Barnard, and G. van Huyssteen. A voice service for user feedback on school meals. In ACM DEV, 2012.
[10]
A. S. Grover, M. Plauché, E. Barnard, and C. Kuun. HIV health information access using spoken dialogue systems: touchtone vs. speech. In ICTD, 2009.
[11]
C. Hartung, Y. Anokwa, W. Brunette, A. Lerer, C. Tseng, and G. Borriello. Open data kit: Tools to build information services for developing regions. In ICTD, 2010.
[12]
Z. Koradia, C. Balachandran, K. Dadheech, M. Shivam, and A. Seth. Experiences of deploying and commercializing a community radio automation system in india. In ACM DEV, 2012.
[13]
Z. Koradia and A. Seth. Phonepeti: exploring the role of an answering machine system in a community radio station in india. In ICTD, 2012.
[14]
F. Kreuter, S. Presser, and R. Tourangeau. Social Desirability Bias in CATI, IVR, and Web Surveys. Public Opinion Quarterly, 72(5):847--865, 2008.
[15]
A. Kumar, D. Chakraborty, H. Chauhan, S. K. Agarwal, and N. Rajput. Folksomaps - towards community driven intelligent maps for developing regions. In ICTD, 2009.
[16]
A. Lerer, M. Ward, and S. Amarasinghe. Evaluation of IVR data collection UIs for untrained rural users. In ACM DEV, 2010.
[17]
I. Medhi, S. Patnaik, E. Brunskill, S. N. Gautama, W. Thies, and K. Toyama. Designing mobile interfaces for novice and low-literacy users. ACM Transactions on Compututer-Human Interaction, 18(1), May 2011.
[18]
P. Mudliar, J. Donner, and W. Thies. Emergent practices around CGNet Swara, voice forum for citizen journalism in rural India. In ICTD, pages 159--168, 2012.
[19]
F. Oberle. Who, why and how often? key elements for the design of a successful speech application taking account of the target groups. In Usability of Speech Dialog Systems, Signals and Commmunication Technologies. Springer, 2008.
[20]
B. Odero, B. Omwenga, M. Masita-Mwangi, P. Githinji, and J. Ledlie. Tangaza: frugal group messaging through speech and text. In ACM DEV, 2010.
[21]
N. Patel, S. Agarwal, N. Rajput, A. Nanavati, P. Dave, and T. S. Parikh. A comparative study of speech and dialed input voice interfaces in rural india. In CHI, 2009.
[22]
N. Patel, D. Chittamuru, A. Jain, P. Dave, and T. S. Parikh. Avaaj otalo: a field study of an interactive voice forum for small farmers in rural india. In CHI, 2010.
[23]
N. Patel, S. R. Klemmer, and T. S. Parikh. An asymmetric communications platform for knowledge sharing with low-end mobile phones. In UIST Adjunct, 2011.
[24]
S. Patnaik, E. Brunskill, and W. Thies. Evaluating the Accuracy of Data Collection on Mobile Phones: A Study of Forms, SMS, and Voice. In ICTD, 2009.
[25]
M. Plauché and M. Prabaker. Tamil market: a spoken dialog system for rural india. In CHI EA, 2006.
[26]
A. A. Raza, M. Pervaiz, C. Milo, S. Razaq, G. Alster, J. Sherwani, U. Saif, and R. Rosenfeld. Viral entertainment as a vehicle for disseminating speech-based services to low-literate users. In ICTD, 2012.
[27]
N. Sambasivan, J. Weber, and E. Cutrell. Designing a phone broadcasting system for urban sex workers in india. In CHI, 2011.
[28]
K. Schroder, C. Johnson, and J. Wiebe. Interactive voice response technology applied to sexual behavior self-reports: A comparison of three methods. AIDS and Behavior, 11, 2007.
[29]
A. Sharma Grover and E. Barnard. The lwazi community communication service: design and piloting of a voice-based information service. In WWW, 2011.
[30]
J. Sherwani, N. Ali, S. Mirza, A. Fatma, Y. Memon, M. Karim, R. Tongia, and R. Rosenfeld. Healthline: Speech-based access to health information by low-literate users. In ICTD, 2009.
[31]
J. Sherwani, S. Palijo, S. Mirza, T. Ahmed, N. Ali, and R. Rosenfeld. Speech vs. touch-tone: Telephony interfaces for information access by low literate users. In ICTD, 2009.
[32]
M. Thulasingam and P. K. Cheriyath. Telephone Survey as a Method of Data Collection in South India. Indian Journal of Community Medicine, 33(4), 2008.
[33]
A. Vashistha and W. Thies. IVR Junction: Building Scalable and Distributed Voice Forums in the Developing World. In NSDR, 2012.

Cited By

View all
  • (2022)Understanding How to Administer Voice Surveys through Smart SpeakersProceedings of the ACM on Human-Computer Interaction10.1145/35556066:CSCW2(1-32)Online publication date: 11-Nov-2022
  • (2021)‘Oshudh Poro’: A Mobile-Phone Application to Support Low-literate Rural Bangladeshi People’s Personal Medication Management [Poster]Proceedings of the 4th ACM SIGCAS Conference on Computing and Sustainable Societies10.1145/3460112.3471975(457-461)Online publication date: 28-Jun-2021
  • (2021)Costs and Benefits of Conducting Voice-based Surveys Versus Keypress-based Surveys on Interactive Voice Response SystemsProceedings of the 4th ACM SIGCAS Conference on Computing and Sustainable Societies10.1145/3460112.3471963(288-298)Online publication date: 28-Jun-2021
  • Show More Cited By

Index Terms

  1. Man versus machine: evaluating IVR versus a live operator for phone surveys in India

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    ACM DEV '13: Proceedings of the 3rd ACM Symposium on Computing for Development
    January 2013
    233 pages
    ISBN:9781450318563
    DOI:10.1145/2442882
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 11 January 2013

    Permissions

    Request permissions for this article.

    Check for updates

    Qualifiers

    • Research-article

    Conference

    ACM DEV '13

    Acceptance Rates

    Overall Acceptance Rate 52 of 164 submissions, 32%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)3
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2022)Understanding How to Administer Voice Surveys through Smart SpeakersProceedings of the ACM on Human-Computer Interaction10.1145/35556066:CSCW2(1-32)Online publication date: 11-Nov-2022
    • (2021)‘Oshudh Poro’: A Mobile-Phone Application to Support Low-literate Rural Bangladeshi People’s Personal Medication Management [Poster]Proceedings of the 4th ACM SIGCAS Conference on Computing and Sustainable Societies10.1145/3460112.3471975(457-461)Online publication date: 28-Jun-2021
    • (2021)Costs and Benefits of Conducting Voice-based Surveys Versus Keypress-based Surveys on Interactive Voice Response SystemsProceedings of the 4th ACM SIGCAS Conference on Computing and Sustainable Societies10.1145/3460112.3471963(288-298)Online publication date: 28-Jun-2021
    • (2021)Karamad: A Voice-based Crowdsourcing Platform for Underserved PopulationsProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445417(1-15)Online publication date: 6-May-2021
    • (2020)Rethinking Design of Digital Platforms for Emergent Users: Findings from a Study with Rural Indian FarmersProceedings of the 11th Indian Conference on Human-Computer Interaction10.1145/3429290.3429297(62-69)Online publication date: 5-Nov-2020
    • (2019)Strengths and limitations of computer assisted telephone interviews (CATI) for nutrition data collection in rural KenyaPLOS ONE10.1371/journal.pone.021005014:1(e0210050)Online publication date: 30-Jan-2019
    • (2019)ReCallProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300399(1-13)Online publication date: 2-May-2019
    • (2015)User Interface Design for Low-literate and Novice UsersFoundations and Trends in Human-Computer Interaction10.1561/11000000478:1(1-72)Online publication date: 1-Mar-2015
    • (2015)Building citizen engagement into the implementation of welfare schemes in rural IndiaProceedings of the Seventh International Conference on Information and Communication Technologies and Development10.1145/2737856.2738027(1-10)Online publication date: 15-May-2015
    • (2015)Testing in the FieldProceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems10.1145/2702613.2732893(1061-1066)Online publication date: 18-Apr-2015
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media