research-article

Participatory Sensing or Participatory Nonsense?: Mitigating the Effect of Human Error on Data Quality in Citizen Science

Authors:

Matthias Budde,

Andrea Schankin,

Julien Hoffmann,

Michael BeiglAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 1, Issue 3

Article No.: 39, Pages 1 - 23

https://doi.org/10.1145/3131900

Published: 11 September 2017 Publication History

Abstract

Citizen Science with mobile and wearable technology holds the possibility of unprecedented observation systems. Experts and policy makers are torn between enthusiasm and scepticism regarding the value of the resulting data, as their decision making traditionally relies on high-quality instrumentation and trained personnel measuring in a standardized way. In this paper, we (1) present an empirical behavior taxonomy of errors exhibited in non-expert smartphone-based sensing, based on four small exploratory studies, and discuss measures to mitigate their effects. We then present a large summative study (N=535) that compares instructions and technical measures to address these errors, both from the perspective of improvements to error frequency and perceived usability. Our results show that (2) technical measures without explanation notably reduce the perceived usability and (3) technical measures and instructions nicely complement each other: Their combination achieves a significant reduction in observed error rates while not affecting the user experience negatively.

References

[1]

Harini Alagarai Sampath, Rajeev Rajeshuni, and Bipin Indurkhya. 2014. Cognitively Inspired Task Design to Improve User Performance on Crowdsourcing Platforms. In 32nd Annual ACM Conference on Human Factors in Computing Systems (CHI ’14). ACM, 3665--3674.

Digital Library

[2]

Victoria Bellotti and Keith Edwards. 2001. Intelligibility and accountability: human considerations in context-aware systems. Human--Computer Interaction 16, 2-4 (2001), 193--212.

Digital Library

[3]

Rick Bonney, Jennifer L Shirk, Tina B Phillips, Andrea Wiggins, Heidi L Ballard, Abraham J Miller-Rushing, Julia K Parrish, et al. 2014. Next steps for citizen science. Science 343, 6178 (2014).

[4]

John Brooke. 1996. SUS -- A quick and dirty usability scale. Usability evaluation in industry 189 (1996).

[5]

Matthias Budde, Pierre Barbera, Rayan El Masri, Till Riedel, and Michael Beigl. 2013. Retrofitting Smartphones to be Used as Particulate Matter Dosimeters. In International Symposium on Wearable Computers (ISWC’13).

Digital Library

[6]

Matthias Budde, Marcel Köpke, and Michael Beigl. 2015. Robust In-situ Data Reconstruction from Poisson Noise for Low-cost, Mobile, Non-expert Environmental Sensing. In International Symposium on Wearable Computers (ISWC’15). ACM, 179--182.

Digital Library

[7]

Matthias Budde, Rikard Öxler, Michael Beigl, and Jussi Holopainen. 2016. Sensified Gaming -- Design Patterns and Game Design Elements for Gameful Environmental Sensing. In 13th Int. Conference on Advances in Computer Entertainment Technology (ACE2016).

Digital Library

[8]

Matthias Budde, Lin Zhang, and Michael Beigl. 2014. Distributed, Low-cost Particulate Matter Sensing: Scenarios, Challenges, Approaches. In ProScience, Vol. 1. (Proc. 1st Int. Conf. Atmospheric Dust (DUST2014)).

[9]

Jeffrey A Burke, Deborah Estrin, Mark Hansen, Andrew Parker, Nithya Ramanathan, Sasank Reddy, and Mani B Srivastava. 2006. Participatory sensing. Center for Embedded Network Sensing (2006).

[10]

Delphine Christin, Andreas Reinhardt, Salil S. Kanhere, and Matthias Hollick. 2011. A survey on privacy in mobile participatory sensing applications. Journal of Systems and Software 84, 11 (2011). Mobile Applications: Status and Trends.

Digital Library

[11]

Anind K. Dey. 2001. Understanding and Using Context. Personal and Ubiquitous Computing 5, 1 (2001), 4--7.

Digital Library

[12]

Anind K. Dey and Alan Newberger. 2009. Support for Context-aware Intelligibility and Control. In CHI ’09. ACM, 859--868.

Digital Library

[13]

J St BT Evans. 1988. The knowledge elicitation problem: a psychological perspective. Behaviour 8 Information Technology 7, 2 (1988).

[14]

Mary M Gardiner, Leslie L Allee, Peter MJ Brown, John E Losey, Helen E Roy, and Rebecca Rice Smyth. 2012. Lessons from lady beetles: accuracy of monitoring data from US and UK citizen-science programs. Frontiers in Ecology and the Environment 10, 9 (2012), 471--476.

[15]

Steven K. Gibb. 2015. Volunteers Against Pollution. Chemical 8 Engineering News (C8EN) 93, 36 (Sept. 2015).

[16]

Mike Harding, Bran Knowles, Nigel Davies, and Mark Rouncefield. 2015. HCI, Civic Engagement 8 Trust. In 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ’15). ACM, New York, NY, USA, 2833--2842.

Digital Library

[17]

John D Hoyt and Harry Wechsler. 1994. Detection of human speech in structured noise. In Acoustics, Speech, and Signal Processing, 1994. ICASSP-94., 1994 IEEE International Conference on, Vol. 2. IEEE, II--237.

[18]

Kuan Lun Huang, Salil S Kanhere, and Wen Hu. 2010. Are you contributing trustworthy data?: the case for a reputation system in participatory sensing. In Modeling, analysis, and simulation of wireless and mobile systems.

Digital Library

[19]

Eiman Kanjo. 2010. Noisespy: A real-time mobile phone platform for urban noise monitoring and mapping. Mobile Networks and Applications 15, 4 (2010), 562--574.

Digital Library

[20]

Sunyoung Kim, Jennifer Mankoff, and Eric Paulos. 2013. Sensr: Evaluating a Flexible Framework for Authoring Mobile Data-collection Tools for Citizen Science. In Computer Supported Cooperative Work (CSCW ’13). ACM, 10.

Digital Library

[21]

Sunyoung Kim, Christine Robson, Thomas Zimmerman, Jeffrey Pierce, and Eben M. Haber. 2011. Creek Watch: Pairing Usefulness and Usability for Successful Citizen Science. In SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). ACM, 2125--2134.

Digital Library

[22]

Eli Kintisch. 2011. How to Grow Your Own Army of Citizen Scientists. (24 Feb. 2011).

[23]

Simon Klakegg, Chu Luo, Jorge Goncalves, Simo Hosio, and Vassilis Kostakos. 2016. Instrumenting Smartphones with Portable NIRS. In Adj. Proceedings UbiComp’16, Workshop in Ubiquitous Mobile Instrumentation (UbiMI).

Digital Library

[24]

N.D. Lane, E. Miluzzo, Hong Lu, D. Peebles, T. Choudhury, and A.T. Campbell. 2010. A survey of mobile phone sensing. IEEE Communications Magazine 48, 9 (2010).

Digital Library

[25]

B. Laugwitz, T. Held, and M. Schrepp. 2008. Construction and evaluation of a user experience questionnaire. In Symposium of the Austrian HCI and Usability Engineering Group. Springer, 63--76.

Digital Library

[26]

Kurt Luther, Scott Counts, Kristin B. Stecher, Aaron Hoff, and Paul Johns. 2009. Pathfinder: An Online Collaboration Environment for Citizen Scientists. In Human Factors in Computing Systems (CHI ’09). ACM, 239--248.

Digital Library

[27]

Nicolas Maisonneuve, Matthias Stevens, and Bartek Ochab. 2010. Participatory noise pollution monitoring using mobile phones. In Information Polity. Vol. 15. 51 -- 71. Issue 1.

Digital Library

[28]

Sebastian Matyas, Peter Kiefer, Christoph Schlieder, and Sara Kleyer. 2011. Wisdom about the Crowd: Assuring Geospatial Data Quality Collected in Location-Based Games. In International Conference on Entertainment Computing (ICEC 2011). 331--336.

Digital Library

[29]

George A. Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review 91 (1956), 81--97.

[30]

L. A. Muratori, P. Salomoni, and G. Pau. 2011. Feeling the pack: Strategies for an optimal participatory system to sense and recognize noise pollution. In 2011 IEEE International Conference on Consumer Electronics - Berlin (ICCE -Berlin). 17--21.

[31]

Donald A. Norman. 1983. Design Rules Based on Analyses of Human Error. Commun. ACM 26, 4 (April 1983), 254--258.

Digital Library

[32]

Donald A. Norman and Pieter Jan Stappers. 2015. DesignX: Complex Sociotechnical Systems. She Ji 1 (2015), 83--106. Issue 2.

[33]

DG Novick and K Ward. 2006. Why Don’t People Read the Manual?. In Int. Conference on Design of Communication (SIGDOC ’06). ACM.

Digital Library

[34]

L. Peterson and M. Peterson. 1959. Short-term retention of individual verbal items. Journal of Experimental Psychology 58 (1959).

[35]

Rajib Kumar Rana, Chun Tung Chou, Salil S Kanhere, Nirupama Bulusu, and Wen Hu. 2010. Ear-phone: an end-to-end participatory urban noise mapping system. In 9th ACM/IEEE International Conference on Information Processing in Sensor Networks. 105--116.

Digital Library

[36]

Sasank Reddy, Deborah Estrin, and Mani Srivastava. 2010. Recruitment Framework for Participatory Sensing Data Collections. In Pervasive Computing. LNCS, Vol. 6030.

Digital Library

[37]

I. Schweizer, R. Bärtl, A. Schulz, F. Probst, and M. Mühläuser. 2011. NoiseMap -- real-time participatory noise maps. In PhoneSense’11.

[38]

L. See, A. Comber, C. Salk, S. Fritz, M. van der Velde, C. Perger, C. Schill, I. McCallum, F. Kraxner, and M. Obersteiner. 2013. Comparing the Quality of Crowdsourced Data Contributed by Expert and Non-Experts. PLoS ONE 8, 7 (2013).

[39]

S Andrew Sheppard and Loren Terveen. 2011. Quality is a verb: the operationalization of data quality in a citizen science community. In Proceedings of the 7th International Symposium on Wikis and Open Collaboration. 29--38.

Digital Library

[40]

S. Andrew Sheppard, Andrea Wiggins, and Loren Terveen. 2014. Capturing Quality: Retaining Provenance for Curated Volunteer Monitoring Data. In 17th ACM Conference on Computer Supported Cooperative Work 8 Social Computing (CSCW ’14). ACM, 1234--1245.

Digital Library

[41]

F Snik, JHH Rietjens, A Apituley, H Volten, B Mijling, A Di Noia, S Heikamp, RC Heinsbroek, OP Hasekamp, and JM Smit. 2014. Mapping atmospheric aerosols with a citizen science network of smartphone spectropolarimeters. Geophysical Research Letters 41, 20 (2014).

[42]

John Sweller. 2002. Visualisation and instructional design. In International Workshop on Dynamic Visualizations and Learning.

[43]

Brett Amy Thelen and Rachel K. Thiet. 2008. Cultivating connection: Incorporating meaningful citizen science into Cape Cod National Seashore’s estuarine research and monitoring programs. Park Science 25, 1 (2008).

[44]

A. Truskinger, Haofan Yang, J. Wimmer, Jinglan Zhang, I. Williamson, and P. Roe. 2011. Large Scale Participatory Acoustic Sensor Data Analysis: Tools and Reputation Models to Enhance Effectiveness. In E-Science.

Digital Library

[45]

Kathleen Tuite, Noah Snavely, Dun-yu Hsiao, Nadine Tabing, and Zoran Popovic. 2011. PhotoCity: Training Experts at Large-scale Image Acquisition Through a Competitive Game. In SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). ACM, 10.

Digital Library

[46]

VDA-QMC. 2011. Messsystem und Messprozess sind zweierlei. QZ 56, 5 (2011).

[47]

Christopher D Wickens, John Lee, Yili D Liu, and Sallie Gordon-Becker. 2014. Introduction to Human Factors Engineering: Pearson New International Edition. Pearson Higher Ed. Second Edition.

Digital Library

[48]

Andrea Wiggins and Yurong He. 2016. Community-based Data Validation Practices in Citizen Science. In 19th ACM Conference on Computer-Supported Cooperative Work 8 Social Computing (CSCW ’16). ACM, 1548--1559.

Digital Library

[49]

Patricia Wright. 1981. “The instructions clearly state …” can’t people read? Applied Ergonomics 12 (September 1981), 131--141. Issue 3.

[50]

Poonam Yadav and John Darlington. 2016. Design Guidelines for the User-Centred Collaborative Citizen Science Platforms. arXiv preprint arXiv:1605.00910 (2016).

Cited By

Ghaffari APirttikangas SGilman E(2025)Understanding Well-Being in Urban Context: A SurveyIEEE Access10.1109/ACCESS.2024.350693513(11136-11158)Online publication date: 2025
https://doi.org/10.1109/ACCESS.2024.3506935
Gilman EBugiotti FKhalid AMehmood HKostakos PTuovinen LYlipulli JSu XFerreira D(2024)Addressing Data Challenges to Drive the Transformation of Smart CitiesACM Transactions on Intelligent Systems and Technology10.1145/366348215:5(1-65)Online publication date: 7-Nov-2024
https://dl.acm.org/doi/10.1145/3663482
Eisma JSchoups GDavids Jvan de Giesen N(2023)A Bayesian model for quantifying errors in citizen science data: application to rainfall observations from NepalHydrology and Earth System Sciences10.5194/hess-27-3565-202327:19(3565-3579)Online publication date: 9-Oct-2023
https://doi.org/10.5194/hess-27-3565-2023
Show More Cited By

Index Terms

Participatory Sensing or Participatory Nonsense?: Mitigating the Effect of Human Error on Data Quality in Citizen Science
1. Human-centered computing

Recommendations

Participatory sensing for community building
CHI EA '11: CHI '11 Extended Abstracts on Human Factors in Computing Systems

In this research, we explore the viability of using participatory sensing as a means to enhance a sense of community. To accomplish this, we are developing and deploying a suite of participatory sensing applications, where users explicitly report on the ...
Integrating participatory sensing in application development practices
FoSER '10: Proceedings of the FSE/SDP workshop on Future of software engineering research

With the widespread capabilities of commodity mobile devices, applications will increasingly incorporate participatory sensing functionality. Participatory sensing directly involves end-users in collecting (and ultimately sharing) information about the ...
Demonstration abstract: participatory sensing enabled environmental monitoring in smart cities
IPSN '14: Proceedings of the 13th international symposium on Information processing in sensor networks

This work describes a system demonstrator for environmental monitoring based on participatory sensing. Mobile environmental sensors carried by citizens are used to measure pollutant concentrations and provide this data to a Unified Sensing Platform. It ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 1, Issue 3

September 2017

2023 pages

EISSN:2474-9567

DOI:10.1145/3139486

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 September 2017

Accepted: 01 June 2017

Revised: 01 May 2017

Received: 01 February 2017

Published in IMWUT Volume 1, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

BMBF
FP7
BMVI

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

40
Total Citations
View Citations
849
Total Downloads

Downloads (Last 12 months)50
Downloads (Last 6 weeks)2

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ghaffari APirttikangas SGilman E(2025)Understanding Well-Being in Urban Context: A SurveyIEEE Access10.1109/ACCESS.2024.350693513(11136-11158)Online publication date: 2025
https://doi.org/10.1109/ACCESS.2024.3506935
Gilman EBugiotti FKhalid AMehmood HKostakos PTuovinen LYlipulli JSu XFerreira D(2024)Addressing Data Challenges to Drive the Transformation of Smart CitiesACM Transactions on Intelligent Systems and Technology10.1145/366348215:5(1-65)Online publication date: 7-Nov-2024
https://dl.acm.org/doi/10.1145/3663482
Eisma JSchoups GDavids Jvan de Giesen N(2023)A Bayesian model for quantifying errors in citizen science data: application to rainfall observations from NepalHydrology and Earth System Sciences10.5194/hess-27-3565-202327:19(3565-3579)Online publication date: 9-Oct-2023
https://doi.org/10.5194/hess-27-3565-2023
Kalupahana ABalaji AXiao XPeh L(2023)SeRaNDiPProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35962527:2(1-38)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3596252
Sharma NColucci-Gray Lvan der Wal RSiddharthan A(2022)Consensus Building in On-Line Citizen ScienceProceedings of the ACM on Human-Computer Interaction10.1145/35555356:CSCW2(1-26)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555535
Schankin ABudde MRiedel TBeigl M(2022)Psychometric Properties of the User Experience Questionnaire (UEQ)Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502098(1-11)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502098
Lin FLee CHo YChen YYen GChang Y(2022)What Kinds of Experiences Do You Desire? A Preliminary Study of the Desired Experiences of Contributors to Location-Based Mobile CrowdsourcingExtended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491101.3519744(1-7)Online publication date: 27-Apr-2022
https://dl.acm.org/doi/10.1145/3491101.3519744
Alfonso LGharesifard MWehn U(2022)Analysing the value of environmental citizen-generated data: Complementarity and cost per observationJournal of Environmental Management10.1016/j.jenvman.2021.114157303(114157)Online publication date: Feb-2022
https://doi.org/10.1016/j.jenvman.2021.114157
Izquierdo-Gómez D(2022)Synergistic use of facebook, online questionnaires and local ecological knowledge to detect and reconstruct the bioinvasion of the Iberian Peninsula by Callinectes sapidus Rathbun, 1896Biological Invasions10.1007/s10530-021-02696-024:4(1059-1082)Online publication date: 17-Jan-2022
https://doi.org/10.1007/s10530-021-02696-0
Li CRiedel TBeigl M(2022)Neural Kernel Network Deep Kernel Learning for Predicting Particulate Matter from Heterogeneous Sensors with UncertaintyInformation Integration and Web Intelligence10.1007/978-3-031-21047-1_22(252-266)Online publication date: 28-Nov-2022
https://dl.acm.org/doi/10.1007/978-3-031-21047-1_22
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents