skip to main content
10.1145/3282933.3282937acmconferencesArticle/Chapter ViewAbstractPublication PagesgisConference Proceedingsconference-collections
research-article

Putting User Reputation on the Map: Unsupervised Quality Control for Crowdsourced Historical Data

Published: 06 November 2018 Publication History

Abstract

In this paper we propose a novel method for quality assessment of crowdsourced data. It computes user reputation scores without requiring ground truth; instead, it is based on the consistency among users. In this pilot study, we perform some explorative data analysis on two real crowdsourcing projects by the New York Public Library: extracting building footprints as polygons from historical insurance atlases, and geolocating historical photographs. We show that the computed reputation scores are plausible and furthermore provide insight into user behavior.

References

[1]
Mohammad Allahbakhsh, Aleksandar Ignjatovic, Boualem Benatallah, Elisa Bertino, Norman Foo, et al. 2012. Reputation management in crowdsourcing systems. In Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom), 2012 8th International Conference on. IEEE, 664--671.
[2]
Benedikt Budig, Thomas C van Dijk, Fabian Feitsch, and Mauricio Giraldo Arteaga. 2016. Polygon consensus: smart crowdsourcing for extracting building footprints from historical maps. In Proceedings of the 24th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems. ACM, 66.
[3]
Yao-Yi Chiang, Stefan Leyk, and Craig A. Knoblock. 2014. A Survey of Digital Map Processing Techniques. Comput. Surveys 47, 1, Article 1 (2014), 1:1--1:44 pages.
[4]
Florian Daniel, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, and Mohammad Allahbakhsh. 2018. Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions. ACM Computing Surveys (CSUR) 51, 1 (2018), 7.
[5]
Muhammad Faisal, Ali Daud, and Abubakr Akram. 2017. Expert Ranking using Reputation and Answer Quality of Co-existing Users. International Arab Journal of Information Technology (IAJIT) 14, 1 (2017).
[6]
Mauricio Giraldo Arteaga. 2013. Historical Map Polygon and Feature Extractor. In Proceedings of the 1st ACM SIGSPATIAL International Workshop on MapInteraction (MapInteract '13). 66--71.
[7]
Vince Grolmusz. 2012. A note on the pagerank of undirected graphs. arXiv preprint arXiv:1205.1960 (2012).
[8]
Shih-Wen Huang and Wai-Tat Fu. 2013. Enhancing Reliability Using Peer Consistency Evaluation in Human Computation. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW '13). ACM, New York, NY, USA, 639--648.
[9]
Amy N Langville and Carl D Meyer. 2004. Deeper inside pagerank. Internet Mathematics 1, 3 (2004), 335--380.
[10]
Edith Law and Luis Von Ahn. 2009. Input-agreement: a new mechanism for collecting data using human computation games. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1197--1206.
[11]
Lawrence Page, Sergey Brin, Rajeev Motwani, Terry Winograd, et al. 1998. The pagerank citation ranking: Bringing order to the web. Technical Report. Computer Science Department, Stanford University.
[12]
Karthikeyan Rajasekharan, Aditya P Mathur, and See-Kiong Ng. 2013. Effective Crowdsourcing for Software Feature Ideation in Online Co-Creation Forums. In SEKE. 119--124.
[13]
Yan Sun and Yuhong Liu. 2012. Security of online reputation systems: The evolution of attacks and defenses. IEEE Signal Processing Magazine 29, 2 (2012), 87--97.
[14]
Bo Waggoner and Yiling Chen. 2014. Output agreement mechanisms and common knowledge. In Second AAAI Conference on Human Computation and Crowdsourcing. 220--226.
[15]
S. K. Warfield, K. H. Zou, and W. M. Wells. 2004. Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE Transactions on Medical Imaging 23, 7 (July 2004), 903--921.

Cited By

View all
  • (2019)Creating Structured, Linked Geographic Data from Historical Maps: Challenges and TrendsUsing Historical Maps in Scientific Studies10.1007/978-3-319-66908-3_3(37-63)Online publication date: 18-Nov-2019

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
GeoHumanities '18: Proceedings of the 2nd ACM SIGSPATIAL Workshop on Geospatial Humanities
November 2018
38 pages
ISBN:9781450360326
DOI:10.1145/3282933
Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 November 2018

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. crowdsourcing
  2. historical data
  3. quality control
  4. user reputation

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

Conference

SIGSPATIAL '18
Sponsor:

Acceptance Rates

GeoHumanities '18 Paper Acceptance Rate 4 of 5 submissions, 80%;
Overall Acceptance Rate 15 of 21 submissions, 71%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)3
  • Downloads (Last 6 weeks)0
Reflects downloads up to 01 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2019)Creating Structured, Linked Geographic Data from Historical Maps: Challenges and TrendsUsing Historical Maps in Scientific Studies10.1007/978-3-319-66908-3_3(37-63)Online publication date: 18-Nov-2019

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media