The power of crowds, leveraging a large number of human contributors and the capabilities of human computation, has enormous potential to address key challenges in the area of multimedia research. This power is, however, of difficult exploitation: challenges arise from the fact that a community of users or workers is a complex and dynamic system highly sensitive to changes in the form and the parameterization of their activities. Since 2012, the International ACM Workshop on Crowdsourcing for Multimedia, CrowdMM, has been the venue for collecting new insights on the effective deployment of crowdsourcing towards boosting Multimedia research.
In its third edition, CrowdMM14 especially focuses on contributions that propose solutions for the key challenges that face widespread adoption of crowdsourcing paradigms in the multimedia research community. These include: identification of optimal crowd members (e.g., user expertise, worker reliability), providing effective explanations (i.e., good task design), controlling noise and quality in the results, designing incentive structures that do not breed cheating, adversarial environments, gathering necessary background information about crowd members without violating privacy, controlling descriptions of task.
The call for papers attracted 26 international submissions (62% increase with respect to the 2013 edition), three of which were short paper submissions. Of these, 8 were accepted as oral presentations and 5 as poster presentations. All papers received at least three double blind reviews, and 3.4 reviews on average.
For a keynote talk, Nhatvi Nguyen (CEO of Microworkers) talks about Crowdsourcing Challenges from Platform Provider's Point of View. In addition to that CrowdMM features this year a crowd-sourced keynote, during which all CrowdMM14 authors give their view on the future and the Challenges that Crowdsourcing has still ahead.
Proceeding Downloads
Microworkers Crowdsourcing Approach, Challenges and Solutions
Founded in May 2009, Microworkers.com is an international Crowdsourcing platform focusing on Microtasks. At present, more than 600,000 users from over 190 countries have already registered to our platform. This extensively diverse workforce is the key ...
A Protocol for Cross-Validating Large Crowdsourced Data: The Case of the LIRIS-ACCEDE Affective Video Dataset
Recently, we released a large affective video dataset, namely LIRIS-ACCEDE, which was annotated through crowdsourcing along both induced valence and arousal axes using pairwise comparisons. In this paper, we design an annotation protocol which enables ...
Modeling Image Appeal Based on Crowd Preferences for Automated Person-Centric Collage Creation
This paper attempts to model IA in personal photo collections through a user-centric perspective. To understand how users deemed an image as being more/less appealing, an extensive crowdsourcing experiment was conducted with 350 workers and five ...
A Multi-task Learning Framework for Time-continuous Emotion Estimation from Crowd Annotations
- Mojtaba Khomami Abadi,
- Azad Abad,
- Ramanathan Subramanian,
- Negar Rostamzadeh,
- Elisa Ricci,
- Jagannadan Varadarajan,
- Nicu Sebe
We propose Multi-task learning (MTL) for time-continuous or dynamic emotion (valence and arousal) estimation in movie scenes. Since compiling annotated training data for dynamic emotion prediction is tedious, we employ crowdsourcing for the same. Even ...
Crowdsourcing for Rating Image Aesthetic Appeal: Better a Paid or a Volunteer Crowd?
Crowdsourcing has the potential to become a preferred tool to study image aesthetic appeal preferences of users. Nevertheless, some reliability issues still exist, partially due to the sometimes doubtful commitment of paid workers to perform the rating ...
Development and Validation of Extrinsic Motivation Scale for Crowdsourcing Micro-task Platforms
In this paper, we introduce a scale for measuring the extrinsic motivation of crowd workers. The new questionnaire is strongly based on the Work Extrinsic Intrinsic Motivation Scale (WEIMS) [17] and theoretically follows the Self-Determination Theory (...
Is That a Jaguar?: Segmenting Ancient Maya Glyphs via Crowdsourcing
Crowdsourcing is popular in multimedia research to obtain image annotation and segmentation data at scale. In the context of analysis of cultural heritage materials, we propose a novel crowdsourced task, namely the segmentation of ancient Maya ...
Making use of Semantic Concept Detection for Modelling Human Preferences in Visual Summarization
In this paper we investigate whether and how the human choice of images for summarizing a visual collection is influenced by the semantic concepts depicted in them. More specifically, by analysing a large collection of human-created visual summaries ...
A Crowdsourcing Procedure for the Discovery of Non-Obvious Attributes of Social Images
Research on mid-level image representations has conventionally concentrated relatively obvious attributes and overlooked non-obvious attributes, i.e., characteristics that are not readily observable when images are viewed independently of their context ...
A Crowdsourced Data Set of Edited Images Online
We present a crowdsourcing approach to tackle the challenge of collecting hard-to-find data. Our immediate need for the data arises because we are studying edited images in context online, and the way that this use impacts users' perceptions. Study of ...
Click'n'Cut: Crowdsourced Interactive Segmentation with Object Candidates
This paper introduces Click'n'Cut, a novel web tool for interactive object segmentation designed for crowdsourcing tasks. Click'n'Cut combines bounding boxes and clicks generated by workers to obtain accurate object segmentations. These segmentations ...
Users Tagging Visual Moments: Timed Tags in Social Video
A timed tag is a tag that a user has assigned to a specific time point in a video. Although timed tags are supported by an increasing number of social video platforms on the Internet, multimedia research remains focused on conventional tags, here called ...
Crowd-based Semantic Event Detection and Video Annotation for Sports Videos
Recent developments in sport analytics have heightened the interest in collecting data on the behavior of individuals and of the entire team in sports events. Rather than using dedicated sensors for recording the data, the detection of semantic events ...
Getting by with a Little Help from the Crowd: Practical Approaches to Social Image Labeling
- Babak Loni,
- Jonathon Hare,
- Mihai Georgescu,
- Michael Riegler,
- Xiaofei Zhu,
- Mohamed Morchid,
- Richard Dufour,
- Martha Larson
Validating user tags helps to refine them, making them more useful for finding images. In the case of interpretation-sensitive tags, however, automatic (i.e., pixel-based) approaches cannot be expected to deliver optimal results. Instead, human input is ...
Index Terms
- Proceedings of the 2014 International ACM Workshop on Crowdsourcing for Multimedia
Recommendations
Acceptance Rates
Year | Submitted | Accepted | Rate |
---|---|---|---|
CrowdMM '14 | 26 | 8 | 31% |
CrowdMM '13 | 16 | 8 | 50% |
Overall | 42 | 16 | 38% |