ABSTRACT
Social science researchers spend significant time annotating behavioral events in video data in order to quantitatively assess interactions [2]. These behavioral events may be instantaneous changes, continuous actions that span unbounded periods of time, or behaviors that would be best described by severity or other scalar ratings. The complexity of these judgments, coupled with the time and effort required to meticulously assess video, results in a training and evaluation process that can take days or weeks. Computational analysis of video data is still limited due to the challenges introduced by objective interpretation and varied contexts. Glance [4] introduced a means of leveraging human intelligence by recruiting crowds of paid online workers to accurately analyze hours of video data in a matter of minutes. This approach has been shown to expedite work in human-centered fields, as well as generate training data for automated recognition systems. In this paper, we describe an interactive demonstration of an improved, more expressive version of Glance that expands the initial set of supported annotation formats (e.g. time range, classification, etc.) from one to nine. Worker interfaces for each of these options are dynamically generated, along with tutorials, based on the analyst's question. These new features allow analysts to acquire more specific information about events in video datasets.
- Allen, J. F. Maintaining knowledge about temporal intervals. Communications of the ACM 26, 11 (1983), 832-843. Google ScholarDigital Library
- Heyman, R. E., Lorber, M. F., Eddy, J. M., West, T., Reis, E. H. T., and Judd, C. M. Handbook of Research Methods in Social and Personality Psychology. Cambridge University Press, 2014, ch. Behavioral observation and coding.Google Scholar
- Laput, G., Lasecki, W. S., Wiese, J., Xiao, R., Bigham, J. P., and Harrison, C. Zensors: Adaptive, rapidly deployable, human-intelligent sensor feeds. In CHI (2015), 1935-1944. Google ScholarDigital Library
- Lasecki, W. S., Gordon, M., Koutra, D., Jung, M. F., Dow, S. P., and Bigham, J. P. Glance: Rapidly coding behavioral video with the crowd. In UIST (2014), 551-562. Google ScholarDigital Library
- Lasecki, W. S., Gordon, M., Leung, W., Lim, E., Bigham, J. P., and Dow, S. P. Exploring privacy and accuracy trade-offs in crowdsourced behavioral video coding. In CHI (2015), 1945-1954. Google ScholarDigital Library
Recommendations
Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments
The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that ...
Make Hay While the Crowd Shines: Towards Efficient Crowdsourcing on the Web
WWW '15 Companion: Proceedings of the 24th International Conference on World Wide WebWithin the scope of this PhD proposal, we set out to investigate two pivotal aspects that influence the effectiveness of crowdsourcing: (i) microtask design, and (ii) workers behavior. Leveraging the dynamics of tasks that are crowdsourced on the one ...
Efficient Crowd Exploration of Large Networks: The Case of Causal Attribution
Accurately and efficiently crowdsourcing complex, open-ended tasks can be difficult, as crowd participants tend to favor short, repetitive "microtasks". We study the crowdsourcing of large networks where the crowd provides the network topology via ...
Comments