Abstract
This paper proposes a design of a system for creating image similarity datasets which are necessary for testing the quality of supervised ranking algorithms. In particular, the main goal is to facilitate the creation of similar images rankings for given a imaginary dataset. The system was designed in a manner that involves user feedback in the process of creating the rankings. In each iteration of ranking construction, the query image and twelve candidates are presented to the user, who is intended to select the most similar one. Moreover, in order to accelerate the method convergence the approach based on simulated annealing is adapted. It initially chooses the images randomly from a dataset and in the later stages the images with rank rate above zero are chosen with certain probability.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Collins, M., Duffy, N.: New ranking algorithms for parsing and tagging: Kernels over discrete structures, and the voted perceptron. In: ACL, pp. 263–270 (2002)
Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., Popović, Z., Foldit Players: Predicting protein structures with a multiplayer online game. Nature 466(7307), 756–760 (2010)
Elo, A.E.: The Rating of Chess Players, Past and Present. Ishi Press (2008)
Geman, S., Geman, D.: Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence 6(6), 721–741 (1984)
Górecki, P., Sopyła, K., Drozda, P.: Ranking by K-means voting algorithm for similar image retrieval. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2012, Part I. LNCS, vol. 7267, pp. 509–517. Springer, Heidelberg (2012)
Herbrich, R., Minka, T., Graepel, T.: TrueSkill(TM): A Bayesian Skill Rating System, vol. 20 (2007)
Joachims, T.: Optimizing search engines using clickthrough data. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2002, pp. 133–142. ACM, New York (2002)
Kawrykow, A., Roumanis, G., Kam, A., Kwak, D., Leung, C., Wu, C., Zarour, E., Sarmenta, L., Blanchette, M., Waldispühl, J., Phylo players: Phylo: A citizen science approach for improving multiple sequence alignment. PLoS ONE 7(3), e31362 (2012)
Khatib, F., Cooper, S., Tyka, M.D., Xu, K., Makedon, I., Popović, Z., Baker, D., Foldit Players: Algorithm discovery by protein folding game players. Proceedings of the National Academy of Sciences 108(47), 18949–18953 (2011)
Korytkowski, M., Nowicki, R., Rutkowski, L., Scherer, R.: Adaboost ensemble of DCOG rough–neuro–fuzzy systems. In: Jędrzejowicz, P., Nguyen, N.T., Hoang, K. (eds.) ICCCI 2011, Part I. LNCS, vol. 6922, pp. 62–71. Springer, Heidelberg (2011)
Page, L., Brin, S., Motwani, R., Winograd, T.: The pagerank citation ranking: Bringing order to the web. Technical Report 1999-66, Stanford InfoLab. Previous number = SIDL-WP-1999-0120 (November 1999)
Paolacci, G., Chandler, J., Ipeirotis, P.G.: Running experiments on amazon mechanical turk. Judgment and Decision Making 5(5), 411–419 (2010)
Ross, J., Irani, L., Silberman, M.S., Zaldivar, A., Tomlinson, B.: Who are the crowdworkers?: shifting demographics in mechanical turk. In: Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA 2010, pp. 2863–2872. ACM, New York (2010)
Shen, L., Joshi, A.K.: Ranking and reranking with perceptron. Machine Learning, 73–96 (2005)
Sopyła, K., Drozda, P., Górecki, P.: Picrank - online crowdsource system for image ranking creation (2012)
Sopyła, K., Drozda, P., Górecki, P.: SVM with CUDA accelerated kernels for big sparse problems. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2012, Part I. LNCS, vol. 7267, pp. 439–447. Springer, Heidelberg (2012)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Drozda, P., Sopyła, K., Górecki, P. (2013). Online Crowdsource System Supporting Ground Truth Datasets Creation. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2013. Lecture Notes in Computer Science(), vol 7894. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-38658-9_48
Download citation
DOI: https://doi.org/10.1007/978-3-642-38658-9_48
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-38657-2
Online ISBN: 978-3-642-38658-9
eBook Packages: Computer ScienceComputer Science (R0)