Skip to main content
Springer Nature Link
Log in

Rethinking the Test Collection Methodology for Personal Self-tracking Data

  • Conference paper
  • First Online:
MultiMedia Modeling (MMM 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11962))

Included in the following conference series:

Abstract

While vast volumes of personal data are being gathered daily by individuals, the MMM community has not really been tackling the challenge of developing novel retrieval algorithms for this data, due to the challenges of getting access to the data in the first place. While initial efforts have taken place on a small scale, it is our conjecture that a new evaluation paradigm is required in order to make progress in analysing, modeling and retrieving from personal data archives. In this position paper, we propose a new model of Evaluation-as-a-Service that re-imagines the test collection methodology for personal multimedia data in order to address the many challenges of releasing test collections of personal multimedia data.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    http://getnarrative.com.

  2. 2.

    https://quantifiedself.com.

  3. 3.

    https://www.nippon.com/en/news/yjj2019070800965/aeon-unit-sumitomo-mitsui-trust-bank-certified-as-1st-ever-info-banks.html.

  4. 4.

    https://every-sense.com/eng/.

References

  1. Aghaei, M., Dimiccoli, M., Ferrer, C.C., Radeva, P.: Towards social pattern characterization in egocentric photo-streams. Comput. Vis. Image Underst. 171, 104–117 (2018)

    Article  Google Scholar 

  2. Alsuhaibani, A., Cox, A., Hopfgartner, F.: Investigating the role of social media during the transition of international students to the UK. In: iConference 2019 Poster Proceedings, IDEALS (2019)

    Google Scholar 

  3. Awad, G., et al.: TRECVID 2018: benchmarking video activity detection, video captioning and matching, video storytelling linking and video search. In: TRECVid 2018: Proceedings of the TREC Video Retrieval Evaluation Conference. NIST, Gaithersburg (2018)

    Google Scholar 

  4. Bailer, W.: Face swapping for solving collateral privacy issues in multimedia analytics. In: MMM 2019, pp. 169–177 (2019)

    Google Scholar 

  5. Cavoukian, A.: Privacy by design: the 7 foundational principles. Implementation and mapping of fair information practices. Information and Privacy Commissioner of Ontario, Canada (2010)

    Google Scholar 

  6. Chang, R.M., Kauffman, R.J., Kwon, Y.: Understanding the paradigm shift to computational social science in the presence of big data. Decis. Support Syst. 63, 67–80 (2014)

    Article  Google Scholar 

  7. Dang-Nguyen, D.T., Piras, L., Riegler, M., Zhou, L., Lux, M., Gurrin, C.: Overview of ImageCLEFlifelog 2018: daily living understanding and lifelog moment retrieval. In: CLEF2018 Working Notes, Avignon, France (2018)

    Google Scholar 

  8. Dix, A., Ellis, G.: The Alan Walks Wales Dataset: Quantified Self and Open Data, pp. 56–66. Open data as open educational resources: case studies of emerging practice, The Open University (2015)

    Google Scholar 

  9. Dodge, M., Kitchin, R.: ‘Outlines of a world coming into existence’: pervasive computing and the ethics of forgetting. Environ. Plann. B Plann. Design 34(3), 431–445 (2007). https://doi.org/10.1068/b32041t

    Article  Google Scholar 

  10. Gemmell, J., Bell, G., Lueder, R., Drucker, S., Wong, C.: Mylifebits: fulfilling the memex vision. In: Proceedings of the Tenth ACM International Conference on Multimedia, MULTIMEDIA 2002, pp. 235–238. ACM, New York (2002)

    Google Scholar 

  11. Gollub, T., Stein, B., Burrows, S.: Ousting ivory tower research: towards a web framework for providing experiments as a service. In: The 35th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2012, Portland, OR, USA, 12–16 August 2012, pp. 1125–1126 (2012)

    Google Scholar 

  12. Gupta, R., Gurrin, C.: Approaches for event segmentation of visual lifelog data. In: Schoeffmann, K., et al. (eds.) MMM 2018. LNCS, vol. 10704, pp. 581–593. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-73603-7_47

    Chapter  Google Scholar 

  13. Gurrin, C., Albatal, R., Joho, H., Ishii, K.: A privacy by design approach to lifelogging. In: Digital Enlightenment Yearbook 2014, pp. 49–73. IOS Press (2014)

    Google Scholar 

  14. Gurrin, C., Joho, H., Hopfgartner, F., Zhou, L., Albatal, R.: Overview of NTCIR-12 lifelog task. In: Kando, N., Kishida, K., Kato, M.P., Yamamoto, S. (eds.) Proceedings of the 12th NTCIR Conference on Evaluation of Information Access Technologies, pp. 354–360 (2016)

    Google Scholar 

  15. Gurrin, C., et al.: Overview of NTCIR-13 lifelog-2 task. In: Proceedings of the 13th NTCIR Conference on Evaluation of Information Access Technologies (2017)

    Google Scholar 

  16. Gurrin, C., et al.: Overview of the NTCIR-14 lifelog-3 task. In: Online Proceedings of the Fourteenth NTCIR Conference (NTCIR-14), NII (2019)

    Google Scholar 

  17. Harman, D.: Information retrieval: the early years. Found. Trends Inf. Retrieval 13(5), 425–577 (2019)

    Article  Google Scholar 

  18. Hayashi, T., Nishida, M., Kitaoka, N., Toda, T., Takeda, K.: Daily activity recognition with large-scaled real-life recording datasets based on deep neural network using multi-modal signals. IEICE Trans. Fundam. Electron. Commun. Comput. Sci. E101.A, 199–210 (2018)

    Article  Google Scholar 

  19. Herruzo, P., Portell, L., Soto, A., Remeseiro, B.: Analyzing first-person stories based on socializing, eating and sedentary patterns. In: Battiato, S., Farinella, G.M., Leo, M., Gallo, G. (eds.) ICIAP 2017. LNCS, vol. 10590, pp. 109–119. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-70742-6_10

    Chapter  Google Scholar 

  20. Hodges, S., et al.: SenseCam: a retrospective memory aid. In: Dourish, P., Friday, A. (eds.) UbiComp 2006. LNCS, vol. 4206, pp. 177–193. Springer, Heidelberg (2006). https://doi.org/10.1007/11853565_11

    Chapter  Google Scholar 

  21. Hopfgartner, F., Davidson, J.: Digital preservation and curation of self-tracking data: a position paper. In: Proceedings of the 1st Workshop on Knowledge Discovery and User Modelling for Smart Cities Co-located with 24th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, UMCit at KDD 2018, London, United Kingdom, 20 August 2018, pp. 1–5 (2018)

    Google Scholar 

  22. Hopfgartner, F.: Evaluation-as-a-service for the computational sciences: overview and outlook. J. Data Inf. Quality 10(4), 15:1–15:32 (2018)

    Google Scholar 

  23. Ionescu, B., et al.: ImageCLEF 2019: multimedia retrieval in lifelogging, medical, nature, and security applications. In: Azzopardi, L., Stein, B., Fuhr, N., Mayr, P., Hauff, C., Hiemstra, D. (eds.) ECIR 2019. LNCS, vol. 11438, pp. 301–308. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-15719-7_40

    Chapter  Google Scholar 

  24. Janin, A., et al.: The ICSI meeting corpus. In: ICASSP 2003, April 2003

    Google Scholar 

  25. Korotitisch, W.J., Nelson-Gray, R.O.: An overview of self-monitoring research in assessment and treatment. Psychol. Assess. 11, 415–425 (1999)

    Article  Google Scholar 

  26. Larson, M. (eds.): Working Notes Proceedings of the MediaEval 2018 Workshop, Sophia Antipolis, France, 29–31 October 2018, CEUR Workshop Proceedings, vol. 2283 (2018). CEUR-WS.org

  27. Li, I., Dey, A.K., Forlizzi, J.: A stage-based model of personal informatics systems. In: Proceedings of the 28th International Conference on Human Factors in Computing Systems, CHI 2010, Atlanta, Georgia, USA, 10–15 April 2010, pp. 557–566 (2010)

    Google Scholar 

  28. Lidon, A., Bolaños, M., Dimiccoli, M., Radeva, P., Garolera, M., Giró, X.: Semantic summarization of egocentric photo stream events. In: LTA@MM (2017)

    Google Scholar 

  29. Lorenz, F., et al.: Countering contextual bias in TV watching behavior: introducing social trend as external contextual factor in TV recommenders. In: Proceedings of the 2017 ACM International Conference on Interactive Experiences for TV and Online Video, Hilversum, The Netherlands, 14–16 June 2017, pp. 21–30 (2017)

    Google Scholar 

  30. Miyanishi, T., Hirayama, J., Kanemura, A., Kawanabe, M.: Answering mixed type questions about daily living episodes. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI 2018, pp. 4265–4271 (2018)

    Google Scholar 

  31. Garcia del Molino, A., Lim, J.H., Tan, A.H.: Predicting visual context for unsupervised event segmentation in continuous photo-streams. In: ACM Multimedia Conference (ACMMM 2018), MM 2018, pp. 10–17. ACM, New York (2018)

    Google Scholar 

  32. Piasek, P.: Case Studies in Therapeutic SenseCam Use Aimed at Identity Maintenance in Early Stage Dementia. Ph.D. thesis, Dublin City University (2015)

    Google Scholar 

  33. Sellen, A.J., Whittaker, S.: Beyond total capture: a constructive critique of lifelogging. Commun. ACM 53(5), 70–77 (2010)

    Article  Google Scholar 

  34. Servia-Rodriguez, S., Wang, L., Zhao, J., Mortier, R., Haddadi, H.: Privacy-preserving personal model training. In: ACM/IEEE International Conference on Internet of Things Design and Implementation, pp. 153–164 (2018)

    Google Scholar 

  35. Smeaton, A.F., et al.: Semantic indexing of wearable camera images: Kids’Cam concepts. In: Proceedings of the 2016 ACM Workshop on Vision and Language Integration Meets Multimedia Fusion (2016)

    Google Scholar 

  36. Stevinson, C., Wiltshire, G., Hickson, M.: Facilitating participation in health-enhancing physical activity: a qualitative study of parkrun. Int. J. Behav. Med. 22(2), 170–177 (2015)

    Article  Google Scholar 

  37. Su, Y.C., Grauman, K.: Detecting engagement in egocentric video. In: Proceedings of the European Conference on Computer Vision (ECCV) (2016)

    Chapter  Google Scholar 

  38. Voorhees, E.M., Harman, D.K.: TREC: Experiment and Evaluation in Information Retrieval. The MIT Press, Cambridge (Digital Libraries and Electronic Publishing) (2005)

    Google Scholar 

  39. Walsh, D., Clough, P., Hall, M.M., Hopfgartner, F., Foster, J., Kontonatsios, G.: Analysis of transaction logs from National Museums Liverpool. In: Doucet, A., Isaac, A., Golub, K., Aalberg, T., Jatowt, A. (eds.) TPDL 2019. LNCS, vol. 11799, pp. 84–98. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30760-8_7

    Chapter  Google Scholar 

  40. Xu, J., Mukherjee, L., Li, Y., Warner, J., Rehg, J.M., Singh, V.: Gaze-enabled egocentric video summarization via constrained submodular maximization. In: Proceedings of CVPR (2015)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Sheffield, Sheffield, UK

    Frank Hopfgartner

  2. Dublin City University, Dublin, Ireland

    Cathal Gurrin

  3. University of Tsukuba, Tsukuba, Japan

    Hideo Joho

Authors
  1. Frank Hopfgartner
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cathal Gurrin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hideo Joho
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Frank Hopfgartner .

Editor information

Editors and Affiliations

  1. Korea Advanced Institute of Science and Technology, Daejeon, Korea (Republic of)

    Yong Man Ro

  2. National Chiao Tung University, Hsinchu, Taiwan

    Wen-Huang Cheng

  3. Korea Advanced Institute of Science and Technology, Daejeon, Korea (Republic of)

    Junmo Kim

  4. National Cheng Kung University, Tainan City, Taiwan

    Wei-Ta Chu

  5. Tsinghua University, Beijing, China

    Peng Cui

  6. Korea Advanced Institute of Science and Technology, Daejeon, Korea (Republic of)

    Jung-Woo Choi

  7. National Tsing Hua University, Hsinchu, Taiwan

    Min-Chun Hu

  8. Ghent University, Ghent, Belgium

    Wesley De Neve

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hopfgartner, F., Gurrin, C., Joho, H. (2020). Rethinking the Test Collection Methodology for Personal Self-tracking Data. In: Ro, Y., et al. MultiMedia Modeling. MMM 2020. Lecture Notes in Computer Science(), vol 11962. Springer, Cham. https://doi.org/10.1007/978-3-030-37734-2_38

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-37734-2_38

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-37733-5

  • Online ISBN: 978-3-030-37734-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics