Skip to main content
SpringerLink
Log in

Relevance ranking in georeferenced video search

  • Original Research Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

The rapid adoption and deployment of ubiquitous video cameras has led to the collection of voluminous amounts of media data. However, indexing and searching of large video databases remain a very challenging task. Recently, some recorded video data are automatically annotated with meta-data collected from various sensors such as Global Positioning System (GPS) and compass devices. In our earlier work, we proposed the notion of a viewable scene model derived from the fusion of location and direction sensor information with a video stream. Such georeferenced media streams are useful in many applications and, very importantly, they can effectively be searched via their meta-data on a large scale. Consequently, search by geo-properties complements traditional content-based retrieval methods. The result of a georeferenced video query will in general consist of a number of video segments that satisfy the query conditions, but with more or less relevance. For example, a building of interest may appear in a video segment, but may only be visible in a corner. Therefore, an essential and integral part of a video query is the ranking of the result set according to the relevance of each clip. An effective result ranking is even more important for video than it is for text search, since the browsing of results can only be achieved by viewing each clip, which is very time consuming. In this study, we investigate and present three ranking algorithms that use spatial and temporal properties of georeferenced videos to effectively rank search results. To allow our techniques to scale to large video databases, we further introduce a histogram-based approach that allows fast online computations. An experimental evaluation demonstrates the utility of the proposed methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. FOVScenes can be collected at any time and timestamped. However, in our experiments, we collected FOVScenes with a fixed interval, at one second.

  2. http://www.maps.google.com/help/maps/streetview.

  3. There is no direct relation between the number of \({\mathbb{F}}{\rm s}\) and the length of a video because \({\mathbb{F}}\) can be sampled with various intervals.

References

  1. Zhang, H.J.: Multimedia content analysis and search: new perspectives and approaches. In: Proceedings of the ACM International Conference on Multimedia, pp. 1–2 (2009)

  2. Ay, S.A., Zimmermann, R., Kim, S.H.: Viewable scene modeling for geospatial video search. In: Proceedings of the ACM International Conference on Multimedia, pp 309–318 (2008)

  3. Rodden, K., Wood, K.R.: How do people manage their digital photographs? In: SIGCHI Conference on Human Factors in Computing Systems, pp. 409–416 (2003)

  4. Toyama, K., Logan, R., Roseway, A.: Geographic location tags on digital images. In: Proceedings of the ACM International Conference on Multimedia, pp. 156–166 (2003)

  5. Naaman, M., Song, Y.J., Paepcke, A., Garcia-Molina, H.: Automatic organization for digital photographs with geographic coordinates. In: 4th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 53–62 (2004)

  6. Pigeau, A., Gelgon, M.: Building and tracking hierarchical geographical & temporal partitions for image collection management on mobile devices. In: Proceedings of the ACM International Conference on Multimedia (2005)

  7. Flickr. http://www.flickr.com

  8. Geobloggers. http://www.geobloggers.com

  9. Woophy. http://www.woophy.com

  10. Epshtein, B., Ofek, E., Wexler, Y., Zhang, P.: Hierarchical photo organization using geo-relevance. In: 15th ACM International Symposium on Advances in Geographic Information Systems (GIS), pp 1–7 (2007)

  11. Simon, I., Seitz, S.M.: Scene segmentation using the wisdom of crowds. In: ECCV ’08: Proceedings of the 10th European Conference on Computer Vision, pp. 541–553 (2008)

  12. Kadobayashi, R., Tanaka, K.: 3D viewpoint-based photo search and information browsing. In: 28th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp 621–622 (2005)

  13. Gautam, S., Sarkis, G., Tjandranegara, E., Zelkowitz, E., Lu, Y.-H., Delp, E.J.: Multimedia for mobile environment: image enhanced navigation. In: Multimedia Content Analysis, Management, and Retrieval (2006)

  14. Torniai, C., Battle, S., Cayzer, S.: Sharing, discovering and browsing geotagged pictures on the web. Technical report, Digital Media Systems Laboratory, HP Laboratories Bristol (2007)

  15. Kennedy, L.S., Naaman, M.: Generating diverse and representative image search results for landmarks. In: WWW ’08: Proceeding of the 17th International Conference on the World Wide Web, pp 297–306 (2008)

  16. Hwang, T.-H., Choi, K.-H., Joo, I.-H., Lee, J.-H.: MPEG-7 metadata for video-based GIS applications. In: Geoscience and Remote Sensing Symposium, vol. 6, pp. 3641–3643 (2003)

  17. Kim, K.-H., Kim, S.-S., Lee, S.-H., Park, J.-H., Lee, J.-H.: The interactive geographic video. In: Geoscience and Remote Sensing Symposium, vol. 1, pp. 59–61 (2003)

  18. Liu, X., Corner, M., Shenoy, P.: SEVA: sensor-enhanced video annotation. In: Proceedings of the ACM International Conference on Multimedia, pp. 618–627 (2005)

  19. Arslan Ay, S., Kim, S.H., Zimmermann, R.: Relevance ranking in georeferenced video search. Technical Report 09-911, University of Southern California, Computer Science Department (2009)

  20. Smeaton, A.F., Over, P., Kraaij, W.: Evaluation campaigns and TRECVid. In: MIR ’06: Proceedings of the 8th ACM International Workshop on Multimedia Information Retrieval, pp. 321–330 (2006)

  21. Smeaton, A.F., Over, P., Kraaij, W.: High-level feature detection from video in TRECVid: a 5-year retrospective of achievements. In: Multimedia Content Analysis, Theory and Applications, pp. 151–174 (2009)

  22. Beard, K., Sharma, V.: Multidimensional ranking for data in digital spatial libraries. Int. J. Digit. Libr. 1(2), pp. 153–160 (1997)

    Article  Google Scholar 

  23. Larson, R.R., Frontiera, P.: Geographic information retrieval (GIR) ranking methods for digital libraries. In: JCDL ’04: Proceedings of the 4th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 415–415 (2004)

  24. Gobel, S., Klein, P.: Ranking mechanisms in meta-data information systems for geo-spatial data. In: Earth Observation and Geo-Spatial Data (EOGEO) Technical Workshop (2002)

  25. Ioannidis, Y.: The history of histograms (abridged). In: Proceedings of VLDB Conference (2003)

  26. Graham, C.H., Bartlett, N.R., Brown, J.L., Hsia, Y., Mueller, C.C., Riggs, L.A.: Vision and Visual Perception. Wiley, New York (1965)

    Google Scholar 

  27. Hecht, E.: Optics, 4th edn. Addison-Wesley, Reading (2001)

    Google Scholar 

  28. Camera Calibration Toolbox for Matlab. http://www.vision.caltech.edu/bouguetj/calib_doc/

  29. Lew, M.S., Sebe, N., Djeraba, C., Jain, R.: Content-based multimedia information retrieval: state of the art and challenges. ACM Trans Multimedia Comput Commun Appl TOMCCAP 2(1), 1–19 (2006)

    Article  Google Scholar 

  30. Datta, R., Joshi, D., Li, J., Wang, J.Z.: Image retrieval: ideas, influences, and trends of the new age. ACM Comput. Surv. 40(2), pp. 1–60 (2008)

    Article  Google Scholar 

  31. Snoek, C.G.M., Worring, M.: Concept-based video retrieval. Found Trends Info Retr 2(4), 215–322 (2009)

    Article  Google Scholar 

  32. Yanagawa, A., Chang, S.-F., Kennedy, L., Hsu, W.: Distributed policy management and comprehension with classified advertisements. Technical Report 222-2006-8, Columbia University ADVENT (2007)

  33. Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Trans Info Syst 20(4), 422–446 (2002)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Southern California, Los Angeles, CA, 90089, USA

    Sakire Arslan Ay

  2. School of Computing, National University of Singapore, Singapore, 117417, Republic of Singapore

    Roger Zimmermann

  3. Department of Computer Science and Information Technology, University of the District of Columbia, Washington, DC, 20008, USA

    Seon Ho Kim

Authors
  1. Sakire Arslan Ay
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Roger Zimmermann
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Seon Ho Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sakire Arslan Ay.

Additional information

Communicated by Thomas Haenselmann.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Arslan Ay, S., Zimmermann, R. & Kim, S.H. Relevance ranking in georeferenced video search. Multimedia Systems 16, 105–125 (2010). https://doi.org/10.1007/s00530-009-0177-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-009-0177-x

Keywords

Search

Navigation