Skip to main content
SpringerLink
Log in

Extreme event detection and assimilation from multimedia sources

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

A new event-based multimedia processing framework for detection, retrieval, and cross-media content assimilation of geo-spatiotemporal phenomena is described. Multimedia information relevant to geo-spatiotemporal events are available from sources such as remote satellites, in-situ sensors as image streams, and other outlets such as news articles, weather bulletins as text documents and in various formats, each with widely varying properties. We pose an event-based framework to automatically detect geo-spatiotemporal phenomena from raw untagged remote-sensing satellite image streams, extract attributes for such events, match the spatiotemporal properties of the detected phenomenon with events in a database to automatically derive a media-independent event description, and subsequently use the media-independent event descriptions to assimilate relevant information of the same event across other media sources from the web using a mashup. A virtual globe interface enables simultaneous visualization of the assimilated spatiotemporal information annotated with Internet sources. This framework is demonstrated for the automatic detection of tropical cyclones from satellite imagery followed by the retrieval and assimilation of related information from government-run weather sites and commercial news portals.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. http://weather.unisys.com/hurricane/index.php

References

  1. Bobick AF, Ivanov YA (1998) Action recognition using probabilistic parsing. In: Conference on computer vision and pattern recognition (CVPR), Santa Barbara, CA, USA. IEEE Comput Society, pp 196–202

  2. Callahan PS, Lungu T (2006) QuikSCAT Science Data Product User’s Manual (v3.0). Jet Propulsion Laboratory, Pasadena, CA

  3. Castaño R, Mazzoni D, Tang N, Greeley R, Doggett T, Cichy B, Chien SA, Davies A (August 20–23, 2006) Onboard classifiers for science event detection on a remote sensing spacecraft. In: Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, USA, pp 845–851

  4. DeMenthon D, Doermann D (2003) Video retrieval using spatio-temporal descriptors. Paper presented at the Proceedings of the eleventh ACM international conference on Multimedia, Berkeley, CA, USA

  5. Dvorak VF (1984) Tropical cyclone intensity analysis using satellite data. vol 11. NOAA Tech. Rep. NESDIS

  6. Fowler C, Qasemizadeh B (2009) Towards a Common Event Model for an Integrated Sensor Information System. Paper presented at the 1st International Workshop on the Semantic Sensor Web (SemSensWeb)

  7. Francois ARJ, Nevatia R, Hobbs J, Bolles RC, Smith JR (2005) VERL: an ontology framework for representing and annotating video events. IEEE Multimed 12(4):76–86

    Article  Google Scholar 

  8. Franklin J (2011) National Hurricane Center Forecast Verification. National Hurricane Center. http://www.nhc.noaa.gov/verification/index.shtml

  9. Ho S-S, Talukder A (2008) Automated cyclone discovery and tracking using knowledge sharing in multiple heterogeneous satellite data. In: International Conference on Knowledge Discovery and Data Mining, Las Vegas, Nevada, USA, pp 928–936

  10. Ho S-s, Talukder A (2009) Utilizing spatio-temporal text information for cyclone eye annotation in satellite data, IJCAI , Pasadena, CA, 13 July, 2009. Paper presented at the Workshop on Cross-Media Information Access and Mining, International Joint Conferences on Artificial Intelligence, Pasadena, CA, USA, July 13, 2009

  11. Hongeng S, Bremond F, Nevatia R (June 13–15 2000) Representation and Optimal Recognition of Human Activities. In: Conference on Computer Vision and Pattern Recognition (CVPR), Hilton Head, SC, USA. pp 1818–1825

  12. Hwang T-H, Choi K-H, Joo I-H, Lee J-H (21–25 July 2003) MPEG-7 metadata for video-based GIS applications. In: Geoscience and Remote Sensing Symposium, 2003. IGARSS ‘03. Proceedings. 2003 IEEE International, pp 3641–3643 vol.3646

  13. Khalid S, Naftel A (2005) Evaluation of Matching Metrics for Trajectory-based Indexing and Retrieval of Video Clips. Paper presented at the Seventh IEEE Workshop on Applications of Computer Vision (WACV/MOTION)

  14. Kim S-S, Park J-H (2004) Linking geographic hypermedia using the remotely sensed data. In: Multimedia and Expo, 2004. ICME ‘04. 2004 IEEE International Conference on, 30–30 June 2004. pp 811–814 Vol.812

  15. Koprinska I, Carrato S (2001) Temporal video segmentation: a survey. Signal Process Image Comm 16(5):477–500

    Article  Google Scholar 

  16. Li Z, Tan Y-P (2005) Event detection using multimodal feature analysis. In: Circuits and Systems, 2005. ISCAS 2005. IEEE International Symposium on, 23–26 May 2005, pp 3845–3848 Vol. 3844

  17. Lippman A (1980) Movie-maps: an application of the optical videodisc to computer graphics. SIGGRAPH Comput Graph 14(3):32–42

    Article  Google Scholar 

  18. Mazzoni D, Garay MJ, Davies R, Nelson D (2007) An operational MISR pixel classifier using support vector machines. Remote Sens Environ 107:149–158

    Article  Google Scholar 

  19. Medioni G, Cohen I, Bremond F, Hongeng S, Nevatia R (2001) Event detection and analysis from video streams. IEEE Trans Pattern Anal Mach Intell 23(8):873–889

    Article  Google Scholar 

  20. Moncrieff S, Dorai C, Venkatesch S (2001) Detecting indexical signs in film audio for scene interpretation. In: Multimedia and Expo, 2001. ICME 2001. IEEE International Conference on, 22–25 Aug. 2001, pp 989–992

  21. Navarrete T, Blat J (2006) A semantic approach for the indexing and retrieval of geo-referenced video. Paper presented at the International Semantic Web Conference (ISWC)

  22. Nguyen NT, Phung DQ, Venkatesh S, Bui H (20–25 June 2005) Learning and detecting activities from movement trajectories using the hierarchical hidden Markov model. In: Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, pp 955–960 vol. 952

  23. Nobre EMN, Camara AS (2002) SPATIAL VIDEO: exploring space using multiple digital videos. Paper presented at the Proceedings of the sixth Eurographics workshop on Multimedia 2001, Manchester, UK

  24. Oliver N, Horvitz E, Garg A (October 2002) Layered representations for recognizing office activity. In: Fourth IEEE International Conference on Multimodal Interaction, Pittsburgh, PA, USA, pp 3–8

  25. Panangadan A, Ho S-S, Talukder A (November 4–6 2009) Cyclone tracking using multiple satellite image sources. In: 17th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, Seattle

  26. Pasch RJ, Stewart SR, Brown DP (2003) Comments on “early detection of tropical cyclones using seawinds-derived vorticity”. Bull Am Meteorol Soc 85(10):1415–1416

    Article  Google Scholar 

  27. Peuquet DJ, Duan N (1995) An event-based spatiotemporal data model (ESTDM) for temporal analysis of geographical data. Int J Geogr Inform Sci 9(1):7–24

    Article  Google Scholar 

  28. Rath TM, Manmatha R (2003) Word image matching using dynamic time warping. In: Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on, 18–20 June 2003, pp II-521-II-527 vol.522

  29. Rhome JR (2009) Technical Summary of the National Hurricane Center Track and Intensity Models. National Hurricane Center. http://www.nhc.noaa.gov/modelsummary.shtml

  30. Rui Y, Gupta A, Acero A (2000) Automatically extracting highlights for TV Baseball programs. Paper presented at the Proceedings of the eighth ACM international conference on Multimedia, Marina del Rey, California, United States

  31. Sakaki T, Okazaki M, Matsuo Y (2010) Earthquake shakes Twitter users: real-time event detection by social sensors. Proceedings of the 19th international conference on World wide web. ACM, Raleigh, North Carolina, USA

  32. Stefanidis A, Eickhorst K, Agouris P, Partsinevelos P (2003) Modeling and comparing change using spatiotemporal helixes. In: 11th ACM international symposium on Advances in geographic information systems (GIS ‘03). ACM, New York, NY, USA, pp 86–93

  33. Wei H, Xiong J, Yan F (2006) Management of Remote Sensing Data and Multimedia Data for Disaster Monitoring. In: Geoscience and Remote Sensing Symposium, 2006. IGARSS 2006. IEEE International Conference on, July 31 2006-Aug. 4 2006. pp 835–837

  34. Westerman U, Jain R (2007) Toward a common event model for multimedia applications. IEEE Multimed 4(1):19–29

    Article  Google Scholar 

  35. Wimmers A, Velden CS (2004) Satellite-based center-fixing of tropical cyclones: new automated approaches. In: 26th Conference on Hurricanes and Tropical Meteorology

  36. Wulder M, White J, McDonald S (2005) Truth in the air: geolocated video validates satellite land cover. GEO World. November

  37. Zeinalipour-Yazti D, Lin S, Gunopulos D (2006) Distributed Spatio-Temporal Similarity Search. Paper presented at the 15th ACM Conference on Information and Knowledge Management, Arlington, VA, USA, November 6–11

  38. Zhai Y, Tian Y-L, Hampapur A (2006) Composite spatio-temporal event detection in multi-camera surveillance networks. In: Semantic Learning Applications in Multimedia

  39. Zotkin D, Duraiswami R, Davis LS (2001) Multimodal 3-D tracking and event detection via the particle filter. In: Detection and Recognition of Events in Video, 2001. Proceedings. IEEE Workshop on, 2001, pp 20–27

Download references

Acknowledgments

The research described in this paper was carried out at the Jet Propulsion Laboratory, California Institute of Technology with funding from the National Aeronautics and Space Administration (NASA) Applied Information Systems Research (AISR) Program. The authors acknowledge the contributions of Eric Rigor, Andrew Bingham, and Shen-shyang Ho.

Author information

Authors and Affiliations

  1. National Institute of Standards and Technology, 100 Bureau Drive, Mail Stop 8940, Gaithersburg, MD, 20899-8940, USA

    Ashit Talukder

  2. Jet Propulsion Laboratory, 4800 Oak Grove Drive, MS 300-123, Pasadena, CA, 91109, USA

    Anand Panangadan

Authors
  1. Ashit Talukder
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anand Panangadan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ashit Talukder.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Talukder, A., Panangadan, A. Extreme event detection and assimilation from multimedia sources. Multimed Tools Appl 70, 237–261 (2014). https://doi.org/10.1007/s11042-012-1088-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-012-1088-y

Keywords

Search

Navigation