Skip to main content
SpringerLink
Log in

PROMETHEUS: heterogeneous sensor database in support of research on human behavioral patterns in unrestricted environments

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

The multi-modal multi-sensor PROMETHEUS database was created in support of research and development activities [PROMETHEUS (FP7-ICT-214901): http://www.prometheus-FP7.eu] aiming at the creation of a framework for monitoring and interpretation of human behaviors in unrestricted indoor and outdoor environments. The distinctiveness of the PROMETHEUS database comes from the unique sensor sets, used in the various recording scenarios, but also from the database design, which covers a range of real-world applications, correlated to smart-home automation and indoors/outdoors surveillance of public areas. Numerous single-person and multi-person scenarios, but also scenarios with interactions between groups of people, motivated by these applications were implemented with the help of skilled actors and supernumerary personnel. In these scenarios, the actors and personnel were instructed to implement a range of typical and atypical behaviors, and simulations of emergency and crisis situations. In summary, the database contains more than 4 h of synchronized recordings from heterogeneous sensors (an infrared motion detection sensor, thermal imaging cameras, overview/surveillance video cameras, close-view video cameras, a 3D camera, a stereoscopic camera, a general-purpose camcoder, microphone arrays, and motion capture equipment) collected in common setups, simulating smart-home environment, airport, and ATM security environment. Selected scenes of the database were annotated for the needs of human detection and tracking. The entire audio part of the database was annotated for the needs of sound event detection, sound source enumeration, emotion recognition, etc.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. PROMETHEUS (FP7-ICT-214901). http://www.prometheus-FP7.eu

  2. CARETAKER project (IST FP6-027231). http://sceptre.king.ac.uk/caretaker/

  3. Smith K., Ba S., Odobez J.-M., Gatica-Perez D.: Tracking the visual focus of attention for a varying number of wandering people. In: IEEE Trans. Pattern Anal. Mach. Intell. 30(7), 1212–1229 (2008)

    Google Scholar 

  4. Carincotte, C., Desurmont, X., Ravera, B., Bremond, F., Orwell, J., Velastin, S., Odobez, J., Corbucci, J., Palo, B.J., Cernocky J.: Toward generic intelligent knowledge extraction from video and audio: the EU-funded CARETAKER project. In: IEEE Conference on Imaging Detection and Prevention (ICDP), London, UK, pp. 470–475, 13–14 June 2006

  5. SERKET project (ITEA). http://www.capvidia.com/files/SERKET_PR.pdf

  6. Desurmont, X., Sebbe, R., Martin, F., Machy, C., Delaigle J.-F.: Performance evaluation of frequent event detection system. In: International IEEE Workshop on Performance Evaluation of Tracking and Surveillance (PETS), New York, USA (2006)

  7. PETS. Performance evaluation of tracking and surveillance. http://www.cvg.cs.rdg.ac.uk/slides/pets.html

  8. Benabbas, Y., Ihaddadene, N., Djerba C.:Global analysis of motion vectors for event detection in crowded scenes. In: Proceedings 11th IEEE International Workshop on PETS, Miami, pp. 109–116, 25 June 2009

  9. Sharma, P.K., Huang, C., Nevatia, R.: Evaluation of people tracking, counting and density estimation in crowded environments. In: Proceedings 11th IEEE International Workshop on PETS, Miami, pp. 39–46, 25 June 2009

  10. Dalley, G., Wang, X., Grimsin W.E.L.: Event detection using an attention-based tracker. In: Proceedings 10th IEEE International Workshop on PETS, Rio de Janeiro, pp. 71–79, 14 Oct 2009

  11. Arsic, D., Hofmann, M., Schulller, B., Rigoll, G.: Multi-camera person tracking and left luggage detection applying homographic transformation. In: Proceedings 10th IEEE International Workshop on PETS, Rio de Janeiro, pp. 55–62, 14 Oct 2007

  12. CAVIAR. Context aware vision using image-based active recognition. EU IST programme project IST 2001 37540. http://homepage.inf.ed.ac.uk.rbf/CAVIARDATA1

  13. Nascimento, J.C., Figueiredo, M.A.T., Marques, J.S.: Recognizing human activities using space dependent switched dynamical models. In: IEEE International Conference on Image Processning, ICIP’2005, Genoa, Italy, 11–14 Sept 2005

  14. HERMES project (IST-2005-027110). http://www.hermes-project.eu/

  15. Mozerov M., Amato A., Roca X., Gonzales J.: Trajectory occlusion handling with multiple-view distance-minimization clustering. Opt. Eng. 47(4), 2021–2029 (2008)

    Article  Google Scholar 

  16. CogVis project (IST-2000-29375). http://cogvis.nada.kth.se/cogvis-home.html

  17. Needham C.J., Santos P.E., Magee D.R., Devin V., Hogg D.C., Cohn A.G.: Protocols from perceptual observations. Artif. Intell. 167(1–2), 103–136 (2005)

    Article  Google Scholar 

  18. CLEAR. http://www.clear-evaluation.org/

  19. Abad, A., Canton-Ferrer, C., Segura, C., Landabaso, J.L., Macho, D., Casas, J.R., Hernando, J., Pardas, M., Nadeu, C.: UPC audio, video and multimodal person tracking systems in the CLEAR evaluation campaign. In: Lecture Notes in Computer Science, vol. 4122, pp. 93–104 (2006)

  20. Black, J., Ellis, T., Makris D.: A distributed database for effective management and evaluation of CCTV systems. In: Velastin, S.A., Remagnino, P. (eds.) Intelligent Distributed Video Surveillance Systems. Institution of Electrical Engineers, London, UK pp. 55–89 (2006)

  21. O’Toole A.J., Harms J., Sow S.L., Hurst D.R., Pappas M.R., Ayyad J.H., Abdi H.: A video database of moving faces and people. In: IEEE Trans. Pattern Anal. Mach. Intell. 27(5), 812–816 (2005)

    Google Scholar 

  22. Ortega-Garcia J. et al.: The multiscenario multienvironment biosecure multimodal database (BMDB). In: IEEE Trans. Pattern Anal. Mach. Intell. 32(6), 1097–1111 (2010)

    Google Scholar 

  23. Ntalampiras, S., Potamitis, I., Ganchev, T., Fakotakis, N.: Audio database in support of potential threat and crisis situation management. In: Proceedings of the 6th Conference on Language Resources and Evaluation, Morocco, pp. 1288–1291 (2008)

  24. Clavel, C., Vasilescu, I., Devillers, L., Ehrette, T.: Fiction database for emotion detection in abnormal situations. In: Proceedings of International Conference on Spoken Language Processing, pp. 2277–2280, Korea, Oct 2004

  25. Ntalampiras, S., Arsic, D., Stormer, A., Ganchev, T. Potamitis, I., Fakotakis N.: PROMETHEUS database: a multimodal corpus for research on modeling and interpreting human behavior. In: IEEE 17th International Conference on Digital Signal Processing 2009, Special Session: Fusion of Heterogeneous Data for Robust Estimation and Classification, Santorini, Greece, pp. 1–8 (2009)

  26. D2.2: Usage scenarios, functional specifications and hardware components. Deliverable D2.2. PROMETHEUS project, Feb 2009

  27. FP7 cooperation work programme 2009–2010: Information and Communication Technologies. European Commission. July 2009

  28. FP7-ICT-SEC-2007-1: Joint call between ICT and security with themes on critical infrastructure protection. In: FP7 Cooperation Work Programme 2007, European Commission. Feb 2007

  29. Ferryman, J., Tweed D.: An overview of the PETS 2007 dataset. In: Proceedings of the Tenth IEEE International Workshop on Performance Evaluation of Tracking and Surveillance, Rio de Janeiro, Oct 2007

  30. Thirde, D., Li, L., Ferryman, J.: An overview of the PETS 2006 dataset. In: Proceedings of the 2nd Joint IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, Beijing, pp. 317–324, Oct 2005

  31. Directive 95/46/EC of the European Parliament and of the Council of 24 October 1995 on the protection of individuals with regard to the processing of personal data and on the free movement of such data OJ. No L. 281, pp. 0031–0050, 23 Nov 1995

  32. Wallhoff, F., Ruß, M., Rigoll, G., Göbel, J., Diehl, H.: Improved image segmentation using photonic mixer devices. In: Proceedings of International Conference on Image Processing, Texas, vol.~VI, pp. 53–56, Sept 2007

  33. PMD Technologies: Data sheet PMD (vision) 3k-s. Online document: http://www.pmdtec.com/fileadmin/pmdtec/downloads/documentation/datenblatt_camcube3.pdf. Accessed 7 June 2012

  34. Lichtenauer, J., Valstar, M., Shen, J., Pantic, M.: Cost-effective solution to synchronized audio-visual capture using multiple sensors. In: Proceedings of the Advanced Video and Signal Based Surveillance, pp. 324–329, 2–4 Sept 2009

  35. Kipp M.: Anvil—a generic annotation tool for multimodal dialogue. In: Proceedings of the 7th European Conference on Speech Communication and Technology, Aalborg, pp. 1367–1370, 3–7 Sept 2001

  36. UK Home Office: Multiple-camera tracking scenario. Online document. Available on-line at: http://scienceandresearch.homeoffice.gov.uk/hosdb/publications/cctv-publications/MCTS_Sce-nario_Definition_Ma1.pdf. Oct 2008

  37. Mariano, V.Y., Min, J., Park, J.-H., Kasturi, R., Mihalcik, D., Doermann, D., Drayer, T.: Performance evaluation of object detection algorithms. In: Proceedings of International Conference on Pattern Recognition, Quebec, pp. 965–969, 11–15 Aug 2002

  38. PRAAT web-site. http://www.fon.hum.uva.nl/praat/

  39. Hiroaki, N., Takanobu, N., Hiroshi, K.: Acoustic-based security system: towards a robust understanding of emergency shout. In: 5th International Conference on Information Assurance and Security, Xian, China, pp. 725–728, 18–20 Aug 2009

  40. Ntalampiras, S., Potamitis, I., Fakotakis, N.: An adaptive framework for acoustic monitoring of potential hazards. EURASIP Journal on Audio, Speech, and Music Processing. 2009. Article ID 594103 (2009). doi:10.1155/2009/594103

  41. Ntalampiras, S., Potamitis, I., Fakotakis, N.: On acoustic surveillance of hazardous situations. In: International Conference on Acoustics, Speech and Signal Processing, Taiwan, Taipei, 19–24 April 2009, pp. 165–168

  42. Ntalampiras, S., Potamitis, I., Fakotakis, N.: A portable system for robust acoustic detection of atypical situations. In: 17th European Signal Processing Conference, Glasgow, Scotland, 24–28 Aug 2009, pp. 1121–1125

  43. Ganchev, T., Mporas, I., Fakotakis, N.: Automatic height estimation from speech in real-world setup. In: 18th European Signal Processing Conference, Aalborg, Danmark, pp. 800–804, 23–27 Aug 2010

  44. Andersson, M., Ntalampiras, S., Ganchev, T., Rydell, J., Ahlberg, J., Fakotakis, N.: Fusion of acoustic and optical sensor data for automatic fight detection in urban environment. In: International Conference on Information Fusion, Edinburgh, UK, pp. 1–8, 26–29 July 2010

Download references

Author information

Authors and Affiliations

  1. System Architectures Group, Dipartimento di Elettronica e Informazione, Politecnico di Milano, Via Ponzio 34/5, Piazza Leonardo da Vinci, 20133, Milan, Italy

    Stavros Ntalampiras

  2. Institute for Human Machine Communication, Technische Universität München, Arcisstr. 21, 80333, Munich, Germany

    Dejan Arsić & Martin Hofmann

  3. Division of Information Systems, FOI, Swedish Defense Research Agency, Linköping, Sweden

    Maria Andersson

  4. Wire Communications Laboratory, Electrical and Computer Engineering Department, University of Patras, Eratosthenous 6, 26500, Rion, Patras, Greece

    Todor Ganchev

Authors
  1. Stavros Ntalampiras
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dejan Arsić
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Martin Hofmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Maria Andersson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Todor Ganchev
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stavros Ntalampiras.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ntalampiras, S., Arsić, D., Hofmann, M. et al. PROMETHEUS: heterogeneous sensor database in support of research on human behavioral patterns in unrestricted environments. SIViP 8, 1211–1231 (2014). https://doi.org/10.1007/s11760-012-0346-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-012-0346-9

Keywords

Search

Navigation