Skip to main content
Springer Nature Link
Log in

A context-aware middleware for real-time semantic enrichment of distributed multimedia metadata

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper investigates the problem of the real-time integration and processing of multimedia metadata collected by a distributed sensor network. The discussed practical problem is the efficiency of the technologies used in creating a Knowledge Base in real-time. Specifically, an approach is proposed for the real-time, rule-based semantic enrichment of lower level context features with higher-level semantics. The distinguishing characteristic is the provision of an intelligent middleware-based architecture on which low level components such as sensors, feature extraction algorithms, data sources, and high level components such as application-specific ontologies can be plugged. Throughout the paper, Priamos, a middleware architecture based on Semantic Web technologies is presented, together with a stress-test of the system’s operation under two test case scenarios: A smart security surveillance application and a smart meeting room application. Performance measurements are conducted and corresponding results are exposed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. Extensible Metadata Platform (XMP): http://www.adobe.com/products/xmp/.

  2. The SmartWeb project: http://smartweb.dfki.de/.

  3. The PICSEL project: http://www.lri.fr/~sais/picsel3/.

  4. The CHIL project: http://chil.server.de.

  5. Talis homepage: http://www.talis.com.

  6. OpenLink software homepage: http://www.openlinksw.com.

  7. According to [11] the most used ontology authoring environments are Protégé, SWOOP and OntoEdit.

  8. Dublin Core Metadata Element Set: http://www.dublincore.org/documents/dces/.

  9. Creative Commons, Describing Copyright in RDF: http://creativecommons.org/ns.

  10. FOAF Vocabulary Specification: http://xmlns.com/foaf/spec/.

  11. XPath 2.0 is a W3C recommendation since Jan. 23, 2007: http://www.w3.org/TR/xpath20/.

  12. Among the most reliable sources is the prominent Swoogle (http://swoogle.umbc.edu). Noticeable results are also produced with the filetype:owl or filetype:rdf google operators.

  13. Not to be confused with external software applications, here by the term “application”, we note an application built on top of the middleware, i.e. the middleware configured for specific sensors, ontology models and message syntax.

  14. Delete or Update functions are not included in the W3C SPARQL recommendation [41].

  15. Note that due to the openness of the Web, no standards can be imposed or strictly followed. The closest approach to standardisation are the W3C or other consortia’s recommendations.

  16. SWSF Overview: http://www.w3.org/Submission/SWSF/.

References

  1. Allen BD, Bishop G, Welch G (2001) Tracking: beyond 15 minutes of thought. SIGGRAPH Course Pack

  2. Baader F, Nutt W (2002) The description logic handbook. In: Baader F, Calvanese D, McGuinness DL, Nardi D, Patel-Schneider PF (eds) Basic description logics. Cambridge University Press, Cambridge, pp 47–100

    Google Scholar 

  3. Bartlett MS, Movellan JR, Sejnowski TJ (2002) Face recognition by independent component analysis. IEEE Trans Neural Netw 13(6):1450–1464

    Article  Google Scholar 

  4. Bechhofer S (2006) DIG 2.0: the DIG description logic interface. http://dig.cs.manchester.ac.uk/

  5. Bechhofer S, van Harmelen F, Hendler J, Horrocks I, McGuinness D, Patel-Schneider P, Stein LA (2004) OWL web ontology language reference. World wide web consortium, recommendation REC-owl-ref-20040210

  6. Beckett D (2004) RDF/XML syntax specification(Revised). http://www.w3.org/TR/rdf-syntax-grammar/ W3C Recommendation

  7. Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720

    Article  Google Scholar 

  8. Black W, McNaught J, Vasilakopoulos A, Zervanou K, Rinaldi F (2003) CAFETIERE: conceptual annotations for facts, events, terms, individual entities and rElations. Tech rep, UMIST, Parmenides TR-U4, 3.1. Manchester, UK

  9. Bradski GR (1998) Computer vision face tracking for use in a perceptual user interface. Intel Technol J (1998)

  10. Buitelaar P, Olejnik D, Sintek M (2004) A protégé plug-in for ontology extraction from text based on linguistic analysis. In: First European semantic web symposium (ESWS). Heraklion, Greece

    Google Scholar 

  11. Cardoso J (2007) The semantic web vision: where are we? IEEE Intell Syst 22(5):84–88

    Article  Google Scholar 

  12. Carroll J, Dickinson I, Dollin C, Reynolds D, Seaborne A, Wilkinson K (2003) Jena: implementing the semantic web recommendations. Tech Rep HPL-2003-146, Hewlett-Packard

  13. Chakravarthy A, Ciravegna F, Lanfranchi V (2006) Cross-media document annotation and enrichment. In: Proc 1st semantic web authoring and annotation workshop (SAAW2006)

  14. Chen H, Finin T, Joshi A (2004) Semantic web in the context broker architecture. In: PERCOM ’04: proc of the second IEEE int’l conf on pervasive computing and communications (PerCom’04). IEEE Computer Society, Washington, DC, p 277

    Chapter  Google Scholar 

  15. Comaniciu D, Meer P (1999) Mean shift analysis and applications. In: Proc of the seventh IEEE int’l conf on computer vision (ICCV’99), vol 2. Kerkyra, Greece, pp 1197–1203

  16. Dey A (2001) Understanding and using context. Personal and Ubiquitous Computing 5(1):4–7. doi:10.1007/s007790170019

    Article  Google Scholar 

  17. Dey A, Abowd G, Salber D (1999) A context-based infrastructure for smart environments. In: First int’l workshop on managing interactions in smart environments (MANSE 99). Dublin, Ireland, pp 114–128

    Google Scholar 

  18. Dou D, Pan J, Qin H, LePendu P (2006) Towards populating and querying the semantic web. In: Proc of 2nd international workshop on scalable semantic web knowledge base systems (SSWS’06). Co-located with ISWC, pp 129–142

  19. Dougherty E, Laplante P (1995) Introduction to real-time imaging, chap. What is real-time processing? Wiley-IEEE, New York, pp 1–9

    Google Scholar 

  20. Etzioni O, Cafarella M, Downey D, Popescu AM, Shaked T, Soderland S, Weld D, Yates A (2005) Unsupervised named-entity extraction from the web: an experimental study. Artif Intell 165(1):91–134. doi:10.1016/j.artint.2005.03.001

    Article  Google Scholar 

  21. Francois ARJ, Nevatia R, Hobbs J, , Bolles RC (2005) VERL: an ontology framework for representing and annotating video events. IEEE MultiMed 12(4):76–86

    Article  Google Scholar 

  22. Goasdoué F, Reynaud C (1999) Modeling information sources for information integration. In: 11th European workshop on knowledge acquisition, modeling and management (EKAW 99), Lecture notes in artificial intelligence, vol 1621. Springer, Dagstuhl Castle, pp 121–138

    Google Scholar 

  23. Hirtle D, Boley H, Grosof B, Kifer M, Sintek M, Tabet S, Wagner G (2006) Schema specification of ruleML 0.91. http://www.ruleml.org/0.91/

  24. Horrocks I (1998) Using an expressive description logic: faCT or fiction? In: Cohn AG, Schubert L, Shapiro S (eds) Principles of knowledge representation and reasoning: proceedings of the 6th int’l conference (KR-98). Morgan Kaufman, San Francisco, pp 636–647

    Google Scholar 

  25. Horrocks I, Patel-Schneider P, van Harmelen F (2003) From \(\mathcal{SHIQ}\) and RDF to OWL: the making of a web ontology language. Journal of Web Semantics: Science, Services and Agents on the World Wide Web 1(1):7–26

    Article  Google Scholar 

  26. Iria J, Ciravegna F, Cimiano P, Lavelli A, Motta E, Gilardoni L, Mönch E (2004) Integrating information extraction, ontology learning and semantic browsing into organizational knowledge processes. In: Proc of the EKAW workshop on the application of language and semantic technologies to support knowledge management processes, at the 14th int’l conf on knowledge engineering and knowledge management

  27. Kagal L, Finin T, Johshi A (2003) A policy language for pervasive computing environment. In: Proc of IEEE fourth international workshop on policies for distributed systems and networks (POLICY’03)

  28. Karame G, Stergiou A, Katsarakis N, Papageorgiou P, Pnevmatikakis A (2007) 2D and 3D face localization for complex scenes. In: Advanced video and signal based surveillance, AVSS 2007. IEEE Conference on, pp 371–376

  29. Kaykova O, Khriyenko O, Kovtun D, Naumenko A, Terziyan, VAAZ (2005) General adaption framework: enabling interoperability for industrial web resources. International Journal on Semantic Web and Information Systems 1(3):31–63

    Google Scholar 

  30. Konstantinou N, Solidakis E, Zoi S, Zafeiropoulos A, Stathopoulos P, Mitrou N (2007) Priamos: a middleware architecture for real-time semantic annotation of context features. In: IET int’l conf on intelligent environments (IE’07). Ulm, Germany, pp 96–103

    Chapter  Google Scholar 

  31. Lassila O, Khushraj D (2005) Contextualizing applications via semantic middleware. In: Proc of the second annual international conference on mobile and ubiquitous systems: networking and services (MOBIQUITOUS’05). IEEE Computer Society, Washington, DC, pp 183–191

    Chapter  Google Scholar 

  32. Lien CC, Chiang CL, Lee CH (2007) Scene-based event detection for baseball videos. J Vis Commun Image Represent 18(1):1–14

    Article  Google Scholar 

  33. Liu S, Xu M, Yi H, Chia LT, Rajan D (2006) Multimodal semantic analysis and annotation for basketball video. EURASIP J Appl Signal Process 2006:1–13

  34. Masuoka R, Parsia B, Labrou Y (2003) Task computing—the semantic web meets pervasive computing. In: 2nd int’l semantic web conference (ISWC’03). Sanibel Island, Florida, USA

    Google Scholar 

  35. May W, Alferes J, Amador R (2005) Rules and rule markup languages for the semantic Web, Lecture notes in computer science, vol 3791, chap active rules in the semantic web: dealing with language heterogeneity. Springer, Berlin, pp 30–44

    Google Scholar 

  36. Motik B, Sattler U (2006) A comparison of reasoning techniques for querying large description logic aBoxes. In: Hermann M, Voronkov A (eds) Proc of the 13th int’l conf on logic for programming artificial intelligence and reasoning (LPAR’06), LNCS, vol 4246. Springer, Phnom Penh, pp 227–241

    Chapter  Google Scholar 

  37. Pandis I, Soldatos J, Paar A, Reuter J, Carras M, Polymenakos L (2005) An ontology-based framework for dynamic resource management in ubiquitous computing environments. In: Proc of the 2nd int’l conf on embedded software and systems (ICESS’05)

  38. Papamarkos G, Poulovassilis A, Wood PT (2003) Event-condition-action rule languages for the semantic web. In: Workshop on semantic web and databases (SWDB 03), pp 309–327

  39. Patel-Schneider P, Horrocks I (2004) OWL web ontology language: semantics and abstract syntax Section 3. Direct model-theoretic semantics. http://www.w3.org/TR/owl-semantics/direct.html#3.1

  40. Petridis K, Anastasopoulos D, Saathoff C, Timmermann N, Kompatsiaris I, Staab S (2006) M-ontoMat-annotizer: image annotation. Linking ontologies and multimedia low-level features. In: Engineered applications of semantic web session (SWEA) at the 10th int’l conf on knowledge-based intelligent information and engineering systems (KES’06). Bournemouth, UK

    Google Scholar 

  41. Prud’hommeaux E, Seaborne A (2008) SPARQL query language for RDF. http://www.w3.org/TR/rdf-sparql-query/

  42. Román M, Hess C, Cerqueira R, Ranganathan A, Campbell R, Nahrstedt K (2002) Gaia: a middleware platform for active spaces. ACM SIGMOBILE Mobile Computing and Communications Review 6(4):65–67

    Article  Google Scholar 

  43. Schroeter R, Hunter J, Guerin J, Khan I, Henderson M (2006) A synchronous multimedia annotation system for secure collaboratories. In: Proceedings of the second IEEE int’l conf on e-science and grid computing (E-SCIENCE’06). IEEE Computer Society, Washington, DC, p 41

    Chapter  Google Scholar 

  44. Seaborne A, Manjunath G (2008) SPARQL/Update: a language for updating RDF graphs http://jena.hpl.hp.com/~afs/SPARQL-Update.html

  45. Sirin E, Parsia B, Grau B, Kalyanpur A, Katz Y (2007) Pellet: a practical OWL-DL reasoner. Journal of Web Semantics: Science, Services and Agents on the World Wide Web 5(2):51–53

    Article  Google Scholar 

  46. Sohn J, Kim NS, Sung W (1999) A statistical model-Based voice activity detection. IEEE Signal Process Lett 6(1):1–3

    Article  Google Scholar 

  47. Stamou G, van Ossenbruggen J, Pan J, Schreiber G, Smith J (2006) Multimedia annotations on the semantic web. IEEE Multimed 13(1):86–90

    Article  Google Scholar 

  48. Stergiou A, Pnevmatikakis A, Polymenakos L (2007) The AIT multimodal person identification system for CLEAR 2007. In: Multimodal technologies for perception of humans, pp 221–232

  49. Toninelli A, Montanari R, Kagal L, Lassila O (2006) The semantic web - ISWC 2006, Lecture notes in computer science, vol 4273, chap A semantic context-aware access control framework for secure collaborations in pervasive computing environments. Springer, Berlin, pp 473–486

    Google Scholar 

  50. Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):71–86

    Article  Google Scholar 

  51. Uren V, Cimiano P, Iria J, Handschuh S, Vargas-Vera M, Motta E, Ciravegna F (2006) Semantic annotation for knowledge management: requirements and a survey of the state of the art. Journal of Web Semantics, Elsevier 4(1):14–28

    Article  Google Scholar 

  52. Vazquez JI, de Ipiña DL, nigo Sedano I (2006) Computational science and its applications—ICCSA 2006, workshop on ubiquitous web systems and intelligence (UWSI 2006), Lecture notes in computer science, vol 3983/2006, chap SOAM: an environment adaptation Model for the pervasive semantic web. Springer, Berlin, pp 108–117

    Google Scholar 

  53. Vembu S, Kiesel M, Sintek M, Baumann S (2006) Towards bridging the semantic gap in multimedia annotation and retrieval. In: Proc of the 1st international workshop on semantic web annotations for multimedia, SWAMM 2006 at the 15th int’l world wide web conference

  54. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: IEEE conf on computer vision and pattern recognition. Hawaii, pp 511–518

  55. Vu VT, Bremond F, Thonnat M (2003) Automatic video interpretation: a novel algorithm for temporal scenario recognition. In: Proc of the 18th int’l joint conf on artificial intelligence (IJCAI 2003), pp 1295–1302

  56. Zafeiropoulos A, Konstantinou N, Arkoulis S, Spanos DE, Mitrou N (2008) A semantic-based architecture for sensor data fusion. In: 2nd int’l conf on mobile ubiquitous computing, systems, services and technologies (UBICOMM’08). Valencia, Spain, pp 116–121

    Chapter  Google Scholar 

  57. Zhang D, Chang SF (2002) Event detection in baseball video using superimposed caption recognition. In: Proc of the tenth ACM int’l conf on multimedia. ACM, New York, pp 315–318

    Chapter  Google Scholar 

Download references

Acknowledgements

The work presented in this paper is carried out within the Priamos project, sponsored by the Greek General Secretarial of Research and Technology “Image, Sound, Language” research and development action. The authors of this paper would like to thank their colleagues in Athens Information Technology (http://www.ait.edu.gr/) for their collaboration.

Author information

Authors and Affiliations

  1. National Technical University of Athens, Heroon Polytechneiou str., 15773, Zografou, Athens, Greece

    Nikolaos Konstantinou, Emmanuel Solidakis, Anastasios Zafeiropoulos, Panagiotis Stathopoulos & Nikolas Mitrou

Authors
  1. Nikolaos Konstantinou
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Emmanuel Solidakis
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Anastasios Zafeiropoulos
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Panagiotis Stathopoulos
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Nikolas Mitrou
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Nikolaos Konstantinou.

Appendices

Appendix A: the Priamos mapping rule language syntax

Below is provided the grammar for the Priamos Rule language. We use the BNF metasyntax to clarify the use of both subsets of the rule language: the Mapping and the Semantic Rule Language.

Appendix B: the Priamos semantic rule language syntax

For both languages, String and Integer denote strings and integers respectively in programming language terms. OntClass and OntProperty refer to the full URIs or the QNames of classes and properties of the ontology, respectively. PathToExecutable is the full path to an executable file of the Operating System’s filesystem, and finally, XPathExpression is an XPath expression used to select nodes from an XML document.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Konstantinou, N., Solidakis, E., Zafeiropoulos, A. et al. A context-aware middleware for real-time semantic enrichment of distributed multimedia metadata. Multimed Tools Appl 46, 425–461 (2010). https://doi.org/10.1007/s11042-009-0361-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-009-0361-1

Keywords