Skip to main content
SpringerLink
Log in

FiM’s DE - the communication package for the creative pipeline

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The FotoInMotion (FiM) project is building a novel media creation platform, leveraging the use of semi-automated analysis and editing tools to empower creators to easily transform static visual acquisitions of real-world events into rich, animated and engaging objects, distributable through common channels. FiM transforms the content creative chain into an integrated pipeline across which media and metadata seamlessly flow and are exploited to produce more complex media objects. One of the addressed challenges consists the need for a seamless and efficient communication across such pipeline and on how to preserve, in a structured manner, all of the involved media and metadata. Existing standardized metadata tools and content wrappers are limited in expressivity and scope and incapable of fully supporting the needs of the content creative pipeline. This paper describes FiM’s new structured data object, i.e. the Digital Event (DE), which acts as a universal vehicle for media and metadata. It builds on well-established and emergent MPEG standards (MPEG-21, MPEG-V, MPEG-7 and MPEG HEIF), to support data diversity, interoperability, packaging and sharing, within complex, Machine Learning enhanced, creative pipelines. Our solution has been validated by creative professionals (photojournalism, fashion marketing and festivals), who have conducted experiments within the context of different creative workflows in real world scenarios. DE’s employment revealed to be advantageous, particularly in the homogenization of the media and metadata representation and packaging and in the normalization of the interaction between different pipeline components.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data availability

Not applicable.

Notes

  1. The metadata content for each DE mentioned in this use case scenario, along with their validating schemas is available at [46]

References

  1. Adobe (n.d.) XMP Specification, https://www.adobe.com/products/xmp.html. Accessed 02 Feb 2021

  2. Advanced Authoring Format page at the Advanced Media Workflow Association Official Website (n.d.) https://www.amwa.tv/aaf. Accessed 02 Feb 2021

  3. Arndt R, Troncy R, Staab S, Hardman L (2009) COMM: a core ontology for multimedia annotation. In: Staab S, Studer R (eds) Handbook on ontologies. Springer, Heidelberg, pp 403–421. https://doi.org/10.1007/978-3-540-92673-3_18

    Chapter  Google Scholar 

  4. Athanasiadis T, Tzouvaras V, Petridis K, Precioso F, Avrithis Y, Kompatsiaris Y (2005) Using a multimedia ontology infrastructure for semantic annotation of multimedia content, 5th International Workshop on knowledge markup and Semantic Annotation, located at the 4rd International Semantic Web Conference ISWC 2005, Galway Ireland, November 7 2005

  5. Baca M (2016)Introduction to metadata: Third Edition, Getty Publications

  6. Baca M, Harpring P (2014) Categories for the Description of Works of Art: Describe and catalogue works of art, architecture, and cultural heritage (CDWA), http://www.getty.edu/research/publications/electronic_publications/cdwa/. Accessed 02 Feb 2021

  7. Blöhdorn S, Petridis K, Saathoff C, Simou N, Tzouvaras V, Avrithis Y, Handschuh S, Kompatsiaris Y, Staab S and Strintzis MG (2005) Semantic annotation of images and videos for multimedia analysis. In: European semantic web conference, Heraklion, May–June 2005. Springer, Berlin, Heidelberg, pp 592–607

  8. Burnett I, Davis S, Drury GM (2005) MPEG-21 digital item declaration and identification – principles and compression. IEEE Trans Multimed 7(3):400–407

  9. Camera & Imaging Products Association (n.d.) Exchangeable image file format for digital still cameras v2.3, http://www.cipa.jp/std/documents/e/DC-008-2012_E.pdf. Accessed 02 Feb 2021

  10. Castro H, Monteiro J, Pereira A, Silva D, Coelho G, Carvalho P (2016) Cognition inspired format for the expression of computer vision metadata. Multimed Tools Appl 75(24):17035–170572015. https://doi.org/10.1007/s11042-015-2974-x

    Article  Google Scholar 

  11. Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The pascal visual object classes (voc) challenge. Int J Comput Vis 88(2):303–338

    Article  Google Scholar 

  12. FotoInMotion Website (n.d.) https://fotoinmotion.eu/. Accessed 02 Feb 2021

  13. Hare JS, Lewis PH, Enser PG, Sandom CJ (2006) Mind the gap: another look at the problem of the semantic gap in image retrieval, in multimedia content analysis, management, and retrieval 2006. Int Soc Optics Photon 6073:607309

  14. HDF5 Group (n.d.) HDF5 Support Page, http://portal.hdfgroup.org/display/HDF5/HDF5. Accessed 02 Feb 2021

  15. Hunter J (2001) Adding multimedia to the semantic web—building an MPEG-7 ontology, Proceedings of the First International Conference on Semantic Web Working, July 2001, pp 261–283

  16. Hunter J, Lagoze C (2001) Combining RDF and XML schemas to enhance interoperability between metadata application profiles. World Wide Web 1:457–466

    Google Scholar 

  17. IPTC ORG Site (n.d.) NewsML-G2 Guidelines, https://www.iptc.org/std/NewsML-G2/guidelines/#quick-start-guide-to-newsml-g2-basics. Accessed 02 Feb 2021

  18. IPTC ORG Site (n.d.) PTC Photo Metadata Standard 2017.1, http://www.iptc.org/std/photometadata/specification/IPTC-PhotoMetadata. Accessed 02 Feb 2021

  19. Isaac A, Troncy R (2004) Designing and using an audio-visual description core ontology, Workshop on Core Ontologies in Ontology Engineering, Northamptonshire UK, 8 October 2004, vol. 118

  20. MPEG Group (2003) Information technology — Multimedia content description interface — Part 5: Multimedia description schemes, International Standard ISO/IEC 15938-5:2003, Geneva Switzerland

  21. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft coco: Common objects in context. In: European conference on computer vision, Zurich Switzerland, 6-12 September 2004. Springer, Cham, pp 740–755

  22. List T, Fisher RB (2004) CVML – An XML-based Computer Vision Markup Language, In Proceedings of the 17th International Conference on Pattern Recognition (ICPR 2004), Cambridge UK, August 23-26 2004, IEEE Computer Society

  23. Makadia A, Pavlovic V, Kumar S (2010) Baselines for Image Annotation. Int J Comput Vis 90(1):88–105

    Article  Google Scholar 

  24. Miller E (1998) An introduction to the resource description framework. Bull Am Soc Inf Sci Technol 25(1):15–19

    Article  Google Scholar 

  25. MPEG Group (2017) Information technology — High efficiency coding and media delivery in heterogeneous environments — Part 12: Image File Format, International Standard ISO/IEC 23008-12:2017, Geneva Switzerland

  26. MPEG Group (n.d.) MPEG-21, https://mpeg.chiariglione.org/standards/mpeg-21. Accessed 02 Feb 2021

  27. MPEG Group (n.d.) MPEG-7, https://mpeg.chiariglione.org/standards/mpeg-7. Accessed 02 Feb 2021

  28. MPEG Group (2002) Information technology — Multimedia content description interface — Part 3: Visual, International Standard ISO/IEC 15938-3:2002, Geneva Switzerland

  29. MPEG Group (n.d.) MPEG-V, https://mpeg.chiariglione.org/standards/mpeg-v. Accessed 02 Feb 2021

  30. MPEG Group (n.d.) MPEG-V Sensory Information, https://mpeg.chiariglione.org/standards/mpeg-v/sensory-information. Accessed 02 Feb 2021

  31. MPEG Group (n.d.) MPEG-V Data formats for interaction devices, https://mpeg.chiariglione.org/standards/mpeg-v/data-formats-interaction. Accessed 02 Feb 2021

  32. MPEG Group (n.d.) MPEG-H, https://mpeg.chiariglione.org/standards/mpeg-h. Accessed 02 Feb 2021

  33. MPEG Group (n.d.) MPEG-H Part 12 Image File Format, https://mpeg.chiariglione.org/standards/mpeg-h/image-file-format. Accessed 02 Feb 2021

  34. MPEG Group (n.d.) MPEG-H Home Site, https://mpeg.chiariglione.org/standards/mpeg-h. Accessed 02 Feb 2021

  35. MPEG Group (2003) Information technology — Multimedia framework (MPEG-21) — Part 3: Digital Item Identification, International Standard ISO/IEC 21000-3:2003, Geneva Switzerland

  36. MPEG Group (2004) Information technology — Multimedia framework (MPEG-21) — Part 5: Rights Expression Language, International Standard ISO/IEC 21000-5:2004, Geneva Switzerland

  37. MPEG Group (2004), Information technology — Multimedia framework (MPEG-21) — Part 7: Digital Item Adaptation, International Standard ISO/IEC 21000-7:2004, Geneva Switzerland

  38. MPEG Group (2005) Information technology — Multimedia framework (MPEG-21) — Part 2: Digital Item Declaration, International Standard ISO/IEC 21000-2:2005, Geneva Switzerland

  39. MPEG Group (2006) Information technology — Multimedia framework (MPEG-21) — Part 4: Intellectual Property Management and Protection Components, International Standard ISO/IEC 21000-4:2006, Geneva Switzerland

  40. MPEG Group (2007) Information technology — Multimedia framework (MPEG-21) — Part 3: Digital Item Identification — Amendment 1: Related identifier types, International Standard Amendment ISO/IEC 21000-3:2003/Amd 1:2007, Geneva Switzerland

  41. MPEG Group (2013) Information technology — Multimedia framework (MPEG-21) — Part 3: Digital Item Identification — Amendment 2: Digital item semantic relationships, International Standard Amendment ISO/IEC 21000-3:2003/Amd 2:2013, Geneva Switzerland

  42. Oberle D, Ankolekar A, Hitzler P et al (2007) DOLCE ergo SUMO: on foundational and domain models in the SmartWeb integrated ontology (SWIntO). J Web Semant Sci Serv Agents World Wide Web 5(3):156–174

    Article  Google Scholar 

  43. Pereira F, Koenen R (2001) MPEG-7: a standard for multimedia content description. Int J Image Graph 1(3):527–547

    Article  Google Scholar 

  44. Project CAVIAR (n.d.) website, http://homepages.inf.ed.ac.uk/rbf/CAVIAR. Accessed 02 Feb 2021

  45. Project ViPER (n.d.) website, http://viper-toolkit.sourceforge.net. Accessed 02 Feb 2021

  46. Repository of FiM DE metadata schemas and samples (n.d.) https://drive.google.com/drive/folders/1QUxCzKk62Z9H-b6X8UXB9MXx9y0f_fE9?usp=sharing. Accessed 02 Feb 2021

  47. SMPTE (2011) Material Exchange Format (MXF) – File Format Specification, SMPTE ST 377-1, 2011 Edition

  48. Thomee B, Shamma DA, Friedland G, Elizalde B, Ni K, Poland D, Borth D, Li LJ (2015) The new data and new challenges in multimedia research, arXiv preprint arXiv:1503.01817, no. 8, March 2015

  49. Tsinaraki C, Polydoros P, Moumoutzis N, Christodoulakis S (2004) Integration of OWL ontologies in MPEG-7 and TV-anytime compliant semantic indexing. 16th International conference on advanced information systems engineering, Riga, June 2004. Lecture notes in computer science, vol. 3084, pp. 398–413. Springer, Heidelberg. https://doi.org/10.1007/978-3-540-25975-6_29

  50. Vellido A, Martín-Guerrero JD, Lisboa PJ (2012) Making machine learning models interpretable. ESANN 12:163–172

    Google Scholar 

  51. Vembu S, Kiesel M, Sintek M and Baumann S (2006) Towards bridging the semantic gap in multimedia annotation and retrieval. In 1st International Workshop on Semantic Web Annotations for Multimedia (SWAMM), as part of the 15th World Wide Web Conference, Edinburgh Scotland, 22 May 2006

  52. Viana P, Alves AP (2010) A semantic management model to enable the integrated management of media and devices. Multimed Tools Appl 49(1):37–62

    Article  Google Scholar 

  53. Weibel S, Kunze J, Lagoze C, Wolf M Dublin core metadata for resource discovery. Internet Eng Task Force RFC 2413(222):132

Download references

Funding

This work was developed with the financial support of the Fundação para a Ciência e Tecnologia (FCT), Portugal, within the scope of the post-Doctoral grant with the reference number SFRH/BPD/108329/2015.

It was also partially funded by the European Union’s Horizon 2020 research and innovation programme under the grant H2020-ICT-20-2017-1-RIA-780612.

Author information

Authors and Affiliations

  1. INESC TEC, Campus da FEUP, Rua Dr. Roberto Frias, 4200 - 465, Porto, Portugal

    H. Castro, M. T. Andrade & P. Viana

  2. FEUP, Rua Dr. Roberto Frias, s/n, 4200-465, Porto, Portugal

    M. T. Andrade

  3. ISEP, R. Dr. António Bernardino de Almeida 431, 4249-015, Porto, Portugal

    P. Viana

Authors
  1. H. Castro
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. M. T. Andrade
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. P. Viana
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to H. Castro.

Ethics declarations

Conflict of interest

Not applicable.

Code availability

Not applicable.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Castro, H., Andrade, M.T. & Viana, P. FiM’s DE - the communication package for the creative pipeline. Multimed Tools Appl 80, 18151–18180 (2021). https://doi.org/10.1007/s11042-020-10282-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-10282-0

Keywords

Search

Navigation