Skip to main content
SpringerLink
Log in

Video-based interactive storytelling using real-time video compositing techniques

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Interactive storytelling systems usually adopt computer graphics to represent virtual story worlds, which facilitates the dynamic generation of visual content. However, the quality of the images and motion produced by these systems is still inferior compared to the high quality experience found in live-action films. Interactive rates in photorealistic rendering for the film industry will not be possible for decades to come. A promising alternative is the replacement of 3D virtual characters with video sequences with real actors. In this paper, we propose a new method for video-based interactive narratives that uses video compositing algorithms that run at truly interactive frame rates. The proposed method is consistent with plots that are generated by nondeterministic planning algorithms. Moreover, we propose a system of artificial intelligent agents that perform the same roles played by filmmaking professionals. A user evaluation of the proposed method is presented. We believe that future improvements of the techniques proposed in this paper represent an important contribution to the quest for new and more immersive forms of interactive cinema.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. Fast Artificial Neural Network Library (FANN) - http://leenissen.dk/fann/

  2. One of the story variants, with English subtitles, can be found as “Modern Little Red Riding Hood” in http://www.icad.puc-rio.br/~logtell/videos.php

References

  1. Ahanger G, Little TDC (1997) A system for customized news delivery from video archives. In: Proceedings of International Conference on Multimedia Computing and Systems, Ottawa, pp 526–533

  2. Arijon D (1976) Grammar of the film language. Silman-James Press, Los Angeles

    Google Scholar 

  3. Barbosa SDJ, Lima ES, Furtado AL, Feijó B (2014) Generation and Dramatization of Detective Stories. SBC Journal on 3D Interactive Systems 5:39–52

    Google Scholar 

  4. Barbosa SDJ, Silva FAG, Furtado AL, Casanova MA (2015) Plot generation with character-based decisions. Comput Entertain 12:1–21

    Article  Google Scholar 

  5. Bocconi S (2006) Vox Populi: generating video documentaries from semantically annotated media repositories. PhD Thesis, Technische Universiteit Eindhoven, Netherlands

  6. Bredow R, Hastings A, Schaub D, Kramer D, Engle R (2015) From Mocap to Movie: The Polar Express. Course # 28, SIGGRAPH 2015, Los Angeles

  7. Brown B (2011) Cinematography: theory and practice: image making for cinematographers and directors. Focal Press, Waltham

    Google Scholar 

  8. Brown M, Lowe D (2007) Automatic panoramic image stitching using invariant features. Int J Comput Vis 74(1):59–77

    Article  Google Scholar 

  9. Brown SL, Vaughan CC (2009) Play: how it shapes the brain, opens the imagination, and invigorates the soul. Avery Publishing Group, New York

    Google Scholar 

  10. Cavazza M, Charles F, Mead S (2002) Character-based interactive storytelling. IEEE Intell Syst, Special issue AI in Interactive Entertainment 17(4):17–24

    MATH  Google Scholar 

  11. Cavazza M, Lugrin J-L, Pizzi D, and Charles F (2007) Madame Bovary on the Holodeck: Immersive Interactive Storytelling. In: Proceedings of the 15th ACM International Conference on Multimedia, Augsburg, pp 651–660

  12. Chabert CF, Einarsson P, Jones A, Lamond B, Ma WC, Sylwan S, Hawkins T, Debevec P (2006) Relighting human locomotion with flowed reflectance fields. In: ACM SIGGRAPH 2006 Sketches, New York

  13. Chua TS, Ruan LQ (1995) A video retrieval and sequencing system. ACM Trans Inf Syst 13(4):373–407

    Article  Google Scholar 

  14. Ciarlini AEM, Pozzer CT, Furtado AL, Feijó B (2005) A logic-based tool for interactive generation and dramatization of stories. In: Proceedings of the International Conference on Advances in Computer Entertainment Technology, Valencia, pp 133–140

  15. Davenport, G., Murtaugh, M (1995) ConText Towards the Evolving Documentary. In: Proceedings of ACM Multimedia '95, San Francisco, pp 377–389

  16. Doria TR, Ciarlini AEM, Andreatta A (2008) A Nondeterministic Model for Controlling the Dramatization of Interactive Stories. In: Proceedings of the ACM Multimedia 2008 - 2nd ACM Workshop on Story Representation, Mechanism and Context - SRMC08, Vancouver, pp 21–26

  17. Ekman P, Friesen WV (1971) Constants across cultures in the face and emotion. J Pers Soc Psychol 17:124–129

    Article  Google Scholar 

  18. Foster J (2010) The green screen handbook: real-world production techniques. Sybex, Indianapolis

    Google Scholar 

  19. Ghallab M, Nau D, Traverso P (2004) Automated planning: theory and practice. Mor-gan Kaufmann Publishers, San Francisco

    MATH  Google Scholar 

  20. Haykin SO (2008) Neural networks and learning machines, 3rd edn. Prentice Hall, Upper Saddle River

    Google Scholar 

  21. Joshi N, Matusik W, Avidan S (2006) Natural video matting using camera arrays. ACM Trans Graph 25:779–786

    Article  Google Scholar 

  22. Jung Von Matt/Spree (2010) Last Call, Berlin http://www.youtube.com/watch?v=qe9CiKnrS1w. Accessed 25 Jan 2016

  23. Liang C, Xu C, Cheng J, Min W, Lu H (2013) Script-to-movie: a computational framework for story movie composition. IEEE Trans Multimedia 15(2):401–414

    Article  Google Scholar 

  24. Lima ES, Feijó B, Pozzer CT, Ciarlini AEM, Barbosa SDJ, Furtado AL, Silva FGA (2012a) Social Interaction for Interactive Storytelling. In: Proceedings of the 11th International Conference on Entertainment Computing Bremen, Germany, pp 1–15

  25. Lima ES, Feijó B, Furtado AL, Pozzer C, Ciarlini A (2012b) Automatic Video Editing For Video-Based Interactive Storytelling. In: Proceedings of the 2012 I.E. International Conference on Multimedia and Expo (ICME), Melbourne, pp 806–811

  26. Lima ES, Feijó B, Furtado AL, Pozzer CT, Ciarlini AEM, Silva FG (2012c) A multi-user natural language Interface for interactive storytelling in TV and cinema. In: Proceedings of the XI Brazilian Symposium on Computer Games and Digital Entertainment, Brasília, pp. 154–161

  27. Lima ES, Feijó B, Barbosa SDJ, Furtado AL, Ciarlini AEM, Pozzer CT (2014) Draw your own story: paper and pencil interactive storytelling. Entertainment Computing 5(1):33–41

    Article  Google Scholar 

  28. Lima ES, Feijó B, Furtado AL (2015) Storytelling Variants: The Case of Little Red Riding Hood. Proceedings of the 14th International Conference on Entertainment Computing (ICEC 2015), Trondheim, pp 286–300

  29. Lima ES, Feijó B, Casanova MA, Furtado AL (2016) Storytelling variants based on semiotic relations. Entertainment Computing 17:31–44

    Article  Google Scholar 

  30. Logtell (2016) Logtell Project Web Site http://www.icad.puc-rio.br/~logtell/. Accessed 25 Jan 2016

  31. Mascelli J (1965) The five C's of cinematography: motion picture filming techniques. Silman-James Press, Los Angeles

    Google Scholar 

  32. Mateas, M (2002) Interactive Drama, art, and artificial intelligence. Doctoral Thesis. School of Computer Science, Carnegie Mellon University, Pittsburgh

  33. Mateas M, Vanouse P, Domike S (2000) Generation of ideologically-biased historical documentaries. In: Proceedings of the Seventeenth National Conference on Artificial intelligence and Twelfth Conference on innovative Applications of Artificial intelligence. AAAI Press, pp 236–242

  34. Mehlmann G, Endrass B, and André E (2011) Modeling and Interpretation of Multithreaded and Multimodal Dialogue. In: Proceedings of the 13th International Conference on Multimodal Interaction (ICMI 2011), pp 385–392

  35. Mehta S, Ramamoorthi R, Meyer M, Hery C (2012) Analytic tangent irradiance environment maps for anisotropic surfaces. Comput Graphics Forum 31(4):1501–1508

    Article  Google Scholar 

  36. Müller W, Spierling U, Stockhausen C (2013) Production and delivery of interactive narratives based on video snippets. In: Proceedings of the 6th international conference on interactive digital storytelling, Istanbul, Turkey, pp 71–82

  37. Ng R, Ramamoorthi R, Hanrahan P (2003) All-frequency shadows using non-linear wavelet lighting approximation. ACM Trans Graph 22(3):376–381

    Article  Google Scholar 

  38. Piacenza A, Guerrini F, Adami N, Leonardi R, Porteous J, Teutenberg J, Cavazza M (2011) Generating Story Variants with Constrained Video Recombination. In: Proceedings of the 19th ACM International Conference on Multimedia 2011, Scottsdale, pp 223–232

  39. Pizzi D, Cavazza M (2007) Affective storytelling based on Characters' feelings. AAAI Fall Symposium on Intelligent Narrative Technologies, Arlington

    Google Scholar 

  40. Porteous J, Benini S, Canini L, Charles F, Cavazza M, Leonardi R (2010) Interactive storytelling via video content recombination. In: Proceedings of the 18th ACM International Conference on Multimedia 2010, Firenze, pp 1715–1718

  41. Porter T, Duff T (1984) Compositing Digital Images. Comput Graph 18(3):253–259

    Article  Google Scholar 

  42. Riedl MO, Young RM (2006) From linear story generation to branching story graphs. IEEE Comput Graph Appl 26(3):23–31

    Article  Google Scholar 

  43. Schoenau-Fog H (2011) Hooked! – Evaluating Engagement as Continuation Desire in Interactive Narratives. In: Fourth International Conference on Interactive Digital Storytelling (ICIDS 2011), Vancouver, pp 219–230

  44. Shen YT, Lieberman H, Davenport G (2009) What’s next? Emergent storytelling from video collections. In: Proceeding of International Conference on Human factors in computing systems, ACM Press, pp 809–818

  45. Silva FGA, Ciarlini AEM, Siqueira SWM (2010) Nondeterministic planning for generating interactive plots. In: 12th Ibero-American conference on AI, Bahía Blanca. Springer, Argentina, pp 133–143

    Google Scholar 

  46. Sun J, Li Y, Kang SB, Shum H-Y (2006) Flash matting. ACM Trans Graph 25(3):772–778

    Article  Google Scholar 

  47. Thompson R, Bowen C (2009) Grammar of the shot. Focal Press, Burlington

    Google Scholar 

  48. Ursu MF, Kegel IC, Williams D, Thomas M, Mayer H, Zsombori V, Uomola ML, Larsson H, Wyver J (2008) ShapeShifting TV: interactive screen media narratives. Multimedia Systems 14(2):115–132

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank CNPq (National Council for Scientific and Technological Development), FAPERJ (Carlos Chagas Filho Research Support Foundation of the State of Rio de Janeiro) and FINEP (Brazilian Innovation Agency), which belong to the Ministry of Science, Technology and Innovation, for the financial support. Also we are grateful to the anonymous reviewers for their valuable comments.

Author information

Authors and Affiliations

  1. Department of Computational Modeling, Rio de Janeiro State University (UERJ), Nova Friburgo, RJ, Brazil

    Edirlei Soares de Lima

  2. Department of Informatics, Pontifical Catholic University of Rio de Janeiro, Rio de Janeiro, RJ, Brazil

    Bruno Feijó & Antonio L. Furtado

Authors
  1. Edirlei Soares de Lima
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bruno Feijó
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Antonio L. Furtado
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Edirlei Soares de Lima.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

de Lima, E.S., Feijó, B. & Furtado, A.L. Video-based interactive storytelling using real-time video compositing techniques. Multimed Tools Appl 77, 2333–2357 (2018). https://doi.org/10.1007/s11042-017-4423-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-4423-5

Keywords

Search

Navigation