Skip to main content
SpringerLink
Log in

Example-based video remixing

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

A video remix is generally created by arranging selected video clips and combining them with other media streams such as audio clips and video transition effects. This paper proposes a system for semi-automatically creating video remixes of good expressive quality. Given multiple original video clips, audio clips, and transition effects as the input, the proposed system generates a video remix by five processes: I) video clip sequence generation, II) audio clip selection, III) audio boundary extraction, IV) video segment extraction, and V) transition effect selection, based on the spatial and temporal structural patterns automatically learned from professionally created video remix examples. Experiments using movie trailers of action genre as video remix examples not only demonstrate that video remixing by professionals can be imitated based on examples but also reveal that the video clip sequence generation and audio clip selection are the most important processes to improve the perceived expressive quality of video remixes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Aoyanagi S, Kourai K, Sato K, Takada T, Sugawara T (2003) Evaluation of new video skimming method using audio and video information. In: Proc. DEWS, 2-A-01

  2. Blairwitch.de, Horror Movie Entertainment. http://www.blairwitch.de/index.php?seitenid=1. Accessed 2008

  3. Bouguet J-Y (2000) Pyramidal implementation of the Lucas Kanade feature tracker. OpenCV Documentation, Microprocessor Research Labs, Intel Corporation

  4. Chen H-W, Kuo J-H, Chu W-T, Wu J-L (2004) Action movies segmentation and summarization based on tempo analysis. In: Proc. ACM SIGMM international workshop on multimedia information retrieval, pp 251–258

  5. Duda RO, Hart PE, Stork DG (2006) Pattern classification, 2nd edn. John Wiley & Sons, 2001. p 7307

  6. Dunker P, Nowak S, Begau A, Lanz C (2008) Content-based mood classification for photos and music: a generic multi-modal classification framework and evaluation approach. In: Proc. international conference on multimedia information retrieval, pp 97–104

  7. Foote J, Cooper M, Girgensohn A (2002) Creating music videos using automatic media analysis. In: Proc. ACM international conference on multimedia, pp 553–560

  8. Hua X-S, Lu L, Zhang H-J (2004) Automatic music video generation based on temporal pattern analysis. In: Proc. ACM international conference on multimedia, pp 472–475

  9. Hua X-S, Lu L, Zhang H-J (2004) Optimization-based automated home video editing system. IEEE Trans Circuits Syst Video Technol 14(5):572–583

    Article  Google Scholar 

  10. Jung B, Song J, Lee Y (2007) A narrative-based abstraction framework for story-oriented video. In: ACM transactions on multimedia computing, communications and applications, vol 3, no 2

  11. Kim J-G, Chang HS, Kim J, Kim H-M (2004) Threshold-based camera motion characterization of MPEG video. ETRI J 26(3):269–272

    Article  Google Scholar 

  12. Kumano M, Ariki Y, Shunto K, Tsukada K (2002) Video editing support system based on video content analysis. In: Proc. Asian conference on computer vision, pp 628–633

  13. Kurihara Y, Nitta N, Babaguchi N (2009) Automatic appropriate segment extraction from shots based on learning from example videos. In: Proc. Pacific-Rim symposium on image and video technology, pp 1082–1093

  14. Lee S-H, Yeh CH, Kuo C-CJ (2004) Video skimming based on story units via general tempo analysis. In: Proc. IEEE international conference on multimedia and expo, vol 2, pp 1099– 1102

  15. Li B, Pan H, Sezan I (2003) A general framework for sports video summarization with its application to soccer. In: Proc. IEEE international conference on acoustics, speech, and signal processing, pp 169–172

  16. Li T, Ogihara M, Li Q (2003) A comparative study on content-based music classification. In: Proc. international ACM SIGIR conference on research and development in information retrieval, pp 282–289

  17. Lienhart R (1999) Abstracting home video automatically. In: Proc. ACM international conference on multimedia, pp 37–40

  18. Liu Z, Wang Y, Chen T (1998) Audio feature extraction and analysis for scene segmentation and classification. J VLSI Signal Process Syst 20(1–2):pp 61–79

    Article  Google Scholar 

  19. Mckinney M, Breebaart J (2003) Features for audio and music classification. In: Proc. international symposium on music information retrieval, pp 151–158

  20. MixMeister BPM Analyzer. http://www.mixmeister.com/download_freestuff.html. Accessed 2008

  21. Mulhem P, Kankanhalli MS, Yi J, Hassan H (2003) Pivot vector space approach for audio-video mixing. IEEE Multimed 10(2):28–40

    Article  Google Scholar 

  22. Nitta N, Takahashi Y, Babaguchi N (2009) Automatic personalized video abstraction for sports videos using metadata. Multimed Tools Appl 41(1):1–25

    Article  Google Scholar 

  23. Rabiner LR (1989) A tutorial on Hidden Markov Models and selected applications in speech recognition. In: Proc. IEEE, vol 77, pp 257–285

  24. Rasheed Z, Sheikh Y, Shah M (2005) On the use of computable features for film classification. IEEE Trans Circuits Syst Video Technol 15(1):52–64

    Article  Google Scholar 

  25. Smeaton AF, Lehane B, O’Connor NE, Brady C, Craig G (2006) Automatically selecting shots for action movie trailers. In: Proc. ACM international workshop on multimedia information retrieval, pp 231–238

  26. Soundtracknet, The Art of Film and Television Music. http://www.soundtrack.net/. Accessed 2008

  27. Sundaram H, Chang S-F (2001) Condensing computable scenes using visual complexity and film syntax analysis. In: Proc. IEEE international conference on mutimedia and expo, pp 389–392

  28. Takemoto R, Yoshitaka A, Hirashima T (2006) Video editing based on movie effects by shot length transition. Technical Report of IEICE PRMU2005-149-183, pp 19–24

  29. Taskiran CM, Pizlo Z, Amir A, Poncelelon D, Delp EJ (2006) Automated video program summarization using speech transcripts. IEEE Trans Multimedia 8(4):775–791

    Article  Google Scholar 

  30. The International Movie Database. http://www.imdb.com/. Accessed 2005

  31. Tjondronegoro D, Chen Y-PP, Pham B (2003) Sports video summarization using highlights and play-breaks. In: Proc. ACM international workshop on multimedia information retrieval, pp 201–208

  32. Viola P, Jones M (2001) Rapid object detection using a boosting cascade of simple features. In: Proc. IEEE conference on computer vision and pattern recognition, vol 1, pp 511–518

  33. Wang J, Xu C, Chung E, Duan L, Wan K, Tian Q (2005) Automatic generation of personalized music sports video. In: Proc. ACM international conference on multimedia, pp 735–744

  34. Yoon J-C, Lee I-K, Byun S (2009) Automated music video generation using multi-level feature-based segmentation. Multimed Tools Appl 41(2):197–214

    Article  Google Scholar 

  35. Zettl H (1999) Sight sound motion: applied media aesthetics, 3rd edn. Wadsworth Publishing

Download references

Acknowledgement

This research was partially supported by a Grant-in-Aid for Young Scientists (B) 20700087 from JSPS and by Core Project from Microsoft Institute for Japanese Academic Research Collaboration. The authors especially thank Jang-il Kim, Yosuke Kurihara, and Guozhen Jiang for their contributions to the development of the system.

Author information

Authors and Affiliations

  1. Graduate School of Engineering, Osaka University, 2-1 Yamada-Oka, Suita, Osaka, 565-0871, Japan

    Naoko Nitta & Noboru Babaguchi

Authors
  1. Naoko Nitta
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Noboru Babaguchi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Naoko Nitta.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Nitta, N., Babaguchi, N. Example-based video remixing. Multimed Tools Appl 51, 649–673 (2011). https://doi.org/10.1007/s11042-010-0633-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-010-0633-9

Keywords

Search

Navigation