Skip to main content
SpringerLink
Log in

A music-driven system for generating apparel display video

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Video plays a great important role in online apparel sales, which is a vital tool for publicity and to provide consumers with space of imagination. However, as the apparel market rapidly updates in large amounts every day, creating videos for fast increasing clothes can be challenging and labor-consuming. Considering this, we present ApVideor, a music-driven video generation system customized for displaying clothes. This system consists of two main modules: music recommendation module and audio-visual synthesis module. The former assists users in searching background music that matches the apparel style, while the latter combines the audio and visuals into a video by music-driven approaches. Our user study suggests that this system makes the video creation process significantly easier and faster than manual creation. Meanwhile, the viewer test suggests that apparel-displaying videos created using our system are of comparable quality to those created manually by people who have worked with video editing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. http://multichannelmerchant.com/mcm-outlook-2013-research-reports/

  2. ApVideor: A Video Automaker for Apparels

  3. https://www.djangoproject.com/

  4. https://deryy.github.io/ApvideorDemo/page.html

  5. https://pytorch.org/docs/stable/torchvision/models.html

  6. https://librosa.github.io

References

  1. Alpert JI, Alpert MI (1990) Music influences on mood and purchase intentions. Psychol Market 7(2):109–133

    Article  Google Scholar 

  2. Bu J, Tan S, Chen C, Wang C, Wu H, Zhang L, He X (2010) Music recommendation by unified hypergraph: combining social media information and music content. In: Proceedings of the 18th ACM international conference on Multimedia. ACM, pp 391–400

  3. Buhler J, Neumeyer D, Deemer R (2010) Hearing the movies: music and sound in film history. Oxford University Press, New York

  4. Caine K (2016) Local standards for sample size at chi. In: Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, pp 981–992

  5. Chao J, Wang H, Zhou W, Zhang W, Yu Y (2011) Tunesensor: a semantic-driven music recommendation service for digital photo albums. In: Proceedings of the 10th International Semantic Web Conference. ISWC2011

  6. Chen HC, Chen AL (2001) A music recommendation system based on music data grouping and user interests. In: Proceedings of the tenth international conference on Information and knowledge management. ACM, pp 231–238

  7. Costa YM, Oliveira LS, Silla CN Jr (2017) An evaluation of convolutional neural networks for music classification using spectrograms. Appl Soft Comput 52:28–38

    Article  Google Scholar 

  8. Defferrard M, Benzi K, Vandergheynst P, Bresson X (2016) Fma: a dataset for music analysis. arXiv:1612.01840

  9. Dias R, Fonseca MJ (2013) Improving music recommendation in session-based collaborative filtering by using temporal context. In: 2013 IEEE 25Th international conference on tools with artificial intelligence. IEEE, pp 783–788

  10. Goto M (2006) A chorus section detection method for musical audio signals and its application to a music listening station. IEEE Trans Audio Speech Lang Process 14 (5):1783–1794

    Article  Google Scholar 

  11. Harte C, Sandler M, Gasser M (2006) Detecting harmonic change in musical audio. In: Proceedings of the 1st ACM workshop on Audio and music computing multimedia. ACM, pp 21–26

  12. Hua XS, Lu L, Zhang H (2004) Automatic music video generation based on temporal pattern analysis. In: Proceedings of the 12th annual ACM international conference on Multimedia. ACM, pp 472–475

  13. Hua XS, Lu L, Zhang H (2004) Optimization-based automated home video editing system. IEEE Trans Circ Syst Video Technol 14(5):572–583

    Article  Google Scholar 

  14. Liao Z, Yu Y, Gong B, Cheng L (2015) Audeosynth: music-driven video montage. ACM Trans Graph (TOG) 34(4):68

    Article  Google Scholar 

  15. Lin JC, Wei WL, Wang HM (2016) Automatic music video generation based on emotion-oriented pseudo song prediction and matching. In: Proceedings of the 2016 ACM on Multimedia Conference. ACM, pp 372–376

  16. Lin JC, Wei WL, Wang HM (2016) Demv-matchmaker: emotional temporal course representation and deep similarity matching for automatic music video generation. In: 2016 IEEE international conference on Acoustics, speech and signal processing (ICASSP). IEEE, pp 2772–2776

  17. Murch W (2001) In the blink of an eye: A perspective on film editing. Silman-James Press

  18. Nam J, Tewfik AH (2005) Detection of gradual transitions in video sequences using b-spline interpolation. IEEE Trans Multimed 7(4):667–679

    Article  Google Scholar 

  19. Park CW, Young SM (1986) Consumer response to television commercials: The impact of involvement and background music on brand attitude formation. J Market Res 23(1):11–24

    Article  MathSciNet  Google Scholar 

  20. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  21. Strähle J, Hohls R (2018) In-Store Music in Fashion Stores. Springer, Singapore, pp 71–92. https://doi.org/10.1007/978-981-10-5637-6_5

    Book  Google Scholar 

  22. Wang J, Chng E, Xu C, Lu H, Tian Q (2007) Generation of personalized music sports video using multimodal cues. IEEE Trans Multimed 9(3):576–588

    Article  Google Scholar 

  23. Wang JC, Yang YH, Jhuo IH, Lin YY, Wang HM, et al. (2012) The acousticvisual emotion guassians model for automatic generation of music video. In: Proceedings of the 20th ACM international conference on Multimedia. ACM, pp 1379–1380

  24. Wang X, Wang Y (2014) Improving content-based and hybrid music recommendation using deep learning. In: Proceedings of the 22nd ACM international conference on Multimedia. ACM, pp 627–636

  25. Wu X, Qiao Y, Wang X, Tang X (2012) Bridging music and image: a preliminary study with multiple ranking cca learning. In: Proceedings of ACM Multimedia

  26. Yapriady B, Uitdenbogerd AL (2005) Combining demographic data with collaborative filtering for automatic music recommendation. In: International conference on knowledge-based and intelligent information and engineering systems. Springer, pp 201–207

  27. Zhang K, Zhang H, Li S, Yang C, Sun L (2018) The pmemo dataset for music emotion recognition. In: Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval. ACM, pp 135–142

Download references

Acknowledgements

This project is supported by the Key Project of National Science Foundation of Zhejiang Province (No. LZ19F020002), the National Natural Science Foundation of China (No. 61672451), Provincial Key Research and Development Plan of Zhejiang Province (No. 2019C03137). And the authors would like to thank the Key Laboratory of Design Intelligence and Digital Creativity of Zhejiang Province, Alibaba-Zhejiang University Joint Institute of Frontier Technologies and State Key Lab of CAD&CG.

Author information

Authors and Affiliations

  1. Laboratory of CAD, CG, Zhejiang University, Hangzhou, China

    Hui Zhang, Kejun Zhang, Yingping Cao, Jun Zheng & Xiaoyi Huang

  2. Alibaba NHCI Lab, Alibaba Group, Hangzhou, China

    Changyuan Yang

  3. International Design Institute, Zhejiang University, Hangzhou, China

    Lingyun Sun

Authors
  1. Hui Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kejun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yingping Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jun Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaoyi Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Changyuan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Lingyun Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kejun Zhang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, H., Zhang, K., Cao, Y. et al. A music-driven system for generating apparel display video. Multimed Tools Appl 79, 5649–5670 (2020). https://doi.org/10.1007/s11042-019-08435-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-08435-x

Keywords

Search

Navigation