skip to main content
10.1145/3581783.3612674acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
demonstration

HoloSinger: Semantics and Music Driven Motion Generation with Octahedral Holographic Projection

Published: 27 October 2023 Publication History

Abstract

Lyrics and music are both significant for a singer to perform a song. Therefore, it is important in singer's motion generation to model both semantic and acoustic correlation with motions at the same time. In this paper, we propose HoloSinger, a novel comprehensive system that synthesizes singing motions according to the given song. Additionally, we present singing avatar with octahedral holographic projection. For singing motion generation, we introduce a Transformer-VAE generative model to decompose lyrics and music, then fuse their impacts to synthesize singer's motions. Extensive experiments and user studies show that our method automatically generates realistic motions that adhere to musical choreography and reflect the lyric semantics appropriately. Furthermore, we design a desktop-level holographic projection device with an octahedral structure. It achieves high-definition holographic projection effects with smaller volume, larger imaging area ratio, and the ability of real-time AI interaction.

References

[1]
Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. 2021. Ai choreographer: Music conditioned 3d dance generation with aist. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 13401--13412.
[2]
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, Vol. 2017. 498--502.
[3]
Abhinanda R Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and Michael J Black. 2021. BABEL: bodies, action and behavior with English labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 722--731.
[4]
Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. 2022. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11050--11059.

Cited By

View all
  • (2024)Multi-Instance Multi-Label Learning for Text-motion RetrievalProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681444(5829-5837)Online publication date: 28-Oct-2024
  • (2024)Realistic Full-Body Motion Generation from Sparse Tracking with State Space ModelProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681013(4024-4033)Online publication date: 28-Oct-2024
  • (2024)DanceCamAnimator: Keyframe-Based Controllable 3D Dance Camera SynthesisProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680980(10200-10209)Online publication date: 28-Oct-2024

Index Terms

  1. HoloSinger: Semantics and Music Driven Motion Generation with Octahedral Holographic Projection

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '23: Proceedings of the 31st ACM International Conference on Multimedia
    October 2023
    9913 pages
    ISBN:9798400701085
    DOI:10.1145/3581783
    Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 27 October 2023

    Check for updates

    Author Tags

    1. dance synthesis
    2. holographic projection system
    3. text2motion

    Qualifiers

    • Demonstration

    Conference

    MM '23
    Sponsor:
    MM '23: The 31st ACM International Conference on Multimedia
    October 29 - November 3, 2023
    Ottawa ON, Canada

    Acceptance Rates

    Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)85
    • Downloads (Last 6 weeks)5
    Reflects downloads up to 15 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Multi-Instance Multi-Label Learning for Text-motion RetrievalProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681444(5829-5837)Online publication date: 28-Oct-2024
    • (2024)Realistic Full-Body Motion Generation from Sparse Tracking with State Space ModelProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681013(4024-4033)Online publication date: 28-Oct-2024
    • (2024)DanceCamAnimator: Keyframe-Based Controllable 3D Dance Camera SynthesisProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680980(10200-10209)Online publication date: 28-Oct-2024

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media