Cross-Modal Approach for Karaoke Artifacts Correction

Abstract

In this chapter, we combine adaptive sampling in conjunction with video analogies (VA) to correct the audio stream in the karaoke environment $\kappa= \left \{\kappa (t) :\ \kappa (t) = \left (U(t),\ K(t)\right ),\ t \in \left ({t}_{s},\ {t}_{e}\right )\right \}$ where t _s and t _e are start time and end time respectively, U(t) is the user multimedia data. We employ multiple streams from the karaoke data $K(t) = \left ({K}_{V }(t),\ {K}_{M}(t),\ {K}_{S}(t)\right )$, where K _V (t), K _M (t) and K _S (t) are the video, musical accompaniment and original singer’s rendition respectively along with the user multimedia data $U(t) = \left ({U}_{A}(t),{U}_{V }(t)\right )$ where U _V (t) is the user video captured with a camera and U _A (t) is the user’s rendition of the song. We analyze the audio and video streaming features $\Psi (\kappa ) = \left \{\Psi (U(t),\ K(t))\right \} = \left \{\Psi (U(t)),\ \Psi (K(t))\right \} = \left \{{\Psi }_{U}(t),\ {\Psi }_{K}(t)\right \}$, to produce the corrected singing, namely output U ^′(t), which is made as close as possible to the original singer’s rendition. Note that Ψ represents any kind of feature processing.