With the popularity of smartwatches, users can access private information stored in the device by simply touching the watch screen. However, smartwatches also expose users to the risk of information leakage because they lack proper authentication schemes. This paper proposes PalmEcho, a multimodal authentication scheme for smartwatches. The basic idea of PalmEcho is to capture the vibration and the sound generated from user’s beating gestures, and then fuse multimodal signals to extract unique features for user authentication. However, signals of beating gestures are short in the time domain, which makes it hard to extract effective features. To address this challenge, our work reveals that the spectral energy distribution of the generated sound is unique to each user and provides rich information for authentication. Moreover, conventional classification networks require large amounts of user data for training, which is not convenient in the user authentication scenario. To address this challenge, we design a prototypical network called BeatNet, which allows users to register with a few samples. Experimental results show that PalmEcho can reach an average F1-score of 94%.