The blossoming auto-driving technology brings not only great convenience to our daily life but also a great safety challenge on the control transition from vehicles to drivers, i.e., takeover. In this article, we first review state-of-the-art research on this topic and propose a general takeover safety framework consisting of three stages: sensing, predicting and responding. Our comparative study reveals that current approaches mainly focus on exploring drivers' physical states to predict how they perform in takeovers. On the other hand, drivers' emotions, another significant factor affecting human behavioral reactions while driving, remain largely uncharted. We then introduce an emotion-aware takeover performance prediction system, AffecTake, that collects and analyzes both drivers' emotional and physical data for takeover safety prediction via deep learning. We then demonstrate the feasibility of AffecTake with a prototype built on readily available auto-driving software and hardware. Preliminary experiments have confirmed that emotions indeed help predict drivers' takeover performance. Lastly, we present some discussions and potential future extensions of the proposed system.