Through the development of pattern recognition methods, many researchers have proposed various systems to control a TV that can be activated automatically without devices. In this paper, we propose a control owner switching system of TV via multi-user face recognition and head gesture recognition. We mainly focus on rotation invariant and wide-range (up to four meters) multi-user face recognition so that the system can be applied to indoor environments. In addition, we propose a deviceless TV control owner switching system using a small behavior of the user. As a method of switching, we suggest switching the TV control ownership by shaking the user's head from left to right for a few times. Experimental result shows that switching system is perfect for transferring control ownership.