Binaural sound is well known as a three dimensional sound which achieves a virtual reality of sound, or an augmented reality of sound. Binaural sound is characterized by ILD (Interaural Level Difference) and IPD (Interaural Phase Difference). In this paper, we propose a method to separate sound sources from an observed mixture signal, without losing respective ILD and IPD. The proposed method is established by improving a conventional sound source separation method based on single voice activity detection which detects segments including only single sound source. The proposed method estimates ILD and IPD from the single voice activity segments. After separating the sound sources by using the conventional method, we give the estimated ILD and IPD on the separated sound sources to refine the binaural characteristics. The effectiveness of the proposed method is clarified from estimation results of ILD and IPD for binaural sounds.