This paper discusses Independent Vector Analysis (IVA)-based moving source separation under a source tracking scenario where at most one speaker moves. Previous research proposed a computationally efficient algorithm to update filters for source separation under a situation where only one speaker is moving. To make this algorithm practical, we need to optimize the filters before one speaker starts moving and select which filter corresponds to the moving speaker. Thus, we applied Geometric Constraint (GC) to increase the filter’s convergence speed and select the filter automatically. Compared with research that applies GC all the time, our contribution is to apply GC only when all speakers are fixed, which results in lower computational complexity while maintaining its separation performance for our scenario.