The time-varying deviations of the phase shifter network (PSN) may cause significant performance degradation in the millimeter-wave (mmWave) multiuser communication system, especially when serving fast-moving users. In this letter, we propose an efficient joint real-time phase shifter network calibration and beam tracking (JRTPSNC-BT) algorithm. In each tracking period, the exploration strategy is determined according to the previously tracked beam directions. Then based on the latest observation and the historical observations, the time-varying channels and the time-varying PSN deviations are jointly tracked and updated. Simulation results show the superiority of our algorithm both in calibration error and spectral efficiency.