This paper addresses multi-agent multi-object tracking with labeled random finite sets via Generalized Covariance Intersection (GCI) fusion. While standard GCI fusion of Labeled Multi-Object (LMO) densities is labelwise and hence fully parallelizable, previous work unfortunately revealed that its fusion performance is highly sensitive to the unavoidable label inconsistencies among different agents. In order to overcome the label inconsistency sensitivity problem, we present a novel approach for the GCI fusion of LMO densities that is both robust to label inconsistencies and computationally efficient. The novel approach consists of, first, finding the best matching between labels of different agents by minimization of a suitable label inconsistency indicator, and, then, performing GCI fusion labelwise according to the obtained label matching. Furthermore, it is shown how the label matching problem, which is at the core of the proposed method, can be formulated as a linear assignment problem of finite length (efficiently solvable in polynomial time by the Hungarian algorithm), exactly for Labeled Multi-Bernoulli densities and approximately for arbitrary LMO densities. Simulation experiments are carried out to demonstrate the robustness and effectiveness of the proposed approach in challenging tracking scenarios.