Patient studies based on diffusion tensor images (DTI) require spatial correspondence between subjects. We propose to obtain the correspondence from white matter tracts, by introducing a new method for nonrigid matching of white matter fiber tracts in DTI. The method boils down to point set registration that involves simultaneously clustering and matching of the data points. The tracts are implicitly warped to a common frame of reference to avoid the potential bias toward one of the datasets. The algorithm gradually refines from global to local registration, which is implemented through deterministic annealing. Special care was taken to incorporate the spatial relation between fiber points and the uncertainty in principal diffusion orientation. As a result, the computed clusters are oriented along the fiber tracts and discriminate between adjacent but distinct fiber tracts. This is validated on synthetic and clinical data. The root-mean-squared distance with respect to expert-annotated landmarks is low (3 mm). In contrast to a state-of-the-art nonrigid registration technique, the proposed method is more robust to residual misalignments in terms of measured fractional anisotropy values.