For mobile robot navigation in crowded environments, hand and body tracking to enable seamless human-robot interaction is a challenging problem. Many existing methods simplify the task with static camera assumptions, initial calibration stages, or ad hoc pose constraints, making them difficult to be applied to assistive robots used for healthcare applications. This paper introduces a method of hand-body association suitable for crowded environments, by incorporating depth cameras. A robust human hand and body detector, optimized for crowded environments, is first introduced. This is followed by a probabilistic framework for associating hands and bodies. Geodesic distances, based on depth information, are employed to isolate points local to a hand, regardless of their Euclidean proximity to points in other regions. This facilitates subsequent hand-body association based on a Bayesian framework with increased association robustness. The accuracy of the proposed method is evaluated using a range of parameters against an existing approach. A public dataset has been created to assess the method's practical value in crowded environments.