This paper describes a Bayesian network for 3D articulated upper body segmentation and tracking from video sequences for which both color and depth information are available. In our upper body model the joints are represented as the parent nodes of the body components nodes which include the head, torso or arms. The upper body components are modeled using a set of planar, linear and Gaussian density functions. The model described in this paper segments and tracks accurately the upper body in different illumination conditions and in the presence of partial occlusions and self occlusions. In addition the current approach allows for automatic segmentation of the upper body without any human intervention allowing for further use of the system in hand gesture or human activity recognition.