A reliable model-based human motion tracking scheme is presented. In this approach, silhouette image sequences from a top-view camera and a side-view camera work together to track human motion in 3D space in real time. The adoption of one top view camera introduces many attractive characteristics. A convenient calibration scheme is presented by decoupling different camera parameters to the largest extent. Point features that are easy to extract and robust to noise are used to track human motion and resolve ambiguity reliably for challenging motions such as an orientation-free walking. In addition, the framework is able to perform quick and satisfactory pose estimation with a real-time optimization processing.