Natural human communication involves complex visual and audio behavior, and often context and joint interaction with the surrounding environment, to create a rich and satisfying experience. However, widely used virtual meeting systems such as WebEx and Skype still provide rather limited functionalities and hardly maintain the experience of an in-person meeting. In particular, traditional systems lack a sense of colocation and interaction as in a face-to-face meeting due to the separate displays of remote participants and poor integration with the shared collaborative contents. As a result, teleimmersive (TI) systems that aim to provide natural user experiences and interaction have attracted increasing research interest [1]. High-end telepresence products such as Cisco TelePresence or HP?s Halo were expressly designed to create the perception of meeting in the same physical space. But to achieve such an experience, these systems require a proprietary installation and high setup costs. Recently, some three-dimensional (3-D) TI systems have been developed to enhance remote collaboration by merging remote participants into the same 3-D virtual space [2]?[4]. However, these systems still fall short of simulating a face-to-face collaboration with the presence of shared contents. Also, the required bulky and expensive hardware with nontrivial calibration and setup hinders their wide adoption. With the wide availability of low-cost, commodity computing devices with embedded video cameras, microphones, and ubiquitous Internet access adequate for real-time media, the dream of high-quality TI communication should finally be within our reach.