We propose a new target representation method, where the temporally obtained targets are jointly represented as a time series function by exploiting their spatially local structure. With this method, we propose a new tracking algorithm, where tracking is formulated as a problem of Gaussian process regression over the joint representation. Numerous experiments on various challenging video sequences demonstrate that our tracker outperforms several other state-of-the-art trackers.