We propose a novel error resilient video coding and streaming scheme specially designed for the streaming of H.264 coded video to portable devices in this work. The scheme is based on the error feedback and tracking techniques, and it utilizes the SP/SI coding method introduced in H.264. During the video streaming process, the sender identifies macroblocks that have been affected by previous errors based on the feedback information from the receiver. Then, it re-encodes these macroblocks into secondary SP macroblocks with predictions from concealed reference frames and uses them to replace the originally coded version in the bit stream. As proven by experimental results, the scheme is capable of stopping or reducing the error propagation effect caused by transmission errors. It demands a small bit rate overhead if there is a transmission error, and no bit rate overhead, otherwise. Experimental results are provided to demonstrate the error resilient performance of the proposed scheme.