The ultralow delay tracking system, which enables seamless actuation in visual feedback applications, draws increasing attention in the fields of robotics and factory automation (FA). Tracking accuracy and rotational robustness are critical in robotics and FA applications. In the real-time scene, the change (e.g., object moving) continues in the real space when tracking processing is performed in the virtual (computing) space. However, existing research focuses primarily on the image-processing accuracy of still images, with few works paying attention to tracking errors introduced by the change that occurs during the processing period. However, making the algorithm robust to rotation comes with a complex model, preventing the whole system from reaching an ultralow delay. This research aims to develop a 1-ms rotation-robust Lucas Kanade (LK)-based tracking algorithm and processing architecture, focusing on minimizing the error between the virtual space’s output and the true state in real space. To achieve the above target, this article proposes: 1) temporal prediction-based temporal iterative tracking (TIT), which compensates for temporal change with temporal prediction, significantly reducing real-time processing error and 2) temporal prediction-based parallel motion estimation, which breaks the data dependency of translation and rotation estimations by temporal prediction, allowing translation and rotation estimation to be performed parallelly with low delay. The proposed methods are implemented as a practical system by integrating a high-speed camera and a field programmable gate array (FPGA). Algorithm evaluation shows that the proposed method outperforms the other five related methods in terms of real-time tracking performance. Hardware evaluation shows that the designed rotation-robust LK-based tracking system supports sensing and processing a 1000-fps sequence with a delay of less than 1 ms/frame while costing resources less than 30%. A video demonstration is ...