Affine-invariant extension of scale-invariant feature transform (ASIFT) algorithm requires a large amount of computation and memory access, and consequently, is hard to process in real time. In order to increase the operation speed of ASIFT algorithm, this paper proposes a new hardware architecture for the ASIFT algorithm. In order to reduce the memory access time, the affine transform is modified to allow external memory access in the raster-scan order with a little accuracy drop. In addition, image filtering with skewed kernel is proposed in order to reduce the memory space for image storage. Additional complexity reduction is attempted to reduce the number of simulated viewpoints. As a result, throughput of the affine transform module is increased to 325% and the proposed hardware processes a video graphics array-sized (640×480) video at 20 fps.