We propose a fast algorithm for the room impulse response (RIR) simulation for a shoe-box shaped room with arbitrary reflection coefficients. Our basic contribution is a numerical procedure that removes the restrictions imposed on earlier results that use the IFFT algorithm to compute the RIRs for multiple observation points simultaneously. We furthermore demonstrate that the RIRs can be computed accurately from only a small number of so-called kernel coefficients, which significantly reduces the computational complexity of our approach. The simulation results show that the new method is highly accurate and orders of magnitude faster than the basic mirror image source method that forms the main alternative.