Memory reference in digital signal processors (DSP) is among the most costly operations due to its long latency and substantial power consumption. In this paper, we present a new method to minimize memory references due to twiddle factors for implementing any existing fast Fourier transform (FFT) algorithms on DSP processors. The new method takes advantage of previously proposed twiddle factor reduction method (TFRM) and twiddle-factor-based butterfly grouping method (TFBBGM). It can compute two butterflies in one stage of any FFT diagram by loading only one twiddle-factor. Further memory reference reduction is done by computing butterflies with the same twiddle factor at the same time in different stages of the FFT diagram. We have applied the new method to implement radix-2 DIT FFT algorithm on TI TMS320C64x DSP. While using only 50% memory space for storing twiddle factors compared to the conventional DIT FFT implementation, the new method achieves an average reduction in the number of memory references by 79% for accessing the twiddle factors, and 17.5% reduction in the number of clock cycles.