In this paper, we present a highly optimized one-chip digital audio decoder supporting both MP3 and AAC standards. The design has been focused on the hardware modularity for both MP3 and AAC. The hardware consists of a Huffman decoder module and 20-bit programmable DSP module. In addition, a hardware module for the prediction tool of AAC has been added to support AAC Main Profile. Utilizing the modularity of the designed hardware, MP3 and AAC decoding algorithms are distributed to each module in such a way that the system resource can be maximally utilized. The designed system can decode MPEG/Audio layer-3 bitstream using only 13.33 MIPS and MPEG-2 AAC Main Profile bitstream using only 16.9 MIPS while the high accuracy of the decoded output PCM can be obtained with it.