In this paper, we propose memory and performance optimized architecture to accelerate the operation speed of adaptive deblocking filter for H.264/JVT/AVC video coding. The proposed deblocking filter executes loading/storing and filtering operations with only 192 cycles for 1 macroblock. Only 2×4×4 internal buffers and 32×16 internal SRAM are adopted for the buffering operation of deblocking filter with I/O bandwidth of 32 bit. The proposed architecture can process the filtering operation for 1 macroblock with less filtering cycles and lower memory sizes than some conventional approaches of realizing deblocking filter. The efficient hardware architecture is implemented with novel data arrangement, hybrid filter scheduling and minimum number of buffer. The proposed architecture is suitable for low cost and real-time applications, and the real-time decoding with 1080HD (1920×1088@30fps) can be easily achieved when working frequency is 70MHz.