This paper proposes a hybrid message-passing decoding algorithm which consumes very low computational complexity, while achieving competitive error performance compared with conventional min-max algorithm. Simulation result on a (255,175) cyclic code shows that this algorithm obtains at least 0.5dB coding gain over other state-of-the-art low-complexity non-binary LDPC (NB-LDPC) decoding algorithms. Based on this algorithm, a partial-parallel decoder architecture is implemented for cyclic NB-LDPC codes, where the variable node units are redesigned and the routing network is optimized for the proposed algorithm. Synthesis results demonstrate that about 24.3% gates and 12% memories can be saved over previous works.