In this paper, we present an efficient moving object segmentation method using motion orientation histogram (MOH) in consideration of variable block-based hardware implementation. In pursuit of both efficiency and reliability, each block motion vector is quantized into one of eight representative orientations. Given a set of motion vectors estimated from regularly divided basic blocks, we adaptively partition the blocks based on entropy for increasing the reliability of estimated motions. We then compute motion orientation histogram (MOH) from appropriately partitioned blocks and quantize them into eight representative orientations. Finally, we decide the object's motion using the quantized orientation of motion and error compensation. Experimental results show that the proposed method can be embedded in an image signal processing (ISP) chip for high-level image processing functions such as object tracking and behavior analysis in consumer surveillance systems.