We describe a flexible and efficient architecture for generic object recognition system based on ensemble classifier in a field programmable gate array (FPGA) environment. We have shown previously utilizing a bag of covariance matrices as object descriptor improves the object recognition accuracy while speed up the learning process. We extend this technique, and present its hardware architecture, as well as object classifier based on on-line variant of random forest (RF) implemented using logarithmic number system (LNS). First, we describe the algorithmic and architecture of our model, comprises several computation modules. Then test and verified the model functionality using numerical simulation. Utilizing examples from GRAZ02 dataset it has been shown that the proposed system gained strong recognition performance over the floating-point and fixed-point precision, even when only 10% training examples are used and is reasonably power efficient.