Early exit has been studied as a way to reduce the complex computation of convolutional neural networks. However, in order to determine whether to exit early in a conventional CNN accelerator, there is a problem that a unit for computing softmax layer having a large hardware overhead is required. To solve this problem, we propose a low cost early exit decision unit. The proposed architecture uses only fully-connected (FC) layer outputs to make early exit decisions, so the computation of the softmax layer is not necessary. Our implementation results show an energy reduction of 68% with an accuracy drop of less than 0.3%.