Binary neural networks (BNNs) largely reduce the memory footprint and computational complexity, so they are gaining interests on various mobile applications. In the BNNs, the first layer often accounts for the largest part of the entire computing time because the layer usually uses multi-bit multiplications. However, traditional hardware designed for BNN computing focuses primarily on the rest layers, resulting in significant performance degradation. In this brief, we introduce Binaryware architecture which achieves the high-performance computation on both the first and rest layers. Experimental results show that our Binaryware improves the throughput per compute area by 1.5– $13.3\times$ on various BNN workloads.