Convolutional Neural Networks (CNNs), a widely recognized deep learning algorithm, have been utilized in various domains such as smart cities and healthcare. However, the remarkable performance of CNNs is accompanied by high resource overhead and deployment complexity. To address these challenges, CNN compilers have been developed to simplify convolutional operations for edge device deployment. One of the crucial components in CNN models is the General Matrix Multiply (GEMM) operation, which serves as the main computational kernel. In previous studies, efforts were made to improve the computation speed of GEMM by modifying the matrix calculation sequence, but they did not fully exploit the computing resources of edge devices. In this paper, we propose a novel GEMM-based acceleration algorithm, named microGEMM. The microGEMM algorithm divides convolutional data to reduce the memory access times during the GEMM calculation process. Moreover, the algorithm employs instruction-level optimization in the GEMM calculation unit, decreasing the cache miss rate. To better evaluate the superiority of microGEMM on resource-constrained devices, two edge-oriented metrics are proposed, namely CCPS & CCPoE. The microGEMM algorithm is implemented in C++ and compared with the standard GEMM algorithm (naiveGEMM) and the GEMM of the open-source Basic Linear Algebra Subprograms (BLAS) library (openblasGEMM). The experimental results demonstrate that microGEMM achieves a significant speedup, ranging from 5.67 × to 14.19 ×, compared to naiveGEMM.