We propose the clustering re-inference algorithm (CRIA) for a deep learning-based hierarchical object detection system. CRIA uses a two-stage inference mechanism, namely a global detection unit (GDU) and a local detection unit (LDU), to detect large and small objects in aerial images, respectively, and a normalized clustering unit (NCU) to locate small objects. The input high-resolution aerial images are compressed and then inferred coarsely by the GDU to detect objects. This result is input to the NCU, which estimates the location of small objects by normalizing the vectors of small object candidates obtained from the GDU and clustering them in four dimensions. Finally, the LDU performs adaptive fine inference on the region estimated by the NCU to detect small objects. The results of an evaluation using the VisDrone dataset, which mainly consists of Full-HD aerial images, show that the proposed method improves the number of successfully detected small objects by 11.6% and the detection accuracy by 20.1% compared with the results obtained using the conventional method.