The majority of the existing object detection models struggle to fully exploit the intimate relationship between scene context and objects, and the feature fusion and proposal generation strategies tend to be relatively basic, resulting in poor model performance. To address these issues, we propose an object detection model based on adaptive multilevel fusion refinement network. First, we propose an adaptive gated fusion (AGF) network that dynamically assesses correlations between objects and scene information, generating a gated feature map to guide feature fusion and extract discriminative joint object-context features. Next, a proposal refinement model is proposed. By utilizing a learnable correlation-weighted coefficient, this model effectively merges low-level features with joint features, thereby mitigating spatial information deficits. We also propose an adaptive multidimensional offset (AMO) strategy, which minimizes the impact of regression deviations on proposal quality by combining information offsets and spatial offsets. To optimize all subtasks, a novel multitask loss function is proposed. Evaluated on the dataset for object detection in aerial images (DOTA) and HRSC2016 datasets shows that our method is superior to compared methods in object detection and harvests 77.26% and 90.68% mean average precision (AP), respectively.