To cope with the complex and constantly changing communication environment, Multi-Link Operation (MLO) has attracted extensive attention in research and development of next-generation Wi-Fi technology, IEEE 802.11be standard (Wi-Fi 7). MLO can transmit information simultaneously on different channels in the same device, which can significantly increase the capacity of Wi-Fi for future communication systems. Previous relevant studies have shown that network throughput is not simply and positively correlated with frame aggregation length. Furthermore, due to the variability of communication environments, the optimal frame aggregation length in the scheduling process is not unique within a given time, and the traditional methods are limited to solving non-convex optimization problems. To fill this gap, we present a novel approach using deep reinforcement learning (DRL) to tackle the optimization of frame aggregation lengths in 802.11be for multiple links. Our research offers a comprehensive depiction of the communication and interaction structure among multiple links and DRL, which helps drive the advancement of artificial intelligence (AI) solutions in future network designs and demonstrates the feasibility of exploiting DRL in next-generation wireless networks. Extensive simulation experiments show that the proposed method can achieve superior performance compared to the existing methods.