Multi-task learning has emerged as a significant topic in artificial intelligence research, where a singular network model performs numerous tasks. This approach simultaneously processes multiple related tasks and shared knowledge, enhancing model generalization while increasing efficiency. This methodology provides innovative solutions to complex real-world problems. However, the single-model-based approach for multi-task learning suffers from inter-task interference in practice. Therefore, the exploration of more efficient multi-task learning strategies, aimed at balancing the synergy and conflict among tasks and minimizing the dependence on computational resources, is pivotal for the field's future progress. We propose a strategy that autonomously adjusts both the parameters and structures of the model to alleviate gradient interference in multi-task learning. This method includes integrating a lightweight, weight-adaptive module that enhances the network's ability to process tasks by optimally balancing parameter sharing and isolation. This adaptation enables the model to share common features across tasks while allocating distinct spaces for each task, thereby reducing interference. Our extensive experimental validation indicates that our framework surpasses other multi-task learning approaches, achieving joint optimization of tasks more effectively. This enhancement not only bolsters performance but also maintains an equilibrium between accuracy and inference speed.