New multimedia services over cellular/WLAN overlay networks require different Quality of Service (QoS) levels. Therefore, an efficient network management system is necessary in order to realize QoS sensitive multimedia services while enhancing network performance. In this paper, we propose a new online network management framework for overlay networks. Our online approach to network management exhibits dynamic adaptability, flexibility, and responsiveness to the traffic conditions in multimedia networks. Simulation results indicate that our proposed framework can strike the appropriate balance between performance criteria under widely varying diverse traffic loads.