Despite extensive research on optimal control formulations for cyber threat mitigation, a significant gap persists between theoretical and practical implementation in real-time scenarios. The open-loop structure of the optimal control framework is insufficiently robust for effectively addressing cyber threats. To overcome this, adopting a model learning process that iteratively updates the optimal control strategy is proposed. This paper proposes an innovative approach to addressing cybersecurity attacks in the Internet of Things (IoT) networks by integrating reinforcement learning (RL) and model predictive control (MPC) in a hybrid framework to optimize control parameters and enhance system effectiveness in combating malware. This novel approach aims to overcome the limitations of the previous approaches and establish superior control strategies for IoT network security. This approach enhances the adaptability and responsiveness of the mitigation process, improving the handling of evolving cyber threats in real-world applications. This framework enhances the security and resilience of IoT networks against malicious activities, offering a robust solution for mitigating cyber threats by leveraging RL algorithms and the proactive capabilities of MPC. A comprehensive evaluation demonstrates the effectiveness and efficiency of the hybrid framework, highlighting its potential to protect IoT networks from evolving cybersecurity risks. The primary aim extends beyond using an RL agent solely for computing control actions to optimize closed-loop performance and stability. It also leverages RL to estimate model parameters that are currently unknown but within known bounds. Our main objective in using the RL agent is to accurately estimate unidentified model parameters within specified limits. The simulation results provide compelling evidence supporting the effectiveness of this methodology in mitigating malware propagation, highlighting its superior performance compared to s...