The Open Radio Access Network (RAN) architecture has introduced new elements in the RAN, i.e., the RAN Intelligent Controllers (RICs), which allow for closed-loop control of the physical infrastructure through custom, data-driven, intelligent applications. At the near-real-time RIC, xApps access RAN nodes through the E2 interface and offer the option to host traditional algorithms or data-driven Deep Reinforcement Learning (DRL) agents to control and optimize RAN functionalities. The O-RAN specifications suggest that Artificial Intelligence (AI) model training should not be done on production RAN deployments to avoid network disruptions and degradation in the Quality of Experience (QoE) of the User Equipments (UEs), suggesting the adoption of offline reinforcement learning. However, this approach limits the exploration phase during training to the static data that has already been collected, potentially affecting the performance of the model and its generalization capabilities. Therefore, a safe environment capable of supporting online reinforcement learning is needed to overcome such constraints and to allow AI agents to perform state explorations freely. In this paper, we present a new control environment based on Gymnasium (gym), a Python library for the creation of reinforcement learning environments, and ns-O-RAN, a software integration between a real-world near-real-time RIC and an Network Simulator 3 (ns-3) simulated RAN, which exposes the RAN Key Performance Indicators (KPIs) through a standardized Application Programming Interface (API) ready to be used by any solving approach. Leveraging ns-O-RAN, we create an environment that dynamically captures the simulated O-RAN telemetry, waits for the agent to compute a decision, receives and delivers such control action to update the RAN configuration in the underlying simulation, allowing the development and test of models in safe and reproducible conditions. Finally, our framework exposes an abstract API interfac...