Network slicing plays a critical role in enabling multiple virtualized and independent network services to be created on top of a common physical network infrastructure. In this paper, we introduce a deep reinforcement learning (DRL)-based radio resource management (RRM) solution for radio access network (RAN) slicing under service-level agreement (SLA) guarantees. The objective of this solution is to minimize the SLA violation. Our method is designed with a two-level scheduling structure that works seamlessly under Open Radio Access Network (O-RAN) architecture. Specifically, at an upper level, a DRL-based inter-slice scheduler is working on a coarse time granularity to allocate resources to network slices. And at a lower level, an existing intra-slice scheduler such as proportional fair (PF) is working on a fine time granularity to allocate slice dedicated resources to slice users. This setting makes our solution O-RAN compliant and ready to be deployed as an ‘xApp’ on the RAN Intelligent Controller (RIC). For performance evaluation and proof of concept purposes, we develop two platforms, one industry-level simulator and one O-RAN compliant testbed; evaluation on both platforms demonstrates our solution’s superior performance over conventional methods.