The Coronavirus disease 2019 (COVID-19) outbreak continues to significantly expose the vulnerabilities of healthcare systems around the world. These unprecedented circumstances create an opportunity for improving healthcare services which is desperately needed. This paper proposes a novel framework that distributes the patients across heterogeneous medical facilities (MFs) so that a weighted sum of the expected service time (EST) and service time tail probability (STTP) for all patients is minimized. We propose a model-based and model-free algorithms to schedule patients requests across the MFs. Our algorithms prioritize the patients with severe/critical conditions over others who can tolerate more delay in service. Based on the model-based approach, we formulate an optimization problem as a convex combination of both EST and STTP metrics, and apply an efficient iterative algorithm to solve it. Then, a more practical model-free scheme is proposed by adopting a deep reinforcement learning approach. Our model-free approach does not rely on pre-defined models or assumptions about the environment. Rather, it learns to choose scheduling decisions solely through observations of the resulting performance of past decisions. Our extensive results demonstrate a significant performance improvement of our proposed scheduling schemes when compared with other algorithms and competitive baselines.