In order to design safe and effective human-in-the-loop systems, developing robust and useful models of human behavior is absolutely vital. However, this problem is highly difficult to address, given that humans often act unpredictably. We investigate the problem of determining prediction sets for human-driven vehicles using Hamilton-Jacobi reachability analysis and empirical observations from driving datasets. Given evaluation metrics of accuracy, precision, and risk, we optimize disturbance bounds to construct forward reachable sets with high precision that satisfy accuracy and risk constraints. To demonstrate the approach, we apply our framework to a lane changing scenario to provide set predictions that provide safety guarantees without being over-conservative. We show an example of this method that allows us to construct a reachable set with over 85% accuracy and under 25% risk.