Autonomous vehicles must be comprehensively tested before being deployed in the real world. Simulators offer the possibility of safe, low-cost development of self-driving systems. In addition to environmental perception, motion planners are particularly important to provide robust and safe driving trajectories. However, evaluating and improving motion planning algorithms for autonomous vehicles requires scalable generation of traffic scenarios. To be useful, these scenarios must be realistic and challenging, but also remain possible to traverse safely. In this work, we introduce a reinforcement learning (RL) approach for automatically generating adversarial trajectories to challenge motion planning algorithms. Given a motion planner, we train our deep RL agent to search the risky trajectory space of an adversarial vehicle. Actor-critic RL optimizes the entire process. Through experiments conducted on multiple routes within a selected intersection scenario, we demonstrate the effectiveness of our proposed framework in generating adversarial trajectories that minimize state-of-the-art scenario criticality measures.