Neural Ordinary Differential Equations (ODEs) have gained traction in many applications. While recent studies have focused on empirically increasing the robustness of neural ODEs against natural or adversarial attacks, certified robustness is still lacking. In this letter, we propose a framework for training a neural ODE using barrier functions and demonstrate improved robustness for classification problems. We further provide the first generalization guarantee of robustness against adversarial attacks using a wait-and-judge scenario approach.