Stress perfusion cardiac magnetic resonance is an important technique for examining and assessing the blood supply of the myocardium. Currently, the majority of clinical perfusion scans are evaluated based on visual assessment by experienced clinicians. This makes the process subjective, and to this end, quantitative methods have been proposed to offer a more user-independent assessment of perfusion. These methods, however, rely on time-consuming deconvolution analysis and are susceptible to data outliers caused by artifacts due to cardiac or respiratory motion. In our work, we introduce a novel deep-learning method that integrates the commonly used Fermi function with a neural network architecture for fast, accurate, and robust myocardial perfusion quantification. This approach employs the Fermi model to ensure that the perfusion maps are consistent with measured data, while also utilizing a prior based on a 3D convolutional neural network to generalize spatio-temporal information across different patient data. Our network is trained within a self-supervised learning framework, which circumvents the need for ground-truth perfusion labels that are challenging to obtain. Furthermore, we extended this training methodology by adopting a technique that ensures estimations are resistant to data outliers, thereby improving robustness against motion artifacts. Our simulation experiments demonstrated an overall improvement in the accuracy and robustness of perfusion parameter estimation, consistently outperforming traditional deconvolution analysis algorithms across varying Signal-to-Noise Ratio scenarios in the presence of data outliers. For the in vivo studies, our method generated robust perfusion estimates that aligned with clinical diagnoses, while being approximately five times faster than conventional algorithms.