The extensive use of artificial intelligence (AI) in the real world brings some potential risks due to the undesirable behavior exhibited by AI systems using data-driven machine learning (ML) at their cores. Thus, preventing undesirable behaviors of ML, such as opacity (lack of transparency and explainability), unfairness (bias or discrimination), unsafety and insecurity, privacy disclosure, etc., is an imperative and pressing challenge. This work proposes an evolutionary constrained learning (ECL) framework for constructing ML models that can satisfy behavioral constraints so that the undesirable behaviors can be prevented. To evaluate our framework, we use it to create neural network models that preclude the undesirable behavior (that is, unfairness) on different benchmark datasets. The experimental results demonstrate the effectiveness of our proposed ECL approach for preventing undesirable behaviors of ML.