Early diagnosis of cardiovascular diseases is essential for an effective treatment, potentially preventing severe health complications and improving clinical outcomes. Electrocardiogram (ECG) and phonocardiogram (PCG) are cost-effective, noninvasive diagnostic tools providing crucial and complementary information about the heart’s electrical and mechanical activities. This paper presents a novel approach to the assessment of cardiovascular health through the multimodal analysis of simultaneously recorded ECG and PCG signals. Combining multimodal analysis and transfer learning on publicly available data, the most successful multimodal approach achieved an accuracy of 82.79%, a ROC AUC score of 91.26%, and a recall of 93.10% demonstrating the potential of these techniques. This study provides a foundation for future research aimed at enhancing the performance of multimodal cardiac abnormality detection systems.