Cardiac Auscultation, an integral part of the physical examination of a patient, is essential for early diagnosis of cardiovascular diseases (CVDs). The ability to accurately diagnose the heart sounds requires experience and expertise, which is lacking in doctors in the early years of clinical practice. Thus, there is a need for an automatic diagnostic tool that would aid medical practitioners with their diagnosis. We propose novel hybrid architectures for classification of unsegmented heart sounds to normal and abnormal classes. We propose two methods, with and without the conventional feature extraction step in the classification pipeline. We demonstrate that the F score using the approach with conventional feature extraction is 1.25 (absolute) more than using a baseline implementation on the Physionet dataset. We also introduce a mechanism to tag predictions as unsure and compare results with a varying threshold.