Recognition of environmental sounds (ES) is a challenging problem due to the unstructured nature and typically noise-like and flat spectrums of these sounds. In the paper, we propose a composite audio feature to capture the different characteristics of ESs by combining spectral and harmonic audio features. In the experiments, thirteen (13) ES categories, namely emergency alarm, car horn, gun, explosion, automobile, motorcycle, helicopter, water, wind, rain, applause, crowd, and laughter are detected based on the proposed feature set and by using the SVM classifier. Extensive experiments have been conducted to demonstrate the effectiveness of the proposed joint feature set for ES classification. Our experiments show that, the proposed feature set ASFCS-H (Audio Spectrum Flatness, Centroid, Spread, and Audio Harmonicity) is quite successful in recognition of ESs with an average F-measure value of 80.6%.