Hydronephrosis, dilation of the urinary collecting system, is one of the most common abnormalities detected in young children. Ultrasound is a routine imaging modality for detecting pediatric hydronephrosis. In this paper, we present a new deep learning-based approach to predict the hydronephrosis severity defined by presence or absence of obstruction or blockage at the junction of the kidney and ureter. First, we semi-automatically segmented the kidney to analyze its appearance that characterizes the obstruction. Then, we developed a deep learning-based model to predict the obstruction using each slice in the ultrasound image. Finally, we fused the information obtained from kidney slices to get the fmal risk score of obstruction. We evaluated the performance of this method on a dataset of volumetric ultrasound images acquired from 54 hydronephrotic kidneys. We obtained an average accuracy of 0.78 to identify clinically relevant obstructions associated with pediatric hydronephrosis.