We present a new semantic segmentation algorithm for obstacle detection in unmanned surface vehicles. The novelty lies in the graphical model that incorporates boat tilt measurements from the on-board inertial measurement unit (IMU). The IMU readings are used to estimate the location of horizon line in the image, and automatically adjusts the priors in the probabilistic semantic segmentation algorithm. We derive the necessary horizon projection equations, an efficient optimization algorithm for the proposed graphical model, and a practical IMU-camera-USV calibration. A new challenging dataset, which is the largest multi-sensor dataset of its kind, is constructed. Results show that the proposed algorithm significantly outperforms state of the art, with 32% improvement in water-edge detection accuracy, an over 15 % reduction of false positive rate, an over 70 % reduction of false negative rate, and an over 55 % increase of true positive rate, while running in real-time on a single core in Matlab.