We propose a novel formulation to use plane primitives in the direct sparse odometry formulation. Unlike existing SLAM works that introduce geometric error terms to account for planes, our method keeps the full error as purely photometric. The proposed system exploits the segmentation masks and plane parameters from a deep neural network and jointly optimizes the camera poses, 3D points, and 3D planes. The tightly-coupled formulation improves the pose estimation by fusing information on high-level geometric primitives and also refines the outputs of the plane detection network during optimization.