Highly accurate sensor extrinsic calibration is critical for data fusion from multiple sensors, such as camera and Inertial Measurement Unit (IMU) sensor suit. A pre-calibrated extrinsics, however, may no longer be accurate due to external disturbances, e.g., vehicle vibration, which will lead to significant performance deterioration of autonomous vehicles. Existing approaches rely on online recalibration at a fixed frequency regardless of whether the extrinsics have actually been changed or recalibration is needed, which is computationally inefficient. In this letter, we present an approach to monitor extrinsic calibration quality for camera-IMU sensor suite to determine when recalibration is actually necessary. We propose an efficient algorithm to detect robust road image features, utilize IMU data to capture the mismatches of those features, and quantify extrinsic calibration error through three commonly-used error metrics. Our algorithm is demonstrated to be effective using both simulated and real-world data.