Recently, networks operate at frequencies over 28 GHz (mmWave) have emerged as a viable solution for 5G mobile networks to provide Gbps data rate. Due to the high directivity and attenuation of mmWave signals, mmWave communication links are highly vulnerable to the frequent mmWave channel blockages, which can trigger excessive handovers. Thanks to its ability to enrich the scattering environment and create reflective signal multipaths, Reconfigurable Intelligent Surface (RIS) has great potential to counter the blockage effect and thus greatly reduce the number of unnecessary handovers. However, this potential has not been well explored. In this paper, we propose a RIS-assisted handover scheme by leveraging deep reinforcement learning (DRL). Under various channel blockage conditions, the DRL agent manages to reduce the cumulative handover overhead by jointly adjusting beamformers and RIS phase shifts. Compared with the existing schemes without considering RIS, the RIS-assisted handover scheme significantly reduces the number of handovers and achieves higher spectrum efficiency. Besides, to alleviate the impact from the limited observations of the fast fading channels, we propose a lightweight algorithm to sense the blockage status and such sensing results can be utilized to improve the performance of model training. Numerical results show that DRL agent is able to further improve the performance when integrated with the blockage status sensing algorithm.