We propose a novel middle level estimation of traffic scenes: Collision Risk Rating (CRR). Given a video sequence from a dashboard camera as input, the objective is to estimate a rate that describes "how likely a collision could happen". CRR's problem setting is similar to that of video classification, but it is more complicated and requires rich feature representations to capture the different aspects of the global scene, as well as additional mechanisms to take care of the internal relationships between rating labels. We propose an approach for CRR that features with: (1) a video representation that computed by learned Two-Stream CNNs; and (2) a rank learning based approach for predicting rating labels. The approach was evaluated on a novel prepared Traffic Collision Dataset, and was confirmed to be superior to state-of- the-art video classification approaches.