Each year, multiple catastrophic events impact vulnerable populations around the planet. Assessing the damage caused by these events in a timely and accurate manner is crucial for efficient execution of relief efforts to help the victims of these calamities. Given the low accessibility of the damaged areas, high-resolution optical satellite imagery has emerged as a valuable source of information to quickly asses the extent of damage by manually analyzing the pre- and postevent imagery of the region. To make this analysis more efficient, multiple learning techniques using a variety of image representations have been proposed. However, most of these representations are prone to variabilities in capture angle, sun location, and seasonal variations. To evaluate these representations in the context of damage detection, we present a benchmark of 86 pre- and postevent image pairs with respective reference data derived from United Nation Operational Satellite Applications Programme (UNOSAT) assessment maps, spanning a total area of 4665 km² from 11 different locations around the world. The technical contribution of our work is a novel image representation based on shape distributions of image patches encoded with locality-constrained linear coding. We empirically demonstrate that our proposed representation provides an improvement of at least 5%, in equal error rate, over alternate approaches. Finally, we present a thorough robustness analysis of the considered representational schemes, with respect to capture-angle variabilities and multiple sensor combinations.