Demand response resources (DRRs) are usually aggregated in order to participate in wholesale electricity markets (e.g., capacity, energy, and ancillary service markets). In such DRR aggregation systems, uncertainty arising from, e.g., random failures, is unavoidable; this paper focuses on assessing the impact of such uncertain phenomena on the reliability of DRR aggregation systems. To this end, we first develop a stochastic hybrid system (SHS) model to capture DRR continuous dynamics, as well as discrete events that arise from failures and repairs. The statistics of the DRR aggregation system state variables can be obtained by using the extended generator of the SHS. Then, we can use these statistics to estimate the value of the probability that the DRR aggregation system can successfully provide a certain amount of power for a period of time. Subsequently, by varying the values of the power to be provided and period duration, we can construct a probability-capacity-duration contour. Capacity-duration curves can be then obtained by setting the probability to desired confidence levels. The proposed method is illustrated through several examples and case studies.