This paper presents a novel contextually supervised optimization-based approach for disaggregating heating, ventilation, and air-conditioning (HVAC) loads using smart meter or Supervisory Control and Data Acquisition data. To disaggregate the load into HVAC loads, large and infrequently used loads (LIUL), and base loads, we formulate an optimization problem to minimize a set of five loss terms, consisting of the reconstruction errors of the overall load profile, the ramp rate losses, and three distinct loss functions linked with the HVAC load, base load, and LIUL, respectively. To enhance accuracy, we incorporate two forms of contextual information into the problem formulation. First, we utilize mutual information to estimate HVAC energy consumption. Second, we employ a base load dictionary to constrain HVAC load estimation errors. The obtained HVAC load profiles are fine-tuned by abnormal ramp detection followed by binary hypothesis testing. The proposed method is developed and tested using sub-metered residential and commercial building data. Simulation results show that the proposed method outperforms existing methods across various data resolutions and load aggregation levels, showing excellent transferability and generalizability.