research-article

Deep Reinforcement Learning for Building HVAC Control

Authors:

Qi ZhuAuthors Info & Claims

DAC '17: Proceedings of the 54th Annual Design Automation Conference 2017

Article No.: 22, Pages 1 - 6

https://doi.org/10.1145/3061639.3062224

Published: 18 June 2017 Publication History

Abstract

Buildings account for nearly 40% of the total energy consumption in the United States, about half of which is used by the HVAC (heating, ventilation, and air conditioning) system. Intelligent scheduling of building HVAC systems has the potential to significantly reduce the energy cost. However, the traditional rule-based and model-based strategies are often inefficient in practice, due to the complexity in building thermal dynamics and heterogeneous environment disturbances. In this work, we develop a data-driven approach that leverages the deep reinforcement learning (DRL) technique, to intelligently learn the effective strategy for operating the building HVAC systems. We evaluate the performance of our DRL algorithm through simulations using the widely-adopted EnergyPlus tool. Experiments demonstrate that our DRL-based algorithm is more effective in energy cost reduction compared with the traditional rule-based approach, while maintaining the room temperature within desired range.

References

[1]

E. Barrett and S. Linder. Autonomous HVAC Control, A Reinforcement Learning Approach. Springer, 2015.

[2]

L. Bottou. Large-scale machine learning with stochastic gradient descent. Proceedings of COMPSTAT. 2010.

[3]

G. T. Costanzo and et al. Experimental analysis of data-driven control for a building heating system. CoRR, abs/1507.03638, 2015.

[4]

EnergyPlus. https://energyplus.net/.

[5]

D. Ernst and et al. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 2005.

Digital Library

[6]

P. Fazenda and et al. Using reinforcement learning to optimize occupant comfort and energy usage in hvac systems. Journal of Ambient Intelligence and Smart Environments, pages 675--690, 2014.

Digital Library

[7]

K. He and et al. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. IEEE International Conference on Computer Vision, 2015.

Digital Library

[8]

G. Hinton, N. Srivastava, and K. Swersky. Lecture 6a overview of mini---batch gradient descent. http://www.es.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf.

[9]

B. Li and L. Xia. A multi-grid reinforcement learning method for energy conservation and comfort of HVAC in buildings. pages 444--449, 2015.

[10]

Y. Ma and et al. Model predictive control for the operation of building cooling systems. IEEE Transactions on Control Systems Technology, 20(3):796--803, 2012.

[11]

M. Maasoumy and et al. Model-based hierarchical optimal control design for HVAC systems. DSCC, 2011.

[12]

V. Mnih and et al. Human-level control through deep reinforcement learning. Nature 518.7540, 2015.

[13]

National Solar Radiation Data Base. http://rredc.nrel.gov.

[14]

D. Nikovski, J. Xu, and M. Nonaka. A method for computing optimal set-point schedules for HVAC systems. REHVA World Congress CLIMA, 2013.

[15]

F. Oldewurtel and et al. Energy efficient building climate control using stochastic model predictive control and weather predictions. ACC, 2010.

[16]

S. J. Olivieri and et al. Evaluation of commercial building demand response potential using optimal short-term curtailment of heating, ventilation, and air-conditioning loads. Journal of Building Performance Simulation, 2014.

[17]

D. Ormoneit and S. Sen. Kernel-based reinforcement learning. Machine Learning, 49(2):161--178, 2002.

Digital Library

[18]

M. Riedmiller. Neural Fitted Q Iteration -- First Experiences with a Data Efficient Neural Reinforcement Learning Method. Springer, 2005.

[19]

D. Silver and et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587), 2016.

[20]

SCE. https://www.sce.com/NR/sc3/tm2/pdf/CE281.pdf.

[21]

A. Standard. Standard 55-2004-thermal environmental conditions for human occupancy. ASHRAE Inc., 2004.

[22]

D. Urieli and P. Stone. A learning agent for heat-pump thermostat control. AAMAS, 2013.

Digital Library

[23]

U.S. DoE. Buildings energy data book.

[24]

C. J. Watkins and P. Dayan. Q-learning. Machine learning, 8(3-4):279--292, 1992.

Digital Library

[25]

T. Wei, Q. Zhu, and M. Maasoumy. Co-scheduling of HVAC control, EV charging and battery usage for building energy efficiency. ICCAD, 2014.

Digital Library

[26]

M. Wetter. Co-simulation of building energy and control systems with the building controls virtual test bed. Journal of Building Performance Simulation, 2011.

[27]

L. Yang and et al. Reinforcement learning for optimal control of low exergy buildings. Applied Energy, 2015.

Cited By

Sun HHu YLuo JGuo QZhao J(2025)Enhancing HVAC Control Systems Using a Steady Soft Actor–Critic Deep Reinforcement Learning ApproachBuildings10.3390/buildings1504064415:4(644)Online publication date: 19-Feb-2025
https://doi.org/10.3390/buildings15040644
Zheng WZabala LFebres JBlum DWang Z(2025)Quantifying and simulating the weather forecast uncertainty for advanced building controlJournal of Building Performance Simulation10.1080/19401493.2025.2453537(1-16)Online publication date: 28-Jan-2025
https://doi.org/10.1080/19401493.2025.2453537
Quang TDoan DPhuong NZhang TGhaffarianhoseini AGhaffarianhoseini A(2025)Predicting indoor temperature and humidity in a naturally ventilated office room using long short-term memory networks model in a tropical climateArchitectural Engineering and Design Management10.1080/17452007.2024.2449244(1-21)Online publication date: 15-Jan-2025
https://doi.org/10.1080/17452007.2024.2449244
Show More Cited By

Recommendations

MARCO - Multi-Agent Reinforcement learning based COntrol of building HVAC systems
e-Energy '20: Proceedings of the Eleventh ACM International Conference on Future Energy Systems

Optimal control of building heating, ventilation, air-conditioning (HVAC) equipment has typically been based on rules and model-based predictive control (MPC). Challenges in developing accurate models of buildings render these approaches sub-optimal and ...
Multi-zone Residential HVAC Control with Satisfying Occupants’ Thermal Comfort Requirements and Saving Energy via Reinforcement Learning
Parallel and Distributed Computing, Applications and Technologies
Abstract
Residential HVAC system control has been focused on thermal comfort and energy consumption. Due to the complexity of the dynamic building thermal model, weather conditions and human activities, traditional methods such as rule-based control (RBC) ...
An online reinforcement learning approach for HVAC control
Abstract
Heating, Ventilation and Air Conditioning (HVAC) optimization for energy consumption reduction is becoming ever more a topic of the utmost environmental and energetic concerns. The two most employed methodologies for optimizing HVAC systems are ...
Highlights
- HVAC optimization with reinforcement learning algorithms.
- Assessment and comparison of three different approaches.
- Online approach with imitation learning provide reliable and inexpensive solution.

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

DAC '17: Proceedings of the 54th Annual Design Automation Conference 2017

June 2017

533 pages

ISBN:9781450349277

DOI:10.1145/3061639

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

EDAC: Electronic Design Automation Consortium
SIGDA: ACM Special Interest Group on Design Automation
IEEE-CEDA

In-Cooperation

SIGBED: ACM Special Interest Group on Embedded Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 June 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Conference

DAC '17

Sponsor:

EDAC
SIGDA

DAC '17: The 54th Annual Design Automation Conference 2017

June 18 - 22, 2017

TX, Austin, USA

Acceptance Rates

Overall Acceptance Rate 1,770 of 5,499 submissions, 32%

Upcoming Conference

DAC '25

Sponsor:
sigda

62nd ACM/IEEE Design Automation Conference

June 22 - 26, 2025

San Francisco , CA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

268
Total Citations
View Citations
3,373
Total Downloads

Downloads (Last 12 months)410
Downloads (Last 6 weeks)62

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sun HHu YLuo JGuo QZhao J(2025)Enhancing HVAC Control Systems Using a Steady Soft Actor–Critic Deep Reinforcement Learning ApproachBuildings10.3390/buildings1504064415:4(644)Online publication date: 19-Feb-2025
https://doi.org/10.3390/buildings15040644
Zheng WZabala LFebres JBlum DWang Z(2025)Quantifying and simulating the weather forecast uncertainty for advanced building controlJournal of Building Performance Simulation10.1080/19401493.2025.2453537(1-16)Online publication date: 28-Jan-2025
https://doi.org/10.1080/19401493.2025.2453537
Quang TDoan DPhuong NZhang TGhaffarianhoseini AGhaffarianhoseini A(2025)Predicting indoor temperature and humidity in a naturally ventilated office room using long short-term memory networks model in a tropical climateArchitectural Engineering and Design Management10.1080/17452007.2024.2449244(1-21)Online publication date: 15-Jan-2025
https://doi.org/10.1080/17452007.2024.2449244
Abdulraheem ALee SJung I(2025)Dynamic Personalized Thermal Comfort Model:Integrating Temporal Dynamics and Environmental Variability with Individual PreferencesJournal of Building Engineering10.1016/j.jobe.2025.111938(111938)Online publication date: Jan-2025
https://doi.org/10.1016/j.jobe.2025.111938
Li RZou Z(2025)How far back shall we peer? Optimal air handling unit control leveraging extensive past observationsBuilding and Environment10.1016/j.buildenv.2024.112347269(112347)Online publication date: Feb-2025
https://doi.org/10.1016/j.buildenv.2024.112347
Kadamala KChambers DBarrett E(2025)Transfer Learning with TD3 for Adaptive HVAC Control in Diverse Building EnvironmentsHighlights in Practical Applications of Agents, Multi-Agent Systems, and Digital Twins: The PAAMS Collection10.1007/978-3-031-73058-0_21(256-267)Online publication date: 3-Jan-2025
https://doi.org/10.1007/978-3-031-73058-0_21
Le TPriya JLe HLe NDuong MCao D(2024)Harnessing artificial intelligence for data-driven energy predictive analytics: A systematic survey towards enhancing sustainabilityInternational Journal of Renewable Energy Development10.61435/ijred.2024.6011913:2Online publication date: 1-Mar-2024
https://doi.org/10.61435/ijred.2024.60119
Yamasaki TMiyasaka F(2024)Development of an Automatic Control System for Individual Air Conditioning Equipment Using Machine Learning機械学習を用いた個別空調設備の自動制御システムの開発Transactions of the Institute of Systems, Control and Information Engineers10.5687/iscie.37.9937:4(99-105)Online publication date: 15-Apr-2024
https://doi.org/10.5687/iscie.37.99
Latoń DGrela JOżadowicz A(2024)Applications of Deep Reinforcement Learning for Home Energy Management Systems: A ReviewEnergies10.3390/en1724642017:24(6420)Online publication date: 20-Dec-2024
https://doi.org/10.3390/en17246420
Ginzburg-Ganz ESegev IBalabanov ASegev EKaully Naveh SMachlev RBelikov JKatzir LKeren SLevron Y(2024)Reinforcement Learning Model-Based and Model-Free Paradigms for Optimal Control Problems in Power Systems: Comprehensive Review and Future DirectionsEnergies10.3390/en1721530717:21(5307)Online publication date: 25-Oct-2024
https://doi.org/10.3390/en17215307
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten