Enhancing Energy Management in Multi-Zone Buildings Using the On-Policy Reinforcement Learning Algorithm SARSA

Abstract

THVAC systems in commercial buildings, particularly open-plan offices, account for substantial energy use. Thermal coupling between adjacent zones complicates control, while conventional rule-based and model-based approaches may not adapt effectively to changing occupancy and environmental conditions. Deep reinforcement learning methods can also impose high computational demands, limiting their practical deployment. This study developed and evaluated a lightweight, computationally inexpensive HVAC control policy for multi-zone open-plan offices using the on-policy reinforcement learning algorithm SARSA. A tabular SARSA controller was implemented in an EnergyPlus–Python co-simulation of a six-zone office model. The agent was trained using realistic occupancy schedules, TMY3 weather data, and 5-minute control intervals. A multi-objective reward function balanced energy efficiency, thermal comfort, and switching frequency. Performance was benchmarked against rule-based, model predictive control (MPC), and deep Q-network (DQN) controllers across several U.S. climate regions. The SARSA controller reduced annual HVAC energy use by 39.39%, from 18,200 to 11,030 kWh. Thermal comfort violations were limited to 2.85%, with an average temperature deviation of 1.2 °F and a 15-minute recovery time. Switching frequency decreased to 4.1 ON/OFF cycles per zone per day, approximately half that of rule-based control. Across different climates, energy savings ranged from 32.6% to 40.7%. These findings show that tabular SARSA is an effective and computationally efficient alternative to deep reinforcement learning for real-time HVAC control in multi-zone smart buildings.