A hierarchical reinforcement learning and predictive control framework for automated multi-variable anesthesia
Abstract
Background
AND
Objective
Automated multi-variable anesthesia must balance conflicting objectives amid patient variability, safety constraints, and partial observability. The state-of-the-art robust approach uses a Genetic Algorithm (GA) to optimize Generalized Predictive Control (GPC) parameters across a population, but the resulting fixed-parameter controller is conservative and slow to adapt. Here, we develop an adaptive drug-administration framework that maintains performance while adapting to patient variability and remaining robust to surgical stimulation and clinician intervention.
Methods
A hierarchical framework is proposed in which a high-level recurrent Reinforcement Learning (RL) agent supervises a low-level multi-variable GPC by dynamically tuning its cost-function weights. Using AReS, a multi-variable anesthesia response simulator with 44 virtual patients (35 for training, 9 held out), the framework is compared with a GA-optimized GPC under surgical stimulation and anesthesiologist intervention, and its generalization is further stress-tested on 24 synthetic in-distribution and out-of-distribution patients.
Results
The RL-GPC framework outperformed the fixed-parameter baseline, achieving 50% shorter induction (median 228 versus 452 seconds) while keeping every monitored variable's global score below 50 for all patients, where the baseline failed in several cases. This performance held on the held-out test set and under stimuli and interventions. On the synthetic cohorts, the framework generalized robustly within the training distribution, with no patient breaching the acceptability threshold, whereas hypnotic control degraded on out-of-distribution dynamics, delineating its operational envelope.
Conclusions
A hierarchical RL-GPC framework can learn a robust, adaptive policy for multi-objective automated anesthesia. These in-silico findings suggest potential benefits for patient safety, standardized care, and reduced clinician workload, and require clinical validation.