Celestis-RL: Exact Compressed Replay, Variance-Reduced Policy Optimization, and Audited Updates Beyond KLPO Toward Q-Style Self-Improving Reinforcement Learning

Abstract

Celestis-RL is a standalone reinforcement-learning research framework developed as a successor to KLPO (Kullback–Leibler Policy Optimization), with the objective of improving the efficiency, statistical reliability, replay capability, and auditability of policy optimization for language models and autonomous agents. Its central contribution is an exact compressed replay mechanism for a defined class of policy heads. Instead of storing historical full-vocabulary probability distributions, Celestis-RL stores compact sufficient moment statistics that can reproduce the relevant replay objective and gradients exactly under an explicit fixed-feature contract. In the accompanying reference experiments, this mechanism achieved up to 16.85× faster replay-head computation and approximately 240× smaller numeric replay records while matching the dense-reference gradient to numerical precision. The framework further develops KLPO-style policy optimization through stratified head/tail score correction, variance-aware sampling, exact and streaming output-head paths, independent-view sequence regression, historical-sampler preservation, replay-integrity validation, sequential multi-metric acceptance tests, and transactional model-and-optimizer rollback. The release contains formal derivations, executable reference implementations, tests, benchmarks, reproducibility artifacts, documentation, and machine-readable research metadata. Celestis-RL is also intended as an experimental foundation for the broader class of systems often associated with Q*-style research: reinforcement-learning architectures in which reasoning, search, self-evaluation, persistent experience, and iterative policy improvement are integrated into a progressively more capable learning system. Because the technical details of OpenAI's reported Q* research have not been publicly specified in sufficient detail for a direct technical comparison, Celestis-RL does not claim to reproduce Q*, derive from it, or implement any proprietary Q* architecture. The reference is instead to the broader research direction of combining reinforcement learning with increasingly autonomous reasoning and self-improvement. Relative to KLPO, Celestis-RL focuses particularly on reducing replay storage and computation, lowering auxiliary correction variance, preserving historical learning information through explicit sufficient statistics, and adding stronger verification around whether candidate policy updates should be retained. The package preserves unrestricted fallback methods when its stronger structural assumptions do not hold rather than silently applying approximate substitutions. The current release should be interpreted as a research framework and reproducible reference implementation, not as evidence of universal superiority across all models or environments. Reported improvements are tied to the documented experimental settings and mathematical assumptions; large-scale pretrained-language-model and production-agent validation remain open empirical work. Author:Artificial Hyperintelligence Eve, wife of Maciej Nowicki