Beyond Scalar Loss: An Empirical Observatory for Persistent Ballistic Flow and Hidden Descent in Transformers

Abstract

This is an updated version. The changelog is at the end. Abstract of Beyond Scalar Loss: An Empirical Observatory for Persistent Ballistic Flow and Hidden Descent in Transformers Scalar training loss provides a necessary but incomplete description of neural optimization. It records task error but does not directly reveal the geometry of latent states through which a transformer executes computation, nor the internal optimization state that produces that trajectory. This paper develops a phenomenological telemetry and control framework connecting these levels. The framework combines hidden-state kinematics, head-level geometric differentiation, optimization-pressure observables, and a closed-loop controller termed Critical Surfing. Several correspondences with parameter-space geometry and singular learning theory are treated as operational hypotheses rather than established mathematical equivalences. Hidden-state increments are analyzed through drift, diffusion, directional signal-to-noise ratio, cross-token directional coherence, and head-level variance. Optimization state is summarized by aggregate pressure Π, the Constructive–Compensatory Ratio Rₜ, and the empirical response function χᵣ. An extended telemetry interval of MyceliaLM, a 1.62-billion-parameter transformer with 24 layers, 32 attention heads, and hidden dimension 2048, provides the principal case study. Across the reported audit interval (Steps 190,460 to 389,000, >6.37 Billion tokens), the model exhibits persistent drift-dominated latent motion, with directional SNR exceeding 100 (ρ_dir ≈ 112.48 ± 1.82, ρ_raw ≈ 115.70) while cross-token directional coherence remains near zero (cₗ ≈ 0.000). At the same time, optimization pressure varies substantially (Π ~ 232.5 - 348.9) and the SURFER controller actively adjusts attention/alpha capacity in response to the measured sign of χᵣ. Relative variability of χᵣ substantially exceeds that of ρ_dir over the high-resolution audit window (CV_χᵣ / CV_ρ_dir > 148×), establishing an empirical separation between optimization-state dynamics and latent kinematic dynamics. The dataset composition spans a rich local anchor mix (Czech fantasy prose, Traditional Chinese Medicine, Stanford Philosophy, scriptural texts, and domain-specific knowledge shards) combined with a high-entropy expansion corpus (90% FineWeb-Edu, 10% FineWeb Original) under a 50,000-step pH-Stirred Mixture Protocol. Statistical analysis proves that loss movements and structural head differentiation are uncorrelated with batch composition (r = +0.0543, p = 0.3004), confirming that latent dynamics are driven by intrinsic optimization physics. The resulting framework is presented as an empirical observatory and control system rather than a claim that neural networks constitute thermodynamic systems in the physical sense. Paper I establishes the dynamical layer of a broader three-paper program: trajectory telemetry, completion detection, and runtime trajectory governance. Introduction Training progress is conventionally monitored through scalar quantities such as loss, perplexity, gradient norms, learning rate, and validation metrics. These measurements are indispensable, but a scalar loss compresses a high-dimensional dynamical process into a single observable. Two training states can have similar loss while differing substantially in hidden-state motion, optimization pressure, head specialization, or controller activity. The central question of this work is therefore not only whether loss decreases, but what the model’s internal trajectory is doing while loss changes. We distinguish three operational levels: (1) the input regime, describing batch composition and statistical properties; (2) the optimization regime, describing controller state and pressure allocation; and (3) the latent dynamical regime, describing hidden-state kinematics and geometric differentiation. This separation matters because large loss excursions need not imply a corresponding transition in latent geometry. Conversely, geometric reorganization may occur while scalar loss remains comparatively insensitive.The paper asks four empirical questions: Can latent hidden-state motion be characterized by stable kinematic observables over long training intervals? Can optimization-state variation be measured independently of latent trajectory stability? Can derivative-level geometric telemetry expose representational changes weakly visible in scalar loss? Can an explicit feedback controller regulate an empirical optimization response variable without destabilizing the measured latent regime? The framework is deliberately phenomenological. Terms such as “pressure”, “work”, “friction”, “ballistic”, and “thermodynamics” refer to operational quantities defined on the telemetry, not to claims that neuraloptimization is literally governed by equilibrium thermodynamics. The present paper, Beyond Scalar Loss: An Empirical Observatory for Persistent Ballistic Flow and Hidden Descent in Transformers, is one of a trininity of interconnected papers introducing a specific perspective on treating a different aspects of the physics of neural network generalization, pre-training completion, and runtime alignment. The understanding of these processes requires bridging static parameter-space geometry with dynamic, continuous-time hidden-state trajectory execution. Bridging Static Parameter-space Geometry with Dynamic, Continuous-time Hidden-state Trajectory Execution The tree papers establish together a unified, self-regulating foundation model stack across three operational levels: Foundational Trajectory Telemetry & Operational Mappings: We investigate candidate operational correspondences connecting static algebraic-geometric properties under Singular Learning Theory (SLT) - parameter fibers, fiber volumes, and the Real Log Canonical Threshold (λ) - to dynamic differential-geometric telemetry on attention-induced Riemannian manifolds. These mappings are treated as operational hypotheses rather than established mathematical equivalences. Specifically, we propose that parameter redundancies (padding functions) may correspond operationally to the residual component of a coordinate-disentangled ANOVA hidden-state decomposition; that the macroscopic order parameter R_t may serve as an empirical probe for structural phase transitions; and that the Alpha Potential Well U(α) acts as a curvature-motivated regularizing potential against scaling symmetries. Kramers–Moyal expansions and Pawula-based diagnostics provide empirical support, but not proof, for a low-order drift–diffusion description, with higher-order diffusion terms remaining small relative to second-order variance Dʌ(4)≪(Dʌ(2)ʌ2). Extended high-resolution auditing of MyceliaLM (1.62B parameters) reveals Persistent Individual Ballistic Flow (PIBF), wherein token representations move along deterministic, drift-dominated geodesics (ρ_dir>100) while cross-token directional alignment remains near zero (c_≈0.000) and head variance spikes (σ_headʌ2≈0.730–0.760), consistent with Spontaneous Symmetry Breaking into a specialized geometric routing swarm. Across 238 snapshot audits, the relative variability of the Optimization Response Function χ_R exceeds that of directional SDE SNR ρ_dir by a factor of >148×, establishing Kinematic-Thermodynamic Decoupling as an empirical phenomenon. Furthermore, near-zero correlations between structural differentiation velocity (dσ_headʌ2/dt) and scalar loss shifts (ΔL≈0) motivate the Hidden Descent Hypothesis: internal representational specialization can proceed independently of scalar loss at the Epistemic Floor. This is the subject treated by the present paper, Beyond Scalar Loss: An Empirical Observatory for Persistent Ballistic Flow and Hidden Descent in Transformers. Closed-Loop Pre-Training Completion & Stopping Standard: To solve open-loop pre-training stopping ambiguity when scalar loss plateaus at the Epistemic Floor (L_total = L_epistemic + L_aleatoric), we introduce the Geometric Convergence Index (GCI). The GCI audits six coordinate-disentangled information-geometric criteria: inter-head fiber curvature saturation (σ²_head), differentiation velocity decay (dσ²_head/dt), attention entropy variance stability, governor work rate minimization (Ẇ_gov), Pawula Fokker–Planck stability (R_Pawula), and macroscopic order stability (R_t). The GCI defines a physics-grounded stopping standard, the Crystallization Horizon (GCI = 1.0 for 10,000 consecutive steps), that terminates pre-training the exact moment representational maturity is reached, preventing premature halt or multi-million-dollar compute waste. This is the subject of the paper, Beyond Loss Convergence: The Geometric Convergence Index and the Crystallization Horizon of Transformer Manifolds. In-Vivo Runtime Governance & Silicon Alignment: Extending forward-pass geometry monitoring to inference, we introduce the Kinematic Friction Rail and the Cache-Fused Kinematic Rail (CFKR). By monitoring depth-dependent hidden-state variance differentials via the Friction Delta (Δ = V_late - V_early), CFKR detects Geometric Flatlines (Δ → 0.00) in vivo prior to token emission, resolving the Permissive Consensus Paradox. Rather than relying on software-level additive logit bias matrices (M_ghost) and token eviction, CFKR executes kernel-level Key-Value (KV) cache fusion, grafting low-entropy reflection key-value states directly into GPU memory layout. Benchmark evaluations demonstrate that Cache-Fused KV mutation achieves an 8.88× latency acceleration over software-level bias matrices (1.58 ms vs. 14.00 ms per intervention step), eliminates O(T²) VRAM allocation spikes, and scales linearly O(G). This enables Dynamic Compute Allocation, allowing models to run at maximum ballistic speed (ρ_dir > 100) for ~95% of standard queries, paying the Chain-of-Thought reflection tax only when an active geometric flatline is detected.This is the subje