Entropy-Guided LLM Decoding via Probabilistic Circuits

Zhizhen Chen Daniel Mingyi Israel Guy Van den Broeck Zhe Zeng Kareem Ahmed Sameer Singh Entropy-aligned K. Cobbe V. Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun Lukasz Kaiser Matthias Plappert Jerry Tworek Jacob Hilton Reiichiro Nakano

· 0 citations · 16 references

TL;DR

This work argues that future entropy is also essential for LLM decoding, and estimates its lower and upper bounds by computing the lower and upper bounds on a hidden Markov model (HMM), which is distilled as a surrogate of the LLM.

Semantic Scholar

View source

Similar papers

#artificial intelligence Preprint Sep 2026

Beyond Truncation: Rethinking LLM Decoding as Ensemble Pruning

This work introduces Mahalanobis-Ensemble Decoding (ME-Decoding), a novel Large Language Model (LLM) decoding framework that frames candidate token selection as ensemble pruning and devise an efficient greedy selection algorithm with near-linear complexity in the candidate size under early stopping, while establishing...

Dun-Yao Xue, Cheng-Shuo Du, Zheng-Bo Wang et al. · 0 citations

Preprint Sep 2026

Approximate maximum-likelihood decoding via truncated free energies

Maximum-likelihood decoding (MLD) achieves the minimum logical error rate of stabilizer codes under known i.i.d. Pauli noise, but its exact evaluation is \#P-hard. Practical pipelines therefore approximate MLD by minimum-weight decoding (MWD), retaining only the lowest-weight recovery per syndrome and discarding the co...

Yuan-Qi Liu, Wei-Lei Zeng, Jun-Jie Wu et al. · 0 citations

Preprint Aug 2026

LLM-Assisted LDPC Decoding via Syndrome-Verified Semantic Priors

Semantic communication exploits the meaning of the payload, which bit-level processing discards. When channel decoding fails on a natural language payload, the errors appear as corrupted characters in the recovered text. A large language model (LLM) infers the intended characters from the semantic context, but it can a...

Sojeong Park, Hyeonsu Lyu, Jaehyun Choi et al. · 1 citation

2026

Gaussian-Elimination-Free BP-OSD Algorithm for Short 5G LDPC Codes

In this paper, we propose a two-stage belief-propagation ordered statistics decoding (BP-OSD)-like algorithm, hereafter referred to as BP-LCGCD, tailored to 5G low-density parity-check (LDPC) codes. When standard BP decoding fails, a modified BP (mBP) with an iteration limit determined by the code’s girth and a normali...

Qian-Fan Wang, Yi-Wen Wang, Jia-Yi Yang et al. · IEEE Transactions on Communi... · 0 citations

Preprint Aug 2026

Logical Neural Belief Propagation for Linear-Complexity Decoding of Surface Codes

Quantum error correction (QEC) requires accurate and efficient decoders, yet belief propagation (BP), despite its linear decoding complexity, often provides insufficient logical accuracy on surface codes. We propose Logical Neural Belief Propagation (L-NBP), a BP-based neural decoder that redirects the decoding objecti...

Heeyoul Kwak, Seong-Joon Park, Dae-Young Yun et al. · 0 citations

#artificial intelligence Preprint Sep 2026

Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference

An empirical error-propagation analysis is developed and finds that 22 layers of accumulated body error do not distinguish flipping from non-flipping steps; the outcome depends primarily on the top-two logit margin at the LM head relative to the directional perturbation between the top-two candidates.

Gao-Yuan Du, A. Khan, Rex Zhou et al. · 0 citations