CAMEL investigates external residual memory for small language models intended for CPU execution. The main study uses a recurrent byte-level student with two compiled memories: an exact surface 4-gram memory and a learned latent residual memory. A 32-byte table selects between the two using support-count bins.On three held-out TinyStories runs, the selected hybrid improves bits per byte over surface memory, dense latent memory, and a 64-byte sparse latent baseline, without reducing next-byte accuracy relative to surface memory. A native lookup benchmark shows lower lookup-and-apply time than dense latent values. The same method does not outperform dense latent memory on enwik8. An address-inclusive benchmark also fails its exact-equivalence criterion because of one Python/C++ address disagreement.The paper also reports development experiments using Qwen3-0.6B as a teacher for a 2.43-million-parameter token-level GRU. In these experiments, exact token-history memory performs better than the learned product-context address. Residual bigram exceptions improve the token baseline on TinyStories at fixed logical work, but the unchanged method fails one of three enwik8 fits. An argmax-preserving projection keeps most of the loss improvement without changing the token baseline's predictions.The paper reports both successful and failed experiments and limits its claims to the tested models, datasets, and CPU kernels.
CAMEL investigates external residual memory for small language models intended for CPU execution. The main study uses a recurrent byte-level student with two compiled memories: an exact surface 4-gram memory and a learned latent residual memory. A 32-byte table selects between the two using support-count bins.On three held-out TinyStories runs, the selected hybrid improves bits per byte over surface memory, dense latent memory, and a 64-byte sparse latent baseline, without reducing next-byte accuracy relative to surface memory. A native lookup benchmark shows lower lookup-and-apply time than dense latent values. The same method does not outperform dense latent memory on enwik8. An address-inclusive benchmark also fails its exact-equivalence criterion because of one Python/C++ address disagreement.The paper also reports development experiments using Qwen3-0.6B as a teacher for a 2.43-million-parameter token-level GRU. In these experiments, exact token-history memory performs better than the learned product-context address. Residual bigram exceptions improve the token baseline on TinyStories at fixed logical work, but the unchanged method fails one of three enwik8 fits. An argmax-preserving projection keeps most of the loss improvement without changing the token baseline's predictions.The paper reports both successful and failed experiments and limits its claims to the tested models, datasets, and CPU kernels.
Posterior Ontology Memory (POM) models uncertainty in the vocabulary used by a symbolic memory, rather than assuming that its semantic schema is fixed. It treats mappings from surface predicates to latent semantic categories as uncertain and carries that uncertainty into prediction. The model combines a partition prior, collapsed Beta–Bernoulli rule models, noisy compiler judgments, and Bayesian model averaging. A temporal version allows schemas to split or merge and rule rates to reset, using exact enumeration for small reference cases and sequential Monte Carlo for approximate inference.In finite grounded-rule experiments, model averaging improves held-out prediction when surface predicates share latent behavior, but the advantage disappears when that assumption breaks down. The temporal study passed seven of eight predeclared gates: delayed split evidence met its criterion, while delayed merge evidence did not. In a separate study of 19 GitHub API migrations, a local language model retrieved 18 correct notices, compared with 7 for token overlap, although direct mutation classification was less reliable. The paper presents a bounded proof of concept for symbolic memory that retains uncertainty over changing semantic schemas.
Posterior Ontology Memory (POM) models uncertainty in the vocabulary used by a symbolic memory, rather than assuming that its semantic schema is fixed. It treats mappings from surface predicates to latent semantic categories as uncertain and carries that uncertainty into prediction. The model combines a partition prior, collapsed Beta–Bernoulli rule models, noisy compiler judgments, and Bayesian model averaging. A temporal version allows schemas to split or merge and rule rates to reset, using exact enumeration for small reference cases and sequential Monte Carlo for approximate inference.In finite grounded-rule experiments, model averaging improves held-out prediction when surface predicates share latent behavior, but the advantage disappears when that assumption breaks down. The temporal study passed seven of eight predeclared gates: delayed split evidence met its criterion, while delayed merge evidence did not. In a separate study of 19 GitHub API migrations, a local language model retrieved 18 correct notices, compared with 7 for token overlap, although direct mutation classification was less reliable. The paper presents a bounded proof of concept for symbolic memory that retains uncertainty over changing semantic schemas.
Henrik W. Lautergold· Figshare· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
