CORE-LLM-Bench: A Controlled Neurosymbolic Benchmark for Ontology-Grounded Reasoning in Large Language Models

Abstract

CORE-LLM-Bench is a controlled neurosymbolic benchmark for evaluating ontology-grounded reasoning in large language models. Version 1.1.1 is a corrective release that supersedes v1.1.0 for future use. It contains 9,048 unique question-hop instances: 6,032 binary question-answering (BQA) instances and 3,016 open-ended question-answering (OEQA) instances. This release corrects source IRIs, completes the OEQA entailment sets, replaces 67 affected FALSE BQA pair members, corrects natural-language identity and namespace annotations, and repairs proof and complexity metadata. The frozen evaluation contains 81,432 accepted observations. Of these, 64,974 validated observations were reused and 16,458 correction-affected evaluation cells were rerun through 16,443 deduplicated paid requests. Model predictions were not manually edited. Reused observations include 15,855 qualified non-exact-prompt natural-language transfers, documented in the release provenance. The complete Zenodo archive includes benchmark JSONs, the canonical corrected Parquet dataset, frozen model responses and predictions, correction provenance, per-observation evaluation outputs, validation and audit material, and release documentation. Source code and the lean reproducibility release are available from GitHub:https://github.com/jloe2911/CORE-LLM-Bench/releases/tag/v1.1.1 The canonical Hugging Face dataset revision is:https://huggingface.co/datasets/jloe2911/CORE-LLM-Bench/tree/v1.1.1 Licensing is component-specific. Repository software is MIT licensed; separable original CORE-LLM-Bench benchmark content is CC BY 4.0; Pizza-derived material follows CC BY 3.0; OWL2Bench-derived material follows Apache-2.0; and Family/FHKB-derived material follows CC BY-SA 3.0. See NOTICE.md for detailed provenance, attribution, and modification information.