Chain-of-thought (CoT) prompting elicits intermediate reasoning steps from large language models (LLMs), but treats reasoning as a strictly linear sequence. Many multi-hop questions instead decompose naturally into a small directed acyclic graph (DAG) of sub-tasks, some of which are mutually independent. We implement GraphThought, a lightweight two-phase prompting framework that (1) asks an LLM to decompose a question into a DAG using a structured text format (GSF), and (2) resolves each node in topological order, passing only the resolved conclusions of its direct predecessors as context. We evaluate a working implementation on 30 questions from the HotpotQA distractor validation set using Google's Gemini 2.5 Flash (free tier), and compare it against a standard, single-call CoT baseline run on the identical 30 questions, same model, same rate limits, and the same manual grading procedure. GraphThought produces a syntactically valid graph on 30/30 questions and reaches a semantically correct final answer on 20/30 (66.7%). The CoT baseline reaches the same 20/30 (66.7%) under a strict grading convention, or 22/30 (73.3%) under a more permissive convention that also credits a correct entity clearly stated earlier in the response even when a different fact appears on the model's designated final-answer line (Section 4, Section 6). Under either convention, GraphThought does not outperform simple prompting on this pilot — at best it ties, at worst it trails by two questions. This report documents a working, reproducible implementation and an honest comparative pilot rather than a benchmark-beating result, and we are explicit throughout about the limits of what a 30-question, single-model, single-annotator pilot can support.
Akmal Rustamov· Zenodo (CERN European Organi...· 0 citations
Chain-of-thought (CoT) prompting elicits intermediate reasoning steps from large language models (LLMs), but treats reasoning as a strictly linear sequence. Many multi-hop questions instead decompose naturally into a small directed acyclic graph (DAG) of sub-tasks, some of which are mutually independent. We implement GraphThought, a lightweight two-phase prompting framework that (1) asks an LLM to decompose a question into a DAG using a structured text format (GSF), and (2) resolves each node in topological order, passing only the resolved conclusions of its direct predecessors as context. We evaluate a working implementation on 30 questions from the HotpotQA distractor validation set using Google's Gemini 2.5 Flash (free tier), and compare it against a standard, single-call CoT baseline run on the identical 30 questions, same model, same rate limits, and the same manual grading procedure. GraphThought produces a syntactically valid graph on 30/30 questions and reaches a semantically correct final answer on 20/30 (66.7%). The CoT baseline reaches the same 20/30 (66.7%) under a strict grading convention, or 22/30 (73.3%) under a more permissive convention that also credits a correct entity clearly stated earlier in the response even when a different fact appears on the model's designated final-answer line (Section 4, Section 6). Under either convention, GraphThought does not outperform simple prompting on this pilot — at best it ties, at worst it trails by two questions. This report documents a working, reproducible implementation and an honest comparative pilot rather than a benchmark-beating result, and we are explicit throughout about the limits of what a 30-question, single-model, single-annotator pilot can support.
Akmal Rustamov· Zenodo (CERN European Organi...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
