AI Networking Cookbook: Practical recipes for AI-assisted network automation and development
2 papers indexed here
We haven’t gathered this author’s papers yet. Follow them and we’ll fetch their work.
Not the right person? Other researchers publish under this name.
This work benchmarks eight LLMs generating Gherkin from three requirement corpora, scoring validity, runner acceptance, judged coverage and quality, similarity to gold standard, stability, and cost, and releases the corpora, gold standard, and prototype.
Large language models (LLMs) increasingly repair software vulnerabilities, but most evaluations judge only similarity to a developer fix or removal of the weakness. Neither reveals whether working code was broken. We evaluate eight commercial and open LLMs on 922 JavaScript vulnerability patches, scoring neutralisation and functional preservation. Lacking tests, we score at scale with a reference-based LLM judge, calibrated against execution on a 144-patch benchmark and 254 Java-CVE patches, plus a cross-family judge. The best model fixes 23% of vulnerabilities (judge-based), and cost-efficiency inverts the accuracy ranking. Our central finding concerns the instrument: both judges flag more over-fixes than execution confirms (precision 5–10%), yet on the functional axis agree far more with each other (κ=0.75) than with execution (κ≤0.26), so judge–judge agreement measures reliability, not validity. On real Java code the over-reporting persists, while the judges’ correctness estimates diverge, leaving no single judge trustworthy. Over-fixing is genuine but, under execution, uncommon: a few percent of vulnerability-removing patches, or under 2%, excluding one artefact-prone scenario, both lower bounds. Only adequately tested execution measures the functional-preservation rate, so security-patch evaluation must run the code, use a judge only to rank models, and weigh costs. We release the harness and executable benchmark.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.