Managed cloud databases such as RDS, Aurora, Cloud SQL, and Azure SQL run inside the cloud provider. You cannot install an agent, read the disk, or access the log files. The only backup tool the provider offers is a snapshot API call. A snapshot is a black box. To read even one row from it, you must first restore the w...
Assaf Natanzon· Proceedings of the 19th ACM...· 0 citations
In the Era by Eon benchmark, each question states the rules for its answer, and code computes the answer from a generated company's data. When agents can run code, the four strongest models each answer 22 to 25 of 27 such questions, so the benchmark barely separates them. We add eight question templates that depend on...
Benjamin Gruenbaum, Doron Porat, Assaf Natanzon et al.· 0 citations
LLM agents for enterprise systems of record cannot be evaluated on customer production data, and no existing substitute provides ground truth. We present the Era by Eon Benchmark for evaluating LLM agents that use enterprise tools. The benchmark is built around a complete fictional company. It includes product simulato...
Benjamin Gruenbaum, Doron Porat, Assaf Natanzon et al.· 3 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
