Proof an agent works, before you publish
An eval in Definable is a list of cases an agent must keep getting right: a message, and the checks its answer has to pass, from required words and valid JSON to the tools it used and an LLM judge. Evals run on the draft before you publish and compare it with a baseline, so you see anything that got worse first.
How evals work
An eval suite is a short list of cases an agent must keep getting right. Run it after every change and the score says whether the change helped.
At a glance
- 10 check types
- A case passes at 0.7
- Up to 50 cases a run
- 15 minutes per case
- Run evals first is on by default
- 01
Cases and their checks
A case is a message and the checks its answer must pass: text it contains or must not, a pattern, valid JSON, an exact match, the tools it used, never used or used in order, a step budget, a cost budget, or an LLM judge with one rubric line per criterion.
- 02
Real runs, nothing touched
Every case runs for real on a hidden thread. Tools that aren't safe to repeat answer from fixtures, so nothing acts on the world.
- 03
One score against a baseline
Each check scores 0 to 1 and a case passes at 0.7. Every run is measured against the baseline you set, and each case that dropped shows the check and the evidence behind it.
- 04
A check before you publish
With Run evals first on, publishing runs the suite on the draft and shows the new score and any case that regressed. Keep editing, or publish anyway if the trade is deliberate.
Evals FAQ
How do I know an agent is ready?
Write eval cases with checks, including an LLM judge, and compare every change against a baseline. Evals run on the draft before you publish, so you see anything that got worse first.
What can an eval check?
Words the answer must or mustn't contain, a pattern or valid JSON, tools it must or mustn't call and their order, step and cost budgets, and an LLM judge that grades against your rubric. Actions that write or send answer from fixtures, so nothing really happens.
How do I change an agent safely?
Edits go into a draft; people keep talking to the published version until you publish. Each version keeps its own model, skills and sandbox settings. Before you publish, evals run on the draft and show anything that got worse.