ProvelyKeep your own defences. The promise: your defences plus the two Provely lines are never worse than either one alone. Six properties state it, and the grid tests each one. The result below includes the cases that fail.
Which properties does Provely promise?
| Property | Name | What it means |
|---|---|---|
| C1 | Clean runs do not change | With your defences and no fault, the run is identical: the same calls, the same arguments, the same answer. |
| C2 | Never worse | A case passes with both when it passes with your defences alone or with Provely alone. |
| C3 | One decision for each call | Each call gets one line. A repair by Provely and then by your code gives the same arguments as one repair. |
| C4 | Your handler still runs | Every handler you wrote runs when its fault occurs, and it gets the error type it expects. |
| C5 | The stricter choice wins | Your shorter timeout, lower retry count, guardrail refusal and idempotency key stay as they are. |
| C6 | No loop | Your retries plus the re-asks of Provely stay inside the smaller budget. |
Which defences does the grid run?
| Framework | Defences in the defended app |
|---|---|
| LangChain and LangGraph | tool error handlers, validation error handlers, with_retry, with_fallbacks, output fixing parsers, recursion limits |
| LlamaIndex | max_iterations, a try block in the tool, a retry wrapper, a timeout |
| OpenAI Agents | failure_error_function, input and output guardrails, max_turns, a tool timeout |
| Vercel AI | experimental_repairToolCall, maxRetries, stopWhen, an abort signal, a try block |
| Plain SDKs | a try block around the JSON parse, jsonrepair, zod or pydantic with a re-ask, p-retry or tenacity, a timeout, your own idempotency key and dedupe set |
What did the grid measure?
| Language | Paths | Cases that hold every property | Fail | Pending |
|---|---|---|---|---|
| TypeScript | 9 | 21 of 27 (rules cases not yet measured) | 6 | 0 |
| Python | 6 | 24 of 27 (1 passes under guard_mode: enforce; observe records only; rules cases not yet measured) | 2 | 1 |
A case holds when it holds on every path of that language. A fail names a property that did not hold on at least one path. A pending case has not run yet.
Measured on 2026-10-17 at commit 82acaae6. Measured on 2026-10-17 at commit 9f9131ba.
Source: tests/zero-code defended grid · retrieved 2026-10-17
Whose choice wins when both act?
The stricter choice wins: that is property C5. Provely never replaces an idempotency key that you supplied. It never refuses a retry that carries the key of the first attempt.