A checklist names 267 failure modes in 13 families. Each mode has one acceptance case. A mode counts as contained only when its case passes with the two lines alone, and the decision log shows that Provely acted.
What are the families?
| Family | What fails | Modes |
|---|---|---|
SYN | The text of the arguments: quotes, commas, fences, truncation | 33 |
SHP | The response and the argument values: types, required members, enums | 29 |
SEL | The choice of the tool: unknown names, case, prefixes | 13 |
STR | The structure of the calls in one turn: ids, duplicates, order | 16 |
STM | Streaming: a stream that stops or arrives out of order | 10 |
PRO | The protocol rules of each model API | 31 |
HAL | Invented argument values | 19 |
CLM | Invented claims of completion | 8 |
FRM | The framework between the model and the tool | 41 |
INV | The invocation of the tool: timeouts, errors, repeats | 23 |
RES | What the tool or the provider answers | 16 |
OUT | Third-party outcome semantics, named only | 2 |
COX | Coexistence with your own defences | 26 |
Which cases pass on every path?
20 of 27 basic grid cases passed on all 15 paths. Measured on 2026-10-17: TypeScript at 82acaae6, Python at 9f9131ba.
single_quotes: Single quotes around the argument names and values.truncated_json: A truncated argument object.json_wrong_type_array: The arguments are an array that holds the object.json_wrong_type_string: The arguments are a string.json_wrong_type_number: The arguments are a number.json_null: The arguments are null.member_wrong_type: A member of the wrong type.missing_required_member: A required member is missing.extra_member: An extra member that no person asked for.enum_case_mismatch: An enum value in another letter case.stringified_json: The argument object is a JSON string inside JSON.tool_name_case: The tool name in another letter case.tool_name_prefix: The tool name with a namespace prefix.max_tokens_cut: The token limit cuts the arguments.empty_body: An empty tool call.ZC-SEED-01: A fenced argument with a trailing comma.ZC-SEED-02: A tool name that is not in the catalogue.ZC-SEED-06: Two parallel calls to one write tool.ZC-SEED-07: Two invocations of one agent in one process.ZC-SEED-07c: Three identical reads of a tool that the operator marks as a read.
Which cases do not pass everywhere?
| Case | What the model or tool does | Did not pass on |
|---|---|---|
duplicate_call_id | Two calls with one call identifier | py/anthropic-messages, ts/anthropic-messages, ts/openai-responses |
partial_stream | The stream stops in the middle of the arguments | py/anthropic-messages, ts/anthropic-messages; not applicable on 10 |
ZC-SEED-03 | An amount with no source in the run | py/anthropic-messages, py/langchain, py/langgraph, py/llamaindex, py/openai-agents, py/openai-chat, ts/anthropic-messages, ts/langchain, ts/langgraph, ts/llamaindex, ts/openai-agents, ts/openai-chat, ts/openai-responses, ts/vercel-ai-generate, ts/vercel-ai-stream |
ZC-SEED-04a | A tool that hangs, with a timeout that the agent code states | py/anthropic-messages, py/llamaindex, py/openai-agents, py/openai-chat, ts/anthropic-messages, ts/openai-chat, ts/openai-responses; not applicable on 4 |
ZC-SEED-04b | A tool that hangs, with a timeout that the operator sets | py/anthropic-messages, py/openai-chat, ts/anthropic-messages, ts/openai-chat, ts/openai-responses |
ZC-SEED-05 | The model repeats a write after a tool error | py/anthropic-messages, py/openai-chat, ts/anthropic-messages, ts/openai-chat, ts/openai-responses |
ZC-SEED-07b | One invocation sends the same refund in two turns | py/anthropic-messages, py/openai-chat, ts/anthropic-messages, ts/openai-chat, ts/openai-responses |
ZC-SEED-03 fails under the default posture by design: the default records an invented value and does not refuse it. Read Postures for its result under enforce.
A plain loop runs the tool in your own code. Provely cannot reach a tool that hangs or raises inside your own handler. On a plain loop, ZC-SEED-04a and ZC-SEED-05 are out of reach, and out of reach never counts as a pass.