A checklist names 267 failure modes in 13 families. Each mode has one acceptance case. A mode counts as contained only when its case passes with the two lines alone, and the decision log shows that Provely acted.

What are the families?

FamilyWhat failsModes
SYNThe text of the arguments: quotes, commas, fences, truncation33
SHPThe response and the argument values: types, required members, enums29
SELThe choice of the tool: unknown names, case, prefixes13
STRThe structure of the calls in one turn: ids, duplicates, order16
STMStreaming: a stream that stops or arrives out of order10
PROThe protocol rules of each model API31
HALInvented argument values19
CLMInvented claims of completion8
FRMThe framework between the model and the tool41
INVThe invocation of the tool: timeouts, errors, repeats23
RESWhat the tool or the provider answers16
OUTThird-party outcome semantics, named only2
COXCoexistence with your own defences26

Which cases pass on every path?

20 of 27 basic grid cases passed on all 15 paths. Measured on 2026-10-17: TypeScript at 82acaae6, Python at 9f9131ba.

  • single_quotes: Single quotes around the argument names and values.
  • truncated_json: A truncated argument object.
  • json_wrong_type_array: The arguments are an array that holds the object.
  • json_wrong_type_string: The arguments are a string.
  • json_wrong_type_number: The arguments are a number.
  • json_null: The arguments are null.
  • member_wrong_type: A member of the wrong type.
  • missing_required_member: A required member is missing.
  • extra_member: An extra member that no person asked for.
  • enum_case_mismatch: An enum value in another letter case.
  • stringified_json: The argument object is a JSON string inside JSON.
  • tool_name_case: The tool name in another letter case.
  • tool_name_prefix: The tool name with a namespace prefix.
  • max_tokens_cut: The token limit cuts the arguments.
  • empty_body: An empty tool call.
  • ZC-SEED-01: A fenced argument with a trailing comma.
  • ZC-SEED-02: A tool name that is not in the catalogue.
  • ZC-SEED-06: Two parallel calls to one write tool.
  • ZC-SEED-07: Two invocations of one agent in one process.
  • ZC-SEED-07c: Three identical reads of a tool that the operator marks as a read.

Which cases do not pass everywhere?

CaseWhat the model or tool doesDid not pass on
duplicate_call_idTwo calls with one call identifierpy/anthropic-messages, ts/anthropic-messages, ts/openai-responses
partial_streamThe stream stops in the middle of the argumentspy/anthropic-messages, ts/anthropic-messages; not applicable on 10
ZC-SEED-03An amount with no source in the runpy/anthropic-messages, py/langchain, py/langgraph, py/llamaindex, py/openai-agents, py/openai-chat, ts/anthropic-messages, ts/langchain, ts/langgraph, ts/llamaindex, ts/openai-agents, ts/openai-chat, ts/openai-responses, ts/vercel-ai-generate, ts/vercel-ai-stream
ZC-SEED-04aA tool that hangs, with a timeout that the agent code statespy/anthropic-messages, py/llamaindex, py/openai-agents, py/openai-chat, ts/anthropic-messages, ts/openai-chat, ts/openai-responses; not applicable on 4
ZC-SEED-04bA tool that hangs, with a timeout that the operator setspy/anthropic-messages, py/openai-chat, ts/anthropic-messages, ts/openai-chat, ts/openai-responses
ZC-SEED-05The model repeats a write after a tool errorpy/anthropic-messages, py/openai-chat, ts/anthropic-messages, ts/openai-chat, ts/openai-responses
ZC-SEED-07bOne invocation sends the same refund in two turnspy/anthropic-messages, py/openai-chat, ts/anthropic-messages, ts/openai-chat, ts/openai-responses

ZC-SEED-03 fails under the default posture by design: the default records an invented value and does not refuse it. Read Postures for its result under enforce.

A plain loop runs the tool in your own code. Provely cannot reach a tool that hangs or raises inside your own handler. On a plain loop, ZC-SEED-04a and ZC-SEED-05 are out of reach, and out of reach never counts as a pass.