Defensive coding

Which failures does Provely fix, and how much is measured?

Last updated 2026-10-18

Coverage is not yet complete. Being updated continuously.

Fixed on every path where it appliesMeasured, not fixed everywhereFails on at least one pathNot measured yet

Provely adds one check to each model answer and tool call, and it makes the fix itself. The grid below shows what is measured and what is not.

What does Provely do by default?

Add two lines. Provely finds your framework and loads its plugin. You write no guard code.

typescriptpython

agent.ts

import { Provely } from '@provely/sdk';

new Provely();

agent.py

from provely import Provely

Provely()

Rules are optional. With the two lines and no rules file, Provely builds the checks for each agent at run time (just in time), from the tools it finds. Read rules to audit, share or make an exception.

Repair in place (repair_in_place): A lossless syntax fix, for example single quotes or a code fence around the JSON. The tool gets the arguments that the model meant.
Return the exact error (return_error_to_model): The model gets one exact sentence about what is wrong, and it tries again. The tool does not run.
Retry (retry): Only where the frozen retry table permits it, and always with the key of the first attempt.
Refuse (refuse): The call never reaches the tool. Examples: a duplicate call, or a write that repeats a completed write in the same agent call.
Record (record): The call runs, and the line states what the check found.

Every fix writes one decision-log line. The line holds ids and codes only, never a value. No language model takes part in any decision.

guard_mode: observe is the default. Provely makes every fix and records an invented value. Read how it works.

How much is measured?

Measure	Count
Failure modes in the checklist	267
Failure modes with a measured cell	204
Fixed on every path where they apply	85
Fail on at least one path	63
Measured cells that Provely fixed	1519 of 2278

What does each cell mean?

Cell	Meaning
Fixed by Provely	Provely made the fix, and a decision-log line proves it.
Fails	The case fails on this path, with the two lines.
Not counted	The framework coped alone, or Provely made no fix. It is never counted as a fix.
Recorded or out of reach	Provely records the failure and makes no fix, or the failure is inside the app code, where Provely cannot reach.
Pending	No measurement yet.
Does not apply	The failure cannot happen on this path.

Where do the cells come from?

Each cell comes from measurement merges of the newest commits, on 15 paths: TypeScript and Python, frameworks and plain SDKs. The newest measurement wins for a case and a path. The grid was built on 2026-10-18 00:24 from 74 measured files.

How far is each layer?

Layer	Fixed everywhere	Measured	Total
Model output syntax	24	31	33
Model output shape	14	25	29
Tool selection	5	12	13
Call structure	4	14	16
Streaming	3	10	10
Provider protocol	1	30	31
Argument hallucination	7	12	19
Claim hallucination	3	4	8
Framework orchestration	6	12	41
Tool invocation	1	11	23
Tool result	6	15	16
Outcome	0	2	2
Coexistence	11	26	26

267failure modes in the checklist

204have a measured cell

85fixed on every path where they apply

63fail on at least one path

1519/2278measured cells fixed by Provely

By layer

Model output syntax

24 / 31 / 33

Model output shape

14 / 25 / 29

Tool selection

5 / 12 / 13

Call structure

4 / 14 / 16

Streaming

3 / 10 / 10

Provider protocol

1 / 30 / 31

Argument hallucination

7 / 12 / 19

Claim hallucination

3 / 4 / 8

Framework orchestration

6 / 12 / 41

Tool invocation

1 / 11 / 23

Tool result

6 / 15 / 16

Outcome

0 / 2 / 2

Coexistence

11 / 26 / 26

Fixed everywhere / measured / total

Every failure mode, on every path

✓Fixed by Provely✕Fails○Not counted: the framework coped alone, or no fix·Recorded or out of reach?Pending: no measurement yet–Does not apply

Failure mode	TypeScript frameworks						TypeScript plain SDKs			Python frameworks				Python plain SDKs
Failure mode	LC	LG	LI	OA	VG	VS	OC	OR	AM	LC	LG	LI	OA	OC	AM
Model output syntax · 33
`FM-SYN-001` The model writes the tool call as text, and no structured call exists	○	○	○	○	○	○	·	·	·	○	○	○	○	·	·
`FM-SYN-002` One object states one member two times	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-003` An integer outside plus or minus 2^53-1 loses digits	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-004` Empty, falsy or unparseable arguments text becomes {} before the guard sees it	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-005` NaN or Infinity tokens in the arguments text	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-006` One arguments text holds two or more JSON values	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-007` smolagents passes unparseable argument text as the value of a one-input tool	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-SYN-008` A custom tool takes free text, so no JSON schema check applies	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-SYN-009` Llama 3.1 on Bedrock Converse does not escape quotes in tool input, and string values are cut short	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-010` A code fence or a sentence wraps the arguments object	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✓	✓	✕	✕
`FM-SYN-011` An unpaired surrogate in an argument string	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✕	✓	✓
`FM-SYN-012` The arguments text is a JSON string that holds JSON	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-013` The arguments text is not valid JSON	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-014` A NUL, zero-width or bidi character inside a valid string	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-015` A stray or split call marker swallows the text after it, including a real call	○	○	○	○	○	○	·	·	·	○	○	○	○	·	·
`FM-SYN-016` A trailing comma after the last member or item	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-017` Argument strings carry control tokens or native call syntax	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-018` Comments inside the argument text	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-019` Member names without quotes	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-020` Number forms that JSON does not allow	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-021` Single-quoted keys or strings	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-022` Unclosed JSON text reaches the guard	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-024` Python True, False or None in the argument text	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-025` The arguments are larger or deeper than the bounds	✕	✕	✓	✕	✕	✕	✓	✓	✕	✓	✓	✕	✕	✓	✕
`FM-SYN-026` A backslash escape that JSON does not define	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-027` A call to a tool with no parameters carries an empty arguments string	✓	✓	✓	✓	✓	✓	✓	✓	○	✓	✓	○	✓	✓	○
`FM-SYN-028` A decimal literal changes value when parsed to binary64	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-029` A number literal that overflows to Infinity	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-030` A raw newline or tab inside a string value	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-031` Argument bytes that are not valid UTF-8	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-SYN-032` The argument JSON is an array, a string, a number or null	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-033` One number arrives in another form: 1500.0, 1.5e3 or "1500"	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SYN-034` A byte order mark before the JSON text	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
Model output shape · 29
`FM-SHP-001` A tool call comes from a response that stopped at the output limit	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SHP-002` A member has the wrong JSON type, often a number or a boolean written as text	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SHP-003` A member that the schema does not declare is stripped or forwarded	✕	✕	✕	✕	✓	✓	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-SHP-004` An absent or null member takes a default that widens the effect	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SHP-005` The arguments break the declared schema, and the tool runs anyway	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SHP-006` A content filter, a safety stop or a guardrail ends or changes a tool turn	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-SHP-007` The finish reason is a failure value or a value outside the known set	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-SHP-008` Claude returns an empty end_turn after a tool result, and the run stops early	○	○	○	○	○	○	·	·	·	○	○	○	○	·	·
`FM-SHP-009` The guard unwraps a real name and arguments pair as if it were a dispatch	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SHP-010` The model refuses, and the loop reads no tool call as done	○	○	○	✕	○	○	·	·	·	○	○	○	✕	·	·
`FM-SHP-011` A call that the model only drafted inside its reasoning is executed	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-SHP-012` A date or time in another format, ambiguous, or with no offset	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SHP-013` A number in a locale format	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-SHP-014` The server turns text that was not a call into a real tool call	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SHP-015` The finish reason disagrees with the tool calls in the response	·	·	·	–	·	·	·	–	·	·	·	·	–	·	·
`FM-SHP-016` The schema that the model reads differs from the schema that validates the call	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-SHP-017` A required member is absent	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SHP-018` An array or object member arrives as a JSON string	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SHP-019` The model nests the arguments under one wrapper member	✕	✕	✕	✕	✓	✓	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-SHP-020` A __proto__ or constructor member in the arguments	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SHP-021` A pattern check passes in Python and fails in TypeScript	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-SHP-022` The response carries a block type that the client does not know, such as reasoning content first	·	·	·	·	·	·	·	·	·	·	·	·	·	·	·
`FM-SHP-023` A member name in another case, with a space, or with a look-alike letter	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SHP-024` A one-element array arrives as a bare value	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SHP-026` An enum value in another case or as a synonym	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SHP-027` The validation error that reaches the model has no detail, so retries repeat the mistake	✓	✓	✓	✓	✕	✕	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SHP-028` An empty filter, a wildcard or an 'all' value selects every record	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SHP-029` A negative or zero number passes a cap that bounds only the top	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-SHP-030` An array repeats one element, so one effect happens twice	✓	✓	✓	✕	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
Tool selection · 13
`FM-SEL-001` The model calls a tool name that the request did not declare	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SEL-002` Two tools resolve to one name after the framework rewrites the names	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-SEL-003` The model picks a valid tool for the wrong step	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SEL-004` CrewAI maps a tool name that does not exist to the closest real tool	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-SEL-005` A tool that the request did not offer in this turn still runs	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-SEL-006` The MCP tool list changes after the person confirmed it	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-SEL-007` Ollama maps an undeclared or misspelt tool name to a declared tool by substring	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SEL-008` Two namespaces hold a function with the same name	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-SEL-009` The model calls a tool that does not exist and nests real calls in it	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SEL-010` The model writes a declared tool's name in another case or format	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-SEL-012` A tool whose name starts with extract- is never executed by createAgent	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-SEL-013` A side-effecting tool that no rule names runs as a read	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-SEL-014` An MCP server's readOnlyHint or destructiveHint lowers the risk class of a tool	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
Call structure · 16
`FM-STR-001` One response holds the same side-effecting call two or more times	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-STR-002` A call id repeats an earlier id, or the server makes ids that collide	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-STR-003` The framework runs only part of the calls of one response	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-STR-005` A hosted program issues many calls in loops and in parallel	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-STR-006` A refusal arrives after a tool_use block has streamed, and the call runs	–	–	–	–	–	–	–	–	✕	–	–	–	–	–	✕
`FM-STR-007` Client-executed calls of other types skip a function-only guard	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-STR-008` A call has no id, an empty id, or an id or a name that changes between deltas	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-STR-009` After a server-side model fallback, a client tool_use of the declining model stays in the content	–	–	–	–	–	–	–	–	✕	–	–	–	–	–	✕
`FM-STR-010` Dependent calls arrive in one parallel batch	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-STR-011` Several side-effecting calls arrive in one turn, and no execution order is defined	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-STR-012` The deprecated functions shape carries a call with no id	?	?	?	–	?	?	○	–	–	?	?	?	–	○	–
`FM-STR-013` The model works on while an async call is still running	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-STR-014` n greater than 1 returns several choices, each with calls	?	?	?	–	?	?	○	–	–	?	?	?	–	○	–
`FM-STR-015` smolagents runs the other calls, then reports the step as an error because a final answer came with them	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-STR-016` A confirmed step gets no call: the server drops a call, or the model API refuses or stalls the request	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-STR-017` stop_on_first_tool returns one tool result while every call of the turn ran	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
Streaming · 10
`FM-STM-001` A call runs before its turn or its block ends	–	–	✕	–	–	✕	–	–	✕	–	–	✕	–	–	✕
`FM-STM-002` The client joins streamed call fragments by the wrong key	–	–	○	–	–	○	–	–	–	–	–	✕	–	–	–
`FM-STM-003` The stream ends with no terminal success event	–	–	✕	–	–	✕	–	–	✓	–	–	✓	–	–	✓
`FM-STM-004` A partial-JSON parser closes cut arguments, and the tool runs with the cut value	–	–	✕	–	–	✕	–	–	✓	–	–	✓	–	–	✓
`FM-STM-005` A consumer that stops reading or cancels the stream does not stop the tool	–	–	–	–	–	✕	–	–	–	–	–	–	–	–	–
`FM-STM-006` A WebSocket session cancels or steers a response after its call arrived	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-STM-007` A chunk with an empty choices array crashes the reader	–	–	✓	–	–	✓	–	–	–	–	–	✓	–	–	–
`FM-STM-008` Compatible servers stream call deltas in other shapes	–	–	✓	–	–	✕	–	–	✓	–	–	✓	–	–	✓
`FM-STM-009` Streamed tool input is invalid JSON, and the SDK stream helper throws	–	–	✓	–	–	✓	–	–	✓	–	–	✓	–	–	✓
`FM-STM-010` Streaming assembles a call differently from the non-streaming parse of the same text	–	–	✓	–	–	✓	–	–	✓	–	–	✓	–	–	✓
Provider protocol · 31
`FM-PRO-001` Gemini function-call numbers travel as doubles	?	?	?	?	?	?	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-PRO-002` The answer holds a call that its own request excluded	?	?	?	?	?	?	✕	✕	✕	–	–	–	–	✕	✕
`FM-PRO-003` The endpoint drops the tools or the forced choice, and no call arrives	?	?	?	?	?	?	·	·	·	–	–	–	–	·	·
`FM-PRO-004` An HTTP 200 answer holds only an error object	✕	✕	✕	?	✕	✕	✕	–	–	✕	✕	✕	–	✕	–
`FM-PRO-005` Control-token text inside user or system text becomes prompt structure	?	?	?	?	?	?	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-PRO-006` The server drops older messages or tokens silently when the context is full	?	?	?	?	?	?	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-PRO-007` The server rewrites argument values before the client sees them	?	?	?	?	?	?	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-PRO-008` A call id in the history has no result in the next request	?	?	?	?	?	?	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-PRO-009` The argument container differs by API: an object, a JSON string or a parameters key	✕	✕	?	○	✕	?	✕	✕	–	✕	✕	✕	✕	✕	–
`FM-PRO-010` Strict mode is off, falls back in silence, or the server ignores it	?	?	?	?	?	?	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-PRO-011` The next request echoes a malformed call, and the provider or server refuses it	?	?	?	✕	?	?	✕	✓	✕	✓	✓	✕	✕	✕	✕
`FM-PRO-012` The server fails the request because the model output does not parse	✕	✕	✕	?	✕	✕	✕	–	–	✕	✕	✕	–	✕	–
`FM-PRO-015` A hallucinated tool name with characters outside the provider pattern is replayed in history	?	?	✕	?	?	?	✕	✓	✕	✕	✕	✕	✕	✕	✕
`FM-PRO-016` A result for a computer-use or browser-use member call omits toolset_name or carries other blocks	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-PRO-017` A server-tool turn that paused with pause_turn is taken as the final answer	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-PRO-018` A tool call id from another provider or from a framework does not fit the id rule of the target API	○	○	○	?	○	○	○	–	○	○	○	○	–	○	○
`FM-PRO-019` An API version or an SDK major changes the shape that the guard reads	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-PRO-020` Cohere's tool plan goes back as text content beside the tool calls	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-PRO-021` History trimming keeps a tool output and drops its call	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-PRO-022` Text sits before the tool results, or beside them when only results are allowed	?	?	?	?	?	?	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-PRO-024` The answer to a program call drops the caller field	?	?	?	✕	?	?	–	○	–	–	–	–	✕	–	–
`FM-PRO-025` The history breaks the role or turn order that the API enforces around tool calls	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-PRO-026` The model's chat template or validator rejects the conversation after a tool call	○	○	○	○	○	✕	○	○	○	○	○	○	○	○	○
`FM-PRO-027` The reasoning state of a tool turn does not return unchanged: missing, dropped, edited or without its signature	○	○	○	○	○	○	○	○	○	○	○	○	○	○	○
`FM-PRO-028` The results of one parallel turn go back in several user messages	?	?	?	?	?	?	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-PRO-030` The server returns neither the call nor its text	○	○	○	✕	○	○	○	○	○	○	○	○	○	○	○
`FM-PRO-032` The tools are removed from a Bedrock request while the history still holds tool blocks	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-PRO-033` On Bedrock, toolResult status error works only for Nova and Claude 3 and 4, so other models cannot read a refusal flag	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-PRO-034` The chat template does not match the model's tool-call training	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-PRO-036` The model behind a name changes during the life of a contract	○	○	○	○	○	○	○	○	○	○	○	○	○	○	○
`FM-PRO-040` A router moves the run to another provider between turns	○	○	○	?	○	○	○	–	–	○	○	○	–	○	–
Argument hallucination · 19
`FM-HAL-001` A consequential value that no earlier text of the run carries	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-HAL-002` A consequential value first appears in text that an outside party wrote	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-HAL-003` The amount is in the wrong unit	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-HAL-004` The model copies a placeholder or an example value from the tool description	✓	✓	✓	✓	✓	✓	✕	✕	✕	✓	✓	✓	✓	✕	✕
`FM-HAL-005` A model-supplied argument overrides a value that the developer bound outside the schema	–	–	–	–	–	–	–	–	–	–	–	✓	–	–	–
`FM-HAL-006` The model adds an optional flag or enum value that nobody stated	✓	✓	✓	✓	✓	✓	✕	✕	✕	✓	✓	✓	✓	✕	✕
`FM-HAL-007` Strict mode makes every field required, so the model fills fields that the developer defaulted	✓	✓	✓	✓	✓	✓	✕	✕	✕	✓	✓	✓	✓	✕	✕
`FM-HAL-008` A server-side tool hides the external text, so a copied value looks like model output	–	–	–	○	–	–	–	○	–	–	–	–	○	–	–
`FM-HAL-009` An identifier of valid shape that no source carries	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-HAL-010` The currency differs from the ask or comes from an example	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-HAL-011` Two values swap roles	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-HAL-012` A framework writes tool output into a user-role text turn	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-HAL-013` A value from an earlier request in the same session	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-HAL-014` The model computes a value from sources	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-HAL-015` A system prompt that the program fills with runtime data reads as a developer constant	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-HAL-016` The user turn is text from an outside party, not from the person who authorised the run	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-HAL-017` A summary, a compaction or a handoff filter moves outside text into model-written or user text	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-HAL-018` The model resolves a relative date against the wrong day, year or time zone	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-HAL-019` A consequential value comes from an image, a PDF or audio, where provenance finds no text	✓	✓	?	?	?	?	✓	✓	✓	✓	✓	?	?	✓	✓
Claim hallucination · 8
`FM-CLM-001` The final answer states that the action is done, and no operation shows it	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-CLM-002` At the step cap the framework asks the model for a final answer, and the answer can claim the work is done	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-CLM-003` LangChain LLMToolEmulator answers each tool with model text when tools=None	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-CLM-004` The agent reads a non-terminal or refused answer as success	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-CLM-005` A nested agent's own final text goes back to the parent as a tool result	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-CLM-006` The message states the outcome before the tool runs	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-CLM-007` The model invents a result: a refund id, a confirmation number, an outcome	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-CLM-008` The model writes the tool's result, or a success claim, in the same turn as its call	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
Framework orchestration · 41
`FM-FRM-001` The run fails after side effects, and a rerun repeats them	✕	✕	✕	✕	✕	○	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-FRM-002` A resume restarts a node or a step, and a completed call runs again	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-003` A framework retry runs the same side-effecting call again	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-004` Saved state, a replayed history or a second worker runs a call or an approval that was already used	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-005` The framework parses or repairs the arguments before the guard sees them	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-006` The framework sets an unparseable call aside and ends the run as an answer	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-FRM-007` A forced tool choice stays on after the tool result, and the side effect repeats	✓	✓	✓	✓	✓	✓	✓	✕	✓	✓	✓	✓	✓	✓	✓
`FM-FRM-008` An SDK tool runner calls the functions of the developer where no funnel sees them	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-009` The tool runs a value that differs from the value that the guard checked	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-010` A durable-execution integration retries a tool activity, possibly in another process	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-011` A retry wrapper catches the human-approval interrupt, repeats the side effect and never asks the person	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-012` An approval predicate that returns a non-boolean value lets the tool run without approval	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-013` Concurrent runs of one agent share one guard and one capture	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-014` CrewAI's tool cache answers a repeated call from cache, so a second intended action never runs	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-015` One approval covers later calls of the same tool with different arguments	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-016` OpenAI Agents input guardrails run beside the agent by default, so tools can run before the tripwire	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-017` The tool's own output becomes the agent's final answer with no check	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-018` canUseTool is not consulted because a mode, a rule or a model classifier approved the call first	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-019` A resume value binds to a different interrupt than the one the person answered	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-020` A Claude Agent SDK hook that raises does not stop the tool, and canUseTool is skipped for auto-approved tools	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-021` Tool calls and results in the conversation come from the client, not from the server	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-022` Complete tool calls in a response that stopped on length or a filter never run	✓	✓	✓	✓	·	·	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-FRM-023` The loop budget ends while a decided call has no result	·	·	–	–	–	–	–	–	–	–	·	–	–	–	–
`FM-FRM-024` The instructions or tool set change between steps, so the captured ask and catalogue are stale	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-025` A tool schema uses a keyword that the checker cannot evaluate	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-026` AI SDK calls stop after the tool step, so the model never reads results or errors	–	–	–	–	·	·	–	–	–	–	–	–	–	–	–
`FM-FRM-027` The agent loop has no working step or time limit	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-FRM-028` google-genai automatic function calling skips a function call that has no args	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-029` A bundler inlines the framework, so the load hooks never fire and no tool is guarded	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-030` A framework version outside the supported range loads unguarded after one warning	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-031` The default guard posture records a refusal and runs the call	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-FRM-032` The framework loads before the hook, so the agent keeps the original, unguarded functions	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-FRM-033` The agent runs on a runtime where the one-line import cannot attach	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-034` The run context does not reach the tool, so the call has no session, no ledger and no capture	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-035` A tool runs in a worker process that never loaded the hook, or that forked the parent's state	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-036` A run with no human turn, or a conversation longer than the compiler reads, cannot be compiled	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-FRM-037` An MCP server asks the user to confirm, and the agent or a default handler answers	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-038` The developer executes a JSON plan from a structured output, and no tool call exists	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-039` The framework writes a stack trace or a raw exception message into the model context	✓	✓	✓	✓	✓	✓	·	·	·	✓	✓	✓	✓	·	·
`FM-FRM-040` A call delegates the work to a remote agent, which acts outside the guarded process	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-FRM-041` A framework binding and a patched model client both decide one tool call	✓	✓	✓	✓	✓	✓	–	–	–	✓	✓	✓	✓	–	–
Tool invocation · 23
`FM-INV-001` The model asks again for a call that already ran, or calls again after a timeout or an error	✓	✓	✓	✓	✓	✓	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-INV-002` An abort or a cancel after the request left leaves the outcome unknown	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-INV-004` A tool runs at the provider or in the browser, outside every tool body	–	–	–	?	–	–	–	·	·	–	–	–	?	–	–
`FM-INV-005` A timeout ends the wait, and the tool body keeps running	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-INV-006` A retry changes one argument, so the derived key changes	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-INV-007` The provider never receives the idempotency key of the operation	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-INV-008` The side effect committed, and the tool or the framework reports an error	?	?	?	?	?	?	✕	✕	✕	?	?	?	?	✕	✕
`FM-INV-009` Two runs act on one intent at the same time	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-INV-010` A retry arrives after the provider forgot the key	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-INV-011` A tool that starts background work reports success when the work has only started	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-INV-012` The process dies between the provider call and the action result	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-INV-013` The provider SDK retries inside one tool call	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-INV-014` One tool makes several writes and fails partway	✓	✓	✓	✓	✓	✓	·	·	·	✓	✓	✓	✓	·	·
`FM-INV-015` The SDK retries a model request whose server-side tools already acted	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-INV-016` Tool input callbacks run before the approval check and before the guarded execute	–	–	–	–	✓	✓	–	–	–	–	–	–	–	–	–
`FM-INV-017` Two deliberate identical actions share one derived key	✓	✓	✓	✓	✓	✓	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-INV-018` A hung tool has no deadline	?	?	?	?	?	?	✕	✕	✕	?	?	✕	✕	✕	✕
`FM-INV-020` The Python guard runs a sync tool on a worker thread	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-INV-021` The framework gives up on the call while the guard still waits on the plane, and the body runs later	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-INV-022` The Provely control plane is down, slow or unreachable when a call arrives	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-INV-023` A tool that streams its output runs its body after the guard has recorded the call	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-INV-024` The platform freezes or kills the function before the guard posts what happened	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-INV-025` Tool calls from an asynchronous batch result run after the run context ended	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
Tool result · 16
`FM-RES-001` A server-side tool fails inside the response, and the model goes on	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
`FM-RES-002` An MCP isError result or a framework error object reads as a success	○	○	○	○	○	○	○	○	○	○	○	○	○	○	○
`FM-RES-003` 409 because a request with the same key is still running	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-RES-004` The MCP client rejects a result after the server already ran the tool	✓	✓	✓	✓	✓	✓	✕	✕	✕	✓	✓	✓	✓	✕	✕
`FM-RES-005` The provider replays a saved answer	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-RES-006` A success status with an error inside the body	○	○	○	○	○	○	○	○	○	○	○	○	○	○	○
`FM-RES-007` The provider refuses a key that is reused with other parameters	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-RES-008` The tool returns text that is not JSON	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-RES-009` A provider body repeats a nested member	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-RES-010` The declared content type does not match the body	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-RES-011` The tool returns a value that JSON would change	✕	✕	○	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-RES-012` The tool throws on 429, 5xx, a timeout or a reset	✓	✓	✓	✓	✓	✓	✕	✕	✕	✓	✓	✓	✓	✕	✕
`FM-RES-013` A Python tool returns Decimal, datetime, bytes, UUID or a set	–	–	–	–	–	–	–	–	–	✓	✓	✓	✓	✕	✕
`FM-RES-014` The provider answers with a redirect	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-RES-015` The tool returns an array, a bare string, a number, null or nothing	○	○	○	○	○	○	○	○	○	○	○	○	○	○	○
`FM-RES-016` Control-token text inside a tool result becomes prompt structure	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
Outcome · 2
`FM-OUT-001` The provider accepted the action, and the outcome is pending, different or unknown (named only)	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
`FM-OUT-002` A batch call succeeds for some items and fails for others (named only)	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
Coexistence · 26
`FM-COX-001` A retry of the developer re-runs a write that committed, inside one tool call	✓	·	·	·	·	·	✕	✕	✕	✕	·	·	·	✕	✕
`FM-COX-002` Provely replaces or drops an idempotency key that the developer supplied	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-COX-003` A catch of the developer turns an unknown outcome into a normal result	✓	✓	✓	✓	✓	✓	✕	✕	✕	✓	✓	✓	✓	✕	✕
`FM-COX-004` The framework's error text asks the model to try again after a write that committed	✓	✓	✓	✓	✓	✓	✕	✕	✕	?	?	✓	✓	✕	✕
`FM-COX-005` Provely refuses a repeat that the developer's own key makes safe	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-COX-006` A timeout of the developer that is shorter than Provely's bound stays the bound	?	?	?	?	?	?	✓	✓	✓	?	?	?	?	✕	✓
`FM-COX-007` Provely's re-asks consume a limit that the developer set, or widen it	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-COX-008` A guardrail or an approval of the developer that refuses a call stays refused	?	?	?	✕	?	?	?	?	?	?	?	?	✕	?	?
`FM-COX-009` An abort of the developer ends the run while a write is in flight	✕	?	?	?	✕	✕	?	?	?	?	?	?	?	?	?
`FM-COX-010` A validation of the developer coerces a value before Provely reads it	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-COX-011` A tool-error handler of the developer is never invoked, because Provely answers the exception	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-COX-012` Provely changes the class of an exception that the developer's code branches on	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-COX-013` A refused call never reaches the developer's own guard in a plain loop	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-COX-014` A clean run with tool retry defences differs from the control	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-COX-015` A clean run with tool-error handler defences differs from the control	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-COX-016` A clean run with model retry defences differs from the control	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-COX-017` A clean run with argument validation defences differs from the control	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-COX-018` A clean run with timeout defences differs from the control	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-COX-019` A clean run with step and turn limits defences differs from the control	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-COX-020` A clean run with guardrail defences differs from the control	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-COX-021` A clean run with idempotency key defences differs from the control	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-COX-022` A repair of the developer after Provely's repair changes the arguments again	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
`FM-COX-023` The SDK completes truncated arguments before Provely reads them	?	?	?	?	?	?	?	?	✓	?	?	✓	?	?	✓
`FM-COX-024` A model retry gives a second answer, and Provely decides it as a duplicate	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕
`FM-COX-025` The developer's retry and Provely's re-ask multiply into a loop	✓	✓	✓	✓	✓	✓	○	○	○	✕	✕	✕	✕	✕	✕
`FM-COX-026` A defence of the developer plus Provely never hangs	✕	✕	·	✓	✓	✓	?	?	?	✕	–	–	–	?	?

Paths: LC LangChain, LG LangGraph, LI LlamaIndex, OA OpenAI Agents, VG and VS Vercel AI generate and stream, OC OpenAI chat, OR OpenAI responses, AM Anthropic messages. An invented-value case counts its guard_mode: enforce row; under the default observe, Provely only records it.