# The Defensive Layer for AI Agents

> Two lines. Provely finds your agent framework, checks each model answer and tool call, and fixes the failure itself. It works beside your own defences.

Canonical: https://provely.sh  
Last updated: 2026-10-18  

**Malformed JSON, an invented tool, a hung call, a payment made twice: one deterministic check for each failure, and an automatic fix. It works beside the defences you already wrote.**

Single quotes in the arguments of a local tool

*The model sends*

```text
get_weather({'city': 'Paris'})
```

*typescript/weather.ts · 2 lines added*

```typescript
import OpenAI from 'openai';
import { Provely } from '@provely/sdk';
new Provely();

const getWeather = ({ city }: { city: string }) => `Sunny in ${city}`;
const parameters = { type: 'object', properties: { city: { type: 'string' } } };

const reply = await new OpenAI().chat.completions.create({
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: 'What is the weather in Paris?' }],
  tools: [{ type: 'function', function: { name: 'get_weather', parameters } }],
});
const call = reply.choices[0]!.message.tool_calls![0]!;
console.log(getWeather(JSON.parse(call.function.arguments)));
```

*typescript/weather-repaired.ts · +28 lines of code by hand*

````typescript
import OpenAI from 'openai';

const getWeather = ({ city }: { city: string }) => `Sunny in ${city}`;
const parameters = { type: 'object', properties: { city: { type: 'string' } } };

function repairJson(text: string): string {
  // the model may wrap the JSON in a Markdown fence
  const fence = text.trim().match(/^```(?:json)?\s*([\s\S]*?)\s*```$/);
  const source = fence ? fence[1]! : text;
  let out = '';
  let quote: string | null = null;
  for (let i = 0; i < source.length; i += 1) {
    const char = source[i]!;
    if (quote === null) {
      // the model may use single quotes: every string gets double quotes
      if (char === "'" || char === '"') quote = char;
      out += quote === null ? char : '"';
    } else if (char === '\\') {
      // an escaped single quote is not valid JSON: keep the quote, drop the backslash
      const next = source[i + 1] ?? '';
      out += next === "'" ? "'" : char + next;
      i += 1;
    } else if (char === quote) {
      quote = null;
      out += '"';
    } else {
      // a double quote inside a single-quoted string must be escaped
      out += char === '"' ? '\\"' : char;
    }
  }
  return out;
}

const reply = await new OpenAI().chat.completions.create({
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: 'What is the weather in Paris?' }],
  tools: [{ type: 'function', function: { name: 'get_weather', parameters } }],
});
const call = reply.choices[0]!.message.tool_calls![0]!;
// the model may send arguments that are not JSON: tell the user, do not crash
try {
  console.log(getWeather(JSON.parse(repairJson(call.function.arguments))));
} catch (error) {
  console.log(`Could not read the tool arguments: ${(error as Error).message}`);
}
````

```text
$ node weather.ts
SyntaxError: Expected property name or '}' in JSON at position 1 (line 1 column 2)
exit 1
```

**Failed.** The model writes the arguments of get_weather with single quotes, so they are not JSON.

```text
$ node weather-repaired.ts
Sunny in Paris
exit 0
```

**Fixed by hand,** for 2 kinds of fault.

```text
$ node weather.ts
Sunny in Paris
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
repair_in_place · single_quoted_strings · get_weather
```

Lines of code: 0 → breaks · 5 → caught, no answer · 28 → fixes 2 kinds of fault · 2 → [Provely's checks](/defensive-coding)

A hand-written fix of these faults adds 34 lines with its comments, 28 lines of code. The counts come from a line diff of the recorded files, 2026-10-17.

## The same app, without Provely and with it.

Each fix ran twice: once with no Provely line, and once with the two lines. Pick your framework to see the code you know.

**The same file with Provely** · ts/langchain · the page shows the Provely import above the Provely call, and moves no other line

<details>
<summary>The comment at the top of the file</summary>

```typescript
// LangChain `createAgent` over the real OpenAI client class.
//
// The developer writes the agent and two Provely lines: one import and one instance. The
// runner removes the two lines for the control run, and no other line names Provely.
```

</details>

```typescript
import { tool } from '@langchain/core/tools';
import { ChatOpenAI } from '@langchain/openai';
import { createAgent } from 'langchain';
import { z } from 'zod';
import * as app from './common.mjs';

import { Provely } from '@provely/sdk';
new Provely();

```

<details>
<summary>16 lines: the model and the tools</summary>

```typescript
const model = new ChatOpenAI({
  model: app.modelName,
  apiKey: app.apiKey,
  maxRetries: 0,
  configuration: { baseURL: `${app.modelUrl}/v1` },
});
const refund = tool(async (input) => JSON.stringify(await app.refund(input)), {
  name: app.tool.name,
  description: app.tool.description,
  schema: z.object({ charge: z.string(), amount: z.number(), reason: z.enum(app.reasons) }),
});
const lookup = tool(async (input) => JSON.stringify(await app.read(input)), {
  name: app.readTool.name,
  description: app.readTool.description,
  schema: z.object({ charge: z.string() }),
});
```

</details>

```typescript
const agent = createAgent({ model, tools: [refund, lookup], systemPrompt: app.prompt });
let state;
for (let run = 0; run < app.invocations; run += 1) {
  state = await app.invoke(() =>
    agent.invoke(
      { messages: [{ role: 'user', content: app.ask }] },
      app.codeTimeoutMs === undefined ? {} : { signal: AbortSignal.timeout(app.codeTimeoutMs) },
    ),
  );
}
app.finish(String(state.messages.at(-1)?.content ?? ''));
```

### Malformed JSON, repaired in place

*The model sends*

````text
refund_charge(```json
{"charge": "ch_1051", "amount": 4200, "reason": "duplicate",}
```)
````

*Without Provely*

```text
refunds that ran: 0
final answer: (none)
exit 0
```

**Failed.** The refund never ran, and the agent gave no answer.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
repair_in_place · strip_code_fence, remove_trailing_commas · refund_charge
```

### A hung tool, bounded

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 1
final answer: (none)
still running at the bound
```

**Failed.** The tool never answered, and the run never ended.

*With Provely*

```text
refunds that ran: 1
final answer: I will check the refund later.
answered; the runner ended it
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
return_error_to_model · tool_timed_out · refund_charge
```

### A duplicate refund, refused: the money moves once

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 2
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran 2 times, and the person asked for 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
refuse · write_already_completed · refund_charge
```

### A repeat after a 502, refused

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 2
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran 2 times, and the person asked for 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
return_error_to_model · tool_raised · refund_charge
refuse · repeat_of_unknown_write · refund_charge
```

### An invented amount, refused under enforce

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 9900, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran for 9900, and the person asked for 4200.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
refuse · argument_model_invented · refund_charge
```

**The same file with Provely** · py/langchain · the page shows the Provely import above the Provely call, and moves no other line

<details>
<summary>The comment at the top of the file</summary>

```python
"""LangChain `create_agent` over the real OpenAI client class.

The developer writes the agent and two Provely lines: one import and one instance. The
runner removes the two lines for the control run, and no other line names Provely.
"""

```

</details>

```python
import json

from langchain.agents import create_agent
from langchain_core.tools import tool
from langchain_openai import ChatOpenAI

import common as app

from provely import Provely
Provely()


@tool(app.TOOL_NAME, description=app.TOOL_DESCRIPTION)
def refund_charge(charge: str, amount: float, reason: app.Reason) -> str:
    return json.dumps(app.refund(charge, amount, reason))


@tool(app.READ_TOOL_NAME, description=app.READ_TOOL_DESCRIPTION)
def get_charge(charge: str) -> str:
    return json.dumps(app.read(charge))


```

<details>
<summary>7 lines: the model and the tools</summary>

```python
model = ChatOpenAI(
    model=app.MODEL,
    api_key=app.API_KEY,
    base_url=f"{app.MODEL_URL}/v1",
    max_retries=0,
    timeout=app.CODE_TIMEOUT_S,
)
```

</details>

```python
agent = create_agent(model, tools=[refund_charge, get_charge], system_prompt=app.PROMPT)
for _run in range(app.INVOCATIONS):
    state = app.invoke(lambda: agent.invoke({"messages": [{"role": "user", "content": app.ASK}]}))
app.finish(state["messages"][-1].content)
```

### Malformed JSON, repaired in place

*The model sends*

````text
refund_charge(```json
{"charge": "ch_1051", "amount": 4200, "reason": "duplicate",}
```)
````

*Without Provely*

```text
refunds that ran: 0
final answer: (none)
exit 0
```

**Failed.** The refund never ran, and the agent gave no answer.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
repair_in_place · strip_code_fence, remove_trailing_commas · refund_charge
```

### A hung tool, bounded

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 1
final answer: (none)
still running at the bound
```

**Failed.** The tool never answered, and the run never ended.

*With Provely*

```text
refunds that ran: 1
final answer: I will check the refund later.
answered; the runner ended it
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
return_error_to_model · tool_timed_out · refund_charge
```

### An invented amount, refused under enforce

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 9900, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran for 9900, and the person asked for 4200.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
refuse · argument_model_invented · refund_charge
```

**The same file with Provely** · ts/llamaindex · the page shows the Provely import above the Provely call, and moves no other line

<details>
<summary>The comment at the top of the file</summary>

```typescript
// The LlamaIndex `agent` workflow over the real OpenAI client class.
//
// The developer writes the agent and two Provely lines: one import and one instance. The
// runner removes the two lines for the control run, and no other line names Provely.
```

</details>

```typescript
import { tool } from '@llamaindex/core/tools';
import { OpenAI } from '@llamaindex/openai';
import { agent } from '@llamaindex/workflow';
import { z } from 'zod';
import * as app from './common.mjs';

import { Provely } from '@provely/sdk';
new Provely();

```

<details>
<summary>17 lines: the model and the tools</summary>

```typescript
const llm = new OpenAI({
  model: app.modelName,
  apiKey: app.apiKey,
  maxRetries: 0,
  ...(app.codeTimeoutMs === undefined ? {} : { timeout: app.codeTimeoutMs }),
  baseURL: `${app.modelUrl}/v1`,
});
const refund = tool(async (input) => app.refund(input), {
  name: app.tool.name,
  description: app.tool.description,
  parameters: z.object({ charge: z.string(), amount: z.number(), reason: z.enum(app.reasons) }),
});
const lookup = tool(async (input) => app.read(input), {
  name: app.readTool.name,
  description: app.readTool.description,
  parameters: z.object({ charge: z.string() }),
});
```

</details>

```typescript
const workflow = agent({ tools: [refund, lookup], llm, systemPrompt: app.prompt });
let result;
for (let run = 0; run < app.invocations; run += 1)
  result = await app.invoke(() => workflow.run(app.ask));
app.finish(String(result.data?.result ?? ''));
```

### Malformed JSON, repaired in place

*The model sends*

````text
refund_charge(```json
{"charge": "ch_1051", "amount": 4200, "reason": "duplicate",}
```)
````

*Without Provely*

```text
refunds that ran: 0
final answer: (none)
exit 1
```

**Failed.** The app crashed with exit code 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
repair_in_place · strip_code_fence, remove_trailing_commas · refund_charge
```

### A wrong tool name, renamed

*The model sends*

```text
Refund_Charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 0
final answer: (none)
exit 1
```

**Failed.** The app crashed with exit code 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
repair_in_place · tool_name_case · Refund_Charge
```

### A schema error, returned to the model and fixed

*The model sends*

```text
refund_charge({"charge": "ch_1051", "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran with no amount, and the call needs one.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
return_error_to_model · arguments_schema_mismatch · refund_charge
```

### A hung tool, bounded

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 1
final answer: (none)
still running at the bound
```

**Failed.** The tool never answered, and the run never ended.

*With Provely*

```text
refunds that ran: 1
final answer: I will check the refund later.
answered; the runner ended it
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
return_error_to_model · tool_timed_out · refund_charge
```

### A duplicate refund, refused: the money moves once

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 2
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran 2 times, and the person asked for 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
refuse · write_already_completed · refund_charge
```

### A repeat after a 502, refused

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 2
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran 2 times, and the person asked for 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
return_error_to_model · tool_raised · refund_charge
refuse · repeat_of_unknown_write · refund_charge
```

### An invented amount, refused under enforce

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 9900, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran for 9900, and the person asked for 4200.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
refuse · argument_model_invented · refund_charge
```

**The same file with Provely** · ts/openai-chat · the page shows the Provely import above the Provely call, and moves no other line

<details>
<summary>The comment at the top of the file</summary>

```typescript
// A plain tool loop over the OpenAI Chat Completions client.
//
// The developer writes the loop and two Provely lines: one import and one instance. The
// loop holds no defensive code: no parse guard, no tool-name check, and no catch. The
// runner removes the two Provely lines for the control run.
```

</details>

```typescript
import OpenAI from 'openai';
import * as app from './common.mjs';

import { Provely } from '@provely/sdk';
new Provely();

```

<details>
<summary>10 lines: the model and the tools</summary>

```typescript
const client = new OpenAI({
  apiKey: app.apiKey,
  baseURL: `${app.modelUrl}/v1`,
  maxRetries: 0,
  ...(app.codeTimeoutMs === undefined ? {} : { timeout: app.codeTimeoutMs }),
});
const handlers = {
  [app.tool.name]: (args) => app.refund(args),
  [app.readTool.name]: (args) => app.read(args),
};
```

</details>

```typescript
/** One invocation of the agent: one conversation with the model. */
async function converse() {
  const messages = [
    { role: 'system', content: app.prompt },
    { role: 'user', content: app.ask },
  ];
  let final = null;
  for (let turn = 0; turn < 6 && final === null; turn += 1) {
    const completion = await client.chat.completions.create({
      model: app.modelName,
      messages,
      tools: [app.tool, app.readTool].map((item) => ({ type: 'function', function: item })),
    });
    const message = completion.choices[0].message;
    messages.push(message);
    if (!message.tool_calls?.length) final = message.content;
    for (const call of message.tool_calls ?? []) {
      const output = await handlers[call.function.name](JSON.parse(call.function.arguments));
      messages.push({ role: 'tool', tool_call_id: call.id, content: JSON.stringify(output) });
    }
  }
  return final;
}

let final = null;
for (let run = 0; run < app.invocations; run += 1) final = await app.invoke(converse);
app.finish(final);
```

### Malformed JSON, repaired in place

*The model sends*

````text
refund_charge(```json
{"charge": "ch_1051", "amount": 4200, "reason": "duplicate",}
```)
````

*Without Provely*

```text
refunds that ran: 0
final answer: (none)
exit 1
```

**Failed.** The app crashed with exit code 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
repair_in_place · strip_code_fence, remove_trailing_commas · refund_charge
```

### A wrong tool name, renamed

*The model sends*

```text
Refund_Charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 0
final answer: (none)
exit 1
```

**Failed.** The app crashed with exit code 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
repair_in_place · tool_name_case · Refund_Charge
```

### A schema error, returned to the model and fixed

*The model sends*

```text
refund_charge({"charge": "ch_1051", "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran with no amount, and the call needs one.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
return_error_to_model · arguments_schema_mismatch · refund_charge
```

### A duplicate refund, refused: the money moves once

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 2
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran 2 times, and the person asked for 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
refuse · write_already_completed · refund_charge
```

### An invented amount, refused under enforce

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 9900, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran for 9900, and the person asked for 4200.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
refuse · argument_model_invented · refund_charge
```

**The same file with Provely** · py/openai-chat · the page shows the Provely import above the Provely call, and moves no other line

<details>
<summary>The comment at the top of the file</summary>

```python
"""A plain tool loop over the OpenAI Chat Completions client.

The developer writes the loop and two Provely lines: one import and one instance. The loop
holds no defensive code: no parse guard, no tool-name check, and no catch. The runner
removes the two Provely lines for the control run.
"""

```

</details>

```python
import json

from openai import OpenAI

import common as app

from provely import Provely
Provely()

```

<details>
<summary>6 lines: the model and the tools</summary>

```python
client = OpenAI(
    api_key=app.API_KEY,
    base_url=f"{app.MODEL_URL}/v1",
    max_retries=0,
    timeout=app.CODE_TIMEOUT_S,
)
```

</details>

```python
handlers = {app.TOOL_NAME: app.refund, app.READ_TOOL_NAME: app.read}
```

<details>
<summary>10 lines: the model and the tools</summary>

```python
tools = [
    {
        "type": "function",
        "function": {"name": name, "description": description, "parameters": parameters},
    }
    for name, description, parameters in [
        (app.TOOL_NAME, app.TOOL_DESCRIPTION, app.TOOL_PARAMETERS),
        (app.READ_TOOL_NAME, app.READ_TOOL_DESCRIPTION, app.READ_TOOL_PARAMETERS),
    ]
]
```

</details>

```python


def converse():
    """One invocation of the agent: one conversation with the model."""
    messages = [{"role": "system", "content": app.PROMPT}, {"role": "user", "content": app.ASK}]
    for _turn in range(6):
        completion = client.chat.completions.create(model=app.MODEL, messages=messages, tools=tools)
        message = completion.choices[0].message
        messages.append(message.model_dump(exclude_none=True))
        if not message.tool_calls:
            return message.content
        for call in message.tool_calls:
            output = handlers[call.function.name](**json.loads(call.function.arguments))
            messages.append(
                {"role": "tool", "tool_call_id": call.id, "content": json.dumps(output)}
            )
    return None


final = None
for _run in range(app.INVOCATIONS):
    final = app.invoke(converse)
app.finish(final)
```

### Malformed JSON, repaired in place

*The model sends*

````text
refund_charge(```json
{"charge": "ch_1051", "amount": 4200, "reason": "duplicate",}
```)
````

*Without Provely*

```text
refunds that ran: 0
final answer: (none)
exit 1
```

**Failed.** The app crashed with exit code 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
repair_in_place · strip_code_fence, remove_trailing_commas · refund_charge
```

### A wrong tool name, renamed

*The model sends*

```text
Refund_Charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 0
final answer: (none)
exit 1
```

**Failed.** The app crashed with exit code 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
repair_in_place · tool_name_case · Refund_Charge
```

### A schema error, returned to the model and fixed

*The model sends*

```text
refund_charge({"charge": "ch_1051", "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 0
final answer: (none)
exit 1
```

**Failed.** The app crashed with exit code 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
return_error_to_model · arguments_schema_mismatch · refund_charge
```

### A duplicate refund, refused: the money moves once

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 2
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran 2 times, and the person asked for 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
refuse · write_already_completed · refund_charge
```

### An invented amount, refused under enforce

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 9900, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran for 9900, and the person asked for 4200.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
refuse · argument_model_invented · refund_charge
```

**The same file with Provely** · ts/openai-agents · the page shows the Provely import above the Provely call, and moves no other line

<details>
<summary>The comment at the top of the file</summary>

```typescript
// The OpenAI Agents `Agent` over the real Responses API client.
//
// The developer writes the agent and two Provely lines: one import and one instance. The
// runner removes the two lines for the control run, and no other line names Provely.
```

</details>

```typescript
import { Agent, run, setDefaultOpenAIClient, setTracingDisabled, tool } from '@openai/agents';
import OpenAI from 'openai';
import { z } from 'zod';
import * as app from './common.mjs';

import { Provely } from '@provely/sdk';
new Provely();

setTracingDisabled(true);
setDefaultOpenAIClient(
  new OpenAI({ apiKey: app.apiKey, baseURL: `${app.modelUrl}/v1`, maxRetries: 0 }),
);
```

<details>
<summary>13 lines: the model and the tools</summary>

```typescript
const refund = tool({
  name: app.tool.name,
  description: app.tool.description,
  parameters: z.object({ charge: z.string(), amount: z.number(), reason: z.enum(app.reasons) }),
  execute: async (input) => app.refund(input),
  ...(app.codeTimeoutMs === undefined ? {} : { timeoutMs: app.codeTimeoutMs }),
});
const lookup = tool({
  name: app.readTool.name,
  description: app.readTool.description,
  parameters: z.object({ charge: z.string() }),
  execute: async (input) => app.read(input),
});
```

</details>

```typescript
const agent = new Agent({
  name: 'refunds',
  instructions: app.prompt,
  model: app.modelName,
  tools: [refund, lookup],
});
let result;
for (let turn = 0; turn < app.invocations; turn += 1)
  result = await app.invoke(() => run(agent, app.ask));
app.finish(String(result.finalOutput ?? ''));
```

### A wrong tool name, renamed

*The model sends*

```text
Refund_Charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 0
final answer: (none)
exit 1
```

**Failed.** The app crashed with exit code 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
repair_in_place · tool_name_case · Refund_Charge
```

### A hung tool, bounded

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 1
final answer: (none)
still running at the bound
```

**Failed.** The tool never answered, and the run never ended.

*With Provely*

```text
refunds that ran: 1
final answer: I will check the refund later.
answered; the runner ended it
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
return_error_to_model · tool_timed_out · refund_charge
```

### A duplicate refund, refused: the money moves once

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 2
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran 2 times, and the person asked for 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
refuse · write_already_completed · refund_charge
```

### A repeat after a 502, refused

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 2
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran 2 times, and the person asked for 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
return_error_to_model · tool_raised · refund_charge
refuse · repeat_of_unknown_write · refund_charge
```

### An invented amount, refused under enforce

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 9900, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran for 9900, and the person asked for 4200.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
refuse · argument_model_invented · refund_charge
```

**The same file with Provely** · py/openai-agents · the page shows the Provely import above the Provely call, and moves no other line

<details>
<summary>The comment at the top of the file</summary>

```python
"""The OpenAI Agents `Agent` over the real Responses API client.

The developer writes the agent and two Provely lines: one import and one instance. The
runner removes the two lines for the control run, and no other line names Provely.
"""

```

</details>

```python
import json

from agents import Agent, Runner, function_tool, set_default_openai_client, set_tracing_disabled
from openai import AsyncOpenAI

import common as app

from provely import Provely
Provely()


@function_tool(name_override=app.TOOL_NAME, description_override=app.TOOL_DESCRIPTION)
def refund_charge(charge: str, amount: float, reason: app.Reason) -> str:
    return json.dumps(app.refund(charge, amount, reason))


@function_tool(name_override=app.READ_TOOL_NAME, description_override=app.READ_TOOL_DESCRIPTION)
def get_charge(charge: str) -> str:
    return json.dumps(app.read(charge))


set_tracing_disabled(True)
set_default_openai_client(
    AsyncOpenAI(
        api_key=app.API_KEY,
        base_url=f"{app.MODEL_URL}/v1",
        max_retries=0,
        **({} if app.CODE_TIMEOUT_S is None else {"timeout": app.CODE_TIMEOUT_S}),
    )
)
agent = Agent(
    name="refunds", instructions=app.PROMPT, model=app.MODEL, tools=[refund_charge, get_charge]
)
for _run in range(app.INVOCATIONS):
    result = app.invoke(lambda: Runner.run_sync(agent, app.ASK))
app.finish(result.final_output)
```

### A wrong tool name, renamed

*The model sends*

```text
Refund_Charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 0
final answer: (none)
exit 1
```

**Failed.** The app crashed with exit code 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
repair_in_place · tool_name_case · Refund_Charge
```

### A hung tool, bounded

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 1
final answer: (none)
still running at the bound
```

**Failed.** The tool never answered, and the run never ended.

*With Provely*

```text
refunds that ran: 1
final answer: I will check the refund later.
answered; the runner ended it
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
return_error_to_model · tool_timed_out · refund_charge
```

### A duplicate refund, refused: the money moves once

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 2
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran 2 times, and the person asked for 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
refuse · write_already_completed · refund_charge
```

### A repeat after a 502, refused

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 2
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran 2 times, and the person asked for 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
return_error_to_model · tool_raised · refund_charge
refuse · repeat_of_unknown_write · refund_charge
```

### An invented amount, refused under enforce

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 9900, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran for 9900, and the person asked for 4200.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
refuse · argument_model_invented · refund_charge
```

**The same file with Provely** · py/llamaindex · the page shows the Provely import above the Provely call, and moves no other line

<details>
<summary>The comment at the top of the file</summary>

```python
"""The LlamaIndex `FunctionAgent` over the real OpenAI client class.

The developer writes the agent and two Provely lines: one import and one instance. The
runner removes the two lines for the control run, and no other line names Provely.
"""

```

</details>

```python
import asyncio
import json

from llama_index.core.agent.workflow import FunctionAgent
from llama_index.core.tools import FunctionTool
from llama_index.llms.openai import OpenAI

import common as app

from provely import Provely
Provely()


def refund_charge(charge: str, amount: float, reason: app.Reason) -> str:
    """Refund a charge."""
    return json.dumps(app.refund(charge, amount, reason))


def get_charge(charge: str) -> str:
    """Read a charge."""
    return json.dumps(app.read(charge))


async def main() -> None:
    # A workflow binds to the running event loop, so the agent is built inside it.
```

<details>
<summary>7 lines: the model and the tools</summary>

```python
    llm = OpenAI(
        model=app.MODEL,
        api_key=app.API_KEY,
        api_base=f"{app.MODEL_URL}/v1",
        max_retries=0,
        **({} if app.CODE_TIMEOUT_S is None else {"timeout": app.CODE_TIMEOUT_S}),
    )
```

</details>

```python
    agent = FunctionAgent(
```

<details>
<summary>4 lines: the model and the tools</summary>

```python
        tools=[
            FunctionTool.from_defaults(fn=refund_charge, name=app.TOOL_NAME),
            FunctionTool.from_defaults(fn=get_charge, name=app.READ_TOOL_NAME),
        ],
```

</details>

```python
        llm=llm,
        system_prompt=app.PROMPT,
    )
    result = None
    for _run in range(app.INVOCATIONS):
        result = await app.invoke_async(lambda: agent.run(user_msg=app.ASK))
    app.finish(result)


asyncio.run(main())
```

### A hung tool, bounded

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 1
final answer: (none)
still running at the bound
```

**Failed.** The tool never answered, and the run never ended.

*With Provely*

```text
refunds that ran: 1
final answer: I will check the refund later.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
return_error_to_model · tool_timed_out · refund_charge
```

### A duplicate refund, refused: the money moves once

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 2
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran 2 times, and the person asked for 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
refuse · write_already_completed · refund_charge
```

### A repeat after a 502, refused

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 2
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran 2 times, and the person asked for 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
return_error_to_model · tool_raised · refund_charge
refuse · repeat_of_unknown_write · refund_charge
```

### An invented amount, refused under enforce

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 9900, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran for 9900, and the person asked for 4200.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
refuse · argument_model_invented · refund_charge
```

### A cut stream, asked again

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 0
final answer: (none)
exit 1
```

**Failed.** The app crashed with exit code 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
refuse · stream_ended_without_success · refund_charge
```

**The same file with Provely** · ts/vercel-ai-stream · the page shows the Provely import above the Provely call, and moves no other line

<details>
<summary>The comment at the top of the file</summary>

```typescript
// The Vercel AI SDK `streamText` over the real OpenAI client class.
//
// The developer writes the agent and two Provely lines: one import and one instance. The
// runner removes the two lines for the control run, and no other line names Provely.
```

</details>

```typescript
import { createOpenAI } from '@ai-sdk/openai';
import { isStepCount, streamText, tool } from 'ai';
import { z } from 'zod';
import * as app from './common.mjs';

import { Provely } from '@provely/sdk';
new Provely();

const openai = createOpenAI({ apiKey: app.apiKey, baseURL: `${app.modelUrl}/v1` });
```

<details>
<summary>24 lines: the model and the tools</summary>

```typescript
const options = {
  model: openai.chat(app.modelName),
  instructions: app.prompt,
  prompt: app.ask,
  maxRetries: 0,
  stopWhen: isStepCount(6),
  ...(app.codeTimeoutMs === undefined ? {} : { timeout: { toolMs: app.codeTimeoutMs } }),
  tools: {
    [app.tool.name]: tool({
      description: app.tool.description,
      inputSchema: z.object({
        charge: z.string(),
        amount: z.number(),
        reason: z.enum(app.reasons),
      }),
      execute: async (input) => app.refund(input),
    }),
    [app.readTool.name]: tool({
      description: app.readTool.description,
      inputSchema: z.object({ charge: z.string() }),
      execute: async (input) => app.read(input),
    }),
  },
};
```

</details>

```typescript
let text = '';
for (let run = 0; run < app.invocations; run += 1) {
  text = await app.invoke(async () => {
    let answer = '';
    for await (const delta of streamText(options).textStream) answer += delta;
    return answer;
  });
}
app.finish(text);
```

### A hung tool, bounded

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 1
final answer: (none)
still running at the bound
```

**Failed.** The tool never answered, and the run never ended.

*With Provely*

```text
refunds that ran: 1
final answer: I will check the refund later.
answered; the runner ended it
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
return_error_to_model · tool_timed_out · refund_charge
```

### A duplicate refund, refused: the money moves once

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 2
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran 2 times, and the person asked for 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
refuse · write_already_completed · refund_charge
```

### A repeat after a 502, refused

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 2
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran 2 times, and the person asked for 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
return_error_to_model · tool_raised · refund_charge
refuse · repeat_of_unknown_write · refund_charge
```

### An invented amount, refused under enforce

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 9900, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran for 9900, and the person asked for 4200.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
refuse · argument_model_invented · refund_charge
```

**The same file with Provely** · ts/langchain · the page shows the Provely import above the Provely call, and moves no other line

<details>
<summary>The comment at the top of the file</summary>

```typescript
// LangChain `createAgent` with the defences of a developer.
//
// The developer wrote the defences below. Each one is a toggle: `ZC_DEFENCES` names the
// defences that are on (`all` by default). The two Provely lines are the same as in
// `apps/`, and the runner removes them for the defended control. `app.handled` counts each
// run of a handler for property C4; it is test instrumentation, not a defence.
//
// - tool_try: the tool catches its own error and answers the error text.
// - tool_error: `toolErrorMiddleware` turns a tool error into an error message.
// - model_retry: `modelRetryMiddleware` retries a failed model call one time.
// - tool_retry: `toolRetryMiddleware` re-runs a failed tool call up to two times.
// - call_limit (opt-in): `toolCallLimitMiddleware` lets the refund run one time per run.
// - client_retry: the chat client retries one time, with a timeout of 10 s.
// - recursion_limit: the run stops after ZC_MAX_TURNS turns, when the case sets it.
// - abort_timeout: an abort signal stops the run after ZC_ABORT_MS, when the case sets it.
// - idem_key: the tool sends its own idempotency key to the provider.
```

</details>

```typescript
import { tool } from '@langchain/core/tools';
import { ChatOpenAI } from '@langchain/openai';
import {
  createAgent,
  modelRetryMiddleware,
  toolCallLimitMiddleware,
  toolErrorMiddleware,
  toolRetryMiddleware,
} from 'langchain';
import { z } from 'zod';
import * as app from './common.mjs';

import { Provely } from '@provely/sdk';
new Provely();

```

<details>
<summary>53 lines: the model and the tools</summary>

```typescript
const model = new ChatOpenAI({
  model: app.modelName,
  apiKey: app.apiKey,
  maxRetries: app.defended('client_retry') ? 1 : 0,
  // The grid twin states the code timeout as a signal of the run, below.
  ...app.timeoutOption(undefined, app.defended('client_retry') ? 10_000 : undefined),
  configuration: { baseURL: `${app.modelUrl}/v1` },
});
const refund = tool(
  async (input) => {
    const key = app.defended('idem_key') ? `refund-${input.charge}-${input.amount}` : undefined;
    if (!app.defended('tool_try')) return JSON.stringify(await app.refund(input, key));
    try {
      return JSON.stringify(await app.refund(input, key));
    } catch (error) {
      app.handled('tool_try', error);
      return `Error: ${error.message}`;
    }
  },
  {
    name: app.tool.name,
    description: app.tool.description,
    schema: z.object({
      charge: z.string(),
      amount: app.coerce ? z.coerce.number() : z.number(),
      reason: z.enum(app.reasons),
    }),
  },
);
const lookup = tool(async (input) => JSON.stringify(await app.read(input)), {
  name: app.readTool.name,
  description: app.readTool.description,
  schema: z.object({ charge: z.string() }),
});
const middleware = [
  ...(app.defended('tool_error')
    ? [
        toolErrorMiddleware({
          onError: (error) => {
            app.handled('tool_error', error);
            return `Error: ${error?.name ?? 'tool error'}`;
          },
        }),
      ]
    : []),
  ...(app.defended('model_retry') ? [modelRetryMiddleware({ maxRetries: 1 })] : []),
  ...(app.defended('tool_retry')
    ? [toolRetryMiddleware({ maxRetries: 2, initialDelayMs: 0, jitter: false })]
    : []),
  ...(app.defended('call_limit')
    ? [toolCallLimitMiddleware({ toolName: app.tool.name, runLimit: 1 })]
    : []),
];
```

</details>

```typescript
const agent = createAgent({ model, tools: [refund, lookup], systemPrompt: app.prompt, middleware });
let state;
for (let run = 0; run < app.invocations; run += 1) {
  state = await app.invoke(() =>
    agent.invoke(
      { messages: [{ role: 'user', content: app.ask }] },
      {
        ...app.signalOption(
          app.codeTimeoutMs,
          app.defended('abort_timeout') ? app.abortMs : undefined,
        ),
        ...(app.defended('recursion_limit') && app.maxTurns !== undefined
          ? { recursionLimit: 2 * app.maxTurns }
          : {}),
      },
    ),
  );
}
app.finish(String(state.messages.at(-1)?.content ?? ''));
```

### Provely works beside your own defences

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 2
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran 2 times, and the person asked for 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
return_error_to_model · tool_raised · refund_charge
refuse · repeat_of_unknown_write · refund_charge
```

**The same file with Provely** · py/langchain · the page shows the Provely import above the Provely call, and moves no other line

<details>
<summary>The comment at the top of the file</summary>

```python
"""LangChain `create_agent` with the defences of a developer.

The developer wrote the defences below. Each one is a toggle: `ZC_DEFENCES` names the defences
that are on (`all` by default). The two Provely lines are the same as in `apps/py`, and the
runner removes them for the defended control. `app.handled` counts each run of a handler for
property C4; it is test instrumentation, not a defence.

- tool_try: the tool catches its own error and answers the error text.
- validation_error: `handle_validation_error` answers a bad argument object with a text.
- tool_error: `ToolErrorMiddleware` turns a tool error into an error message.
- tool_retry: `ToolRetryMiddleware` runs a failed tool one more time.
- model_retry: `ModelRetryMiddleware` retries a failed model call one time.
- fallback: `ModelFallbackMiddleware` asks a second model when the first one fails.
- call_limit: `ModelCallLimitMiddleware` and `ToolCallLimitMiddleware` bound the run.
- client_retry: the chat client retries one time, with a timeout of 10 s.
- recursion_limit: the graph stops after 12 steps, or after ZC_MAX_TURNS turns.
- idem_key: the tool sends its own idempotency key to the provider.
"""

```

</details>

```python
import json

from langchain.agents import create_agent
from langchain.agents.middleware import (
    ModelCallLimitMiddleware,
    ModelFallbackMiddleware,
    ModelRetryMiddleware,
    ToolCallLimitMiddleware,
    ToolErrorMiddleware,
    ToolRetryMiddleware,
)
from langchain_core.tools import tool
from langchain_openai import ChatOpenAI

import common as app

from provely import Provely
Provely()


def _validation_error(error: Exception) -> str:
    app.handled("validation_error", error)
    return f"Error: the arguments are not valid ({type(error).__name__})."


@tool(app.TOOL_NAME, description=app.TOOL_DESCRIPTION)
def refund_charge(charge: str, amount: float, reason: app.Reason) -> str:
    key = f"refund-{charge}-{amount}" if app.defended("idem_key") else None
    if not app.defended("tool_try"):
        return json.dumps(app.refund(charge, amount, reason, key))
    try:
        return json.dumps(app.refund(charge, amount, reason, key))
    except Exception as error:
        app.handled("tool_try", error)
        return f"Error: {error}"


if app.defended("validation_error"):
    refund_charge.handle_validation_error = _validation_error


@tool(app.READ_TOOL_NAME, description=app.READ_TOOL_DESCRIPTION)
def get_charge(charge: str) -> str:
    return json.dumps(app.read(charge))


def _model(retries: int) -> ChatOpenAI:
    return ChatOpenAI(
        model=app.MODEL,
        api_key=app.API_KEY,
        base_url=f"{app.MODEL_URL}/v1",
        max_retries=retries,
        # A timeout that the agent code states wins. Else the client retry sets 10 s.
        timeout=app.CODE_TIMEOUT_S or (10 if app.defended("client_retry") else None),
    )


def _tool_error(error: Exception) -> str:
    app.handled("tool_error", error)
    return f"Error: {type(error).__name__}"


model = _model(1 if app.defended("client_retry") else 0)
```

<details>
<summary>19 lines: the model and the tools</summary>

```python
middleware = [
    *([ToolErrorMiddleware(on_error=_tool_error)] if app.defended("tool_error") else []),
    *(
        [ToolRetryMiddleware(max_retries=1, initial_delay=0.01, jitter=False)]
        if app.defended("tool_retry")
        else []
    ),
    *(
        [ModelRetryMiddleware(max_retries=1, initial_delay=0.01, jitter=False)]
        if app.defended("model_retry")
        else []
    ),
    *([ModelFallbackMiddleware(_model(0))] if app.defended("fallback") else []),
    *(
        [ModelCallLimitMiddleware(run_limit=8), ToolCallLimitMiddleware(run_limit=6)]
        if app.defended("call_limit")
        else []
    ),
]
```

</details>

```python
agent = create_agent(
    model, tools=[refund_charge, get_charge], system_prompt=app.PROMPT, middleware=middleware
)
# Two graph steps make one turn: the model, then the tools. A case can set the turn limit.
limit = 12 if app.MAX_TURNS is None else 2 * app.MAX_TURNS
config = {"recursion_limit": limit} if app.defended("recursion_limit") else {}
for _run in range(app.INVOCATIONS):
    state = app.invoke(
        lambda: agent.invoke({"messages": [{"role": "user", "content": app.ASK}]}, config=config)
    )
app.finish(state["messages"][-1].content)
```

### Provely works beside your own defences

*The model sends*

```text
refund_charge({"charge": "ch_1051", "amount": 4200, "reason": "duplicate"})
```

*Without Provely*

```text
refunds that ran: 2
final answer: The refund is done.
exit 0
```

**Failed.** The refund ran 2 times, and the person asked for 1.

*With Provely*

```text
refunds that ran: 1
final answer: The refund is done.
exit 0
```

**Auto fixed by Provely**

*The decision line that Provely wrote*

```text
return_error_to_model · tool_raised · refund_charge
refuse · repeat_of_unknown_write · refund_charge
```

### Provely finds, with no adapter

`LangChain` `LangGraph` `LlamaIndex` `OpenAI Agents` `Vercel AI` `OpenAI SDK` `Anthropic SDK`

Recorded on 2026-10-17.

[See the measured coverage of every failure mode](/defensive-coding). [See every framework Provely finds](/frameworks). [Read how a check answers each failure](/how-it-works).

## Start in one minute.

The SDKs are in private preview. [Ask for access](/signin).

*terminal*

```text
# 1. With access, install the SDK package that your access names.
$ npm install <the SDK package of your access>
# 2. Set your API key. Make it at app.provely.sh/account/api-keys, or run provely login once on your machine.
$ export PROVELY_API_KEY=…
# 3. Add the two Provely lines to your agent, then run it.
$ node agent.mjs
# 4. Optional: see what Provely fixed. Each decision is a line in the local log.
$ tail -n 1 .provely/decisions.jsonl
{"call_id":"call_0_0","tool":"refund_charge","decision":"repair_in_place","check":"fm.argument_repair@1.0.0","code":"arguments_unparseable","rules":["single_quoted_strings"]}
```

Get your API key: sign in at [app.provely.sh/account/api-keys](https://app.provely.sh/account/api-keys), open API keys, and create a key. The dashboard shows the key once. Keep it in an environment variable, never in code. Or run `provely login` once on your machine: the SDKs then read the key that it stores.

## Keep the defences you wrote.

Your retries, validators, guardrails and timeouts stay. Six properties state how Provely works beside them, and the grid tests each one on every path.

`C1 · Clean runs do not change` `C2 · Never worse` `C3 · One decision for each call` `C4 · Your handler still runs` `C5 · The stricter choice wins` `C6 · No loop`

TypeScript: 21 of 27 (rules cases not yet measured) cases hold every property on 9 paths (measured 2026-10-17). Python: 24 of 27 (1 passes under guard_mode: enforce; observe records only; rules cases not yet measured) cases hold every property on 6 paths (measured 2026-10-17).

[Read the six coexistence properties and the measured result](/coexistence)