The Defensive Layer for AI Agents

// LangChain `createAgent` over the real OpenAI client class.//// The developer writes the agent and two Provely lines: one import and one instance. The// runner removes the two lines for the control run, and no other line names Provely.

const model = new ChatOpenAI({  model: app.modelName,  apiKey: app.apiKey,  maxRetries: 0,  configuration: { baseURL: `${app.modelUrl}/v1` },});const refund = tool(async (input) => JSON.stringify(await app.refund(input)), {  name: app.tool.name,  description: app.tool.description,  schema: z.object({ charge: z.string(), amount: z.number(), reason: z.enum(app.reasons) }),});const lookup = tool(async (input) => JSON.stringify(await app.read(input)), {  name: app.readTool.name,  description: app.readTool.description,  schema: z.object({ charge: z.string() }),});

"""LangChain `create_agent` over the real OpenAI client class.The developer writes the agent and two Provely lines: one import and one instance. Therunner removes the two lines for the control run, and no other line names Provely."""

model = ChatOpenAI(    model=app.MODEL,    api_key=app.API_KEY,    base_url=f"{app.MODEL_URL}/v1",    max_retries=0,    timeout=app.CODE_TIMEOUT_S,)

// The LlamaIndex `agent` workflow over the real OpenAI client class.//// The developer writes the agent and two Provely lines: one import and one instance. The// runner removes the two lines for the control run, and no other line names Provely.

const llm = new OpenAI({  model: app.modelName,  apiKey: app.apiKey,  maxRetries: 0,  ...(app.codeTimeoutMs === undefined ? {} : { timeout: app.codeTimeoutMs }),  baseURL: `${app.modelUrl}/v1`,});const refund = tool(async (input) => app.refund(input), {  name: app.tool.name,  description: app.tool.description,  parameters: z.object({ charge: z.string(), amount: z.number(), reason: z.enum(app.reasons) }),});const lookup = tool(async (input) => app.read(input), {  name: app.readTool.name,  description: app.readTool.description,  parameters: z.object({ charge: z.string() }),});

// A plain tool loop over the OpenAI Chat Completions client.//// The developer writes the loop and two Provely lines: one import and one instance. The// loop holds no defensive code: no parse guard, no tool-name check, and no catch. The// runner removes the two Provely lines for the control run.

const client = new OpenAI({  apiKey: app.apiKey,  baseURL: `${app.modelUrl}/v1`,  maxRetries: 0,  ...(app.codeTimeoutMs === undefined ? {} : { timeout: app.codeTimeoutMs }),});const handlers = {  [app.tool.name]: (args) => app.refund(args),  [app.readTool.name]: (args) => app.read(args),};

"""A plain tool loop over the OpenAI Chat Completions client.The developer writes the loop and two Provely lines: one import and one instance. The loopholds no defensive code: no parse guard, no tool-name check, and no catch. The runnerremoves the two Provely lines for the control run."""

client = OpenAI(    api_key=app.API_KEY,    base_url=f"{app.MODEL_URL}/v1",    max_retries=0,    timeout=app.CODE_TIMEOUT_S,)

tools = [    {        "type": "function",        "function": {"name": name, "description": description, "parameters": parameters},    }    for name, description, parameters in [        (app.TOOL_NAME, app.TOOL_DESCRIPTION, app.TOOL_PARAMETERS),        (app.READ_TOOL_NAME, app.READ_TOOL_DESCRIPTION, app.READ_TOOL_PARAMETERS),    ]]

// The OpenAI Agents `Agent` over the real Responses API client.//// The developer writes the agent and two Provely lines: one import and one instance. The// runner removes the two lines for the control run, and no other line names Provely.

const refund = tool({  name: app.tool.name,  description: app.tool.description,  parameters: z.object({ charge: z.string(), amount: z.number(), reason: z.enum(app.reasons) }),  execute: async (input) => app.refund(input),  ...(app.codeTimeoutMs === undefined ? {} : { timeoutMs: app.codeTimeoutMs }),});const lookup = tool({  name: app.readTool.name,  description: app.readTool.description,  parameters: z.object({ charge: z.string() }),  execute: async (input) => app.read(input),});

"""The OpenAI Agents `Agent` over the real Responses API client.The developer writes the agent and two Provely lines: one import and one instance. Therunner removes the two lines for the control run, and no other line names Provely."""

"""The LlamaIndex `FunctionAgent` over the real OpenAI client class.The developer writes the agent and two Provely lines: one import and one instance. Therunner removes the two lines for the control run, and no other line names Provely."""

    llm = OpenAI(        model=app.MODEL,        api_key=app.API_KEY,        api_base=f"{app.MODEL_URL}/v1",        max_retries=0,        **({} if app.CODE_TIMEOUT_S is None else {"timeout": app.CODE_TIMEOUT_S}),    )

        tools=[            FunctionTool.from_defaults(fn=refund_charge, name=app.TOOL_NAME),            FunctionTool.from_defaults(fn=get_charge, name=app.READ_TOOL_NAME),        ],

// The Vercel AI SDK `streamText` over the real OpenAI client class.//// The developer writes the agent and two Provely lines: one import and one instance. The// runner removes the two lines for the control run, and no other line names Provely.

const options = {  model: openai.chat(app.modelName),  instructions: app.prompt,  prompt: app.ask,  maxRetries: 0,  stopWhen: isStepCount(6),  ...(app.codeTimeoutMs === undefined ? {} : { timeout: { toolMs: app.codeTimeoutMs } }),  tools: {    [app.tool.name]: tool({      description: app.tool.description,      inputSchema: z.object({        charge: z.string(),        amount: z.number(),        reason: z.enum(app.reasons),      }),      execute: async (input) => app.refund(input),    }),    [app.readTool.name]: tool({      description: app.readTool.description,      inputSchema: z.object({ charge: z.string() }),      execute: async (input) => app.read(input),    }),  },};

// LangChain `createAgent` with the defences of a developer.//// The developer wrote the defences below. Each one is a toggle: `ZC_DEFENCES` names the// defences that are on (`all` by default). The two Provely lines are the same as in// `apps/`, and the runner removes them for the defended control. `app.handled` counts each// run of a handler for property C4; it is test instrumentation, not a defence.//// - tool_try: the tool catches its own error and answers the error text.// - tool_error: `toolErrorMiddleware` turns a tool error into an error message.// - model_retry: `modelRetryMiddleware` retries a failed model call one time.// - tool_retry: `toolRetryMiddleware` re-runs a failed tool call up to two times.// - call_limit (opt-in): `toolCallLimitMiddleware` lets the refund run one time per run.// - client_retry: the chat client retries one time, with a timeout of 10 s.// - recursion_limit: the run stops after ZC_MAX_TURNS turns, when the case sets it.// - abort_timeout: an abort signal stops the run after ZC_ABORT_MS, when the case sets it.// - idem_key: the tool sends its own idempotency key to the provider.

const model = new ChatOpenAI({  model: app.modelName,  apiKey: app.apiKey,  maxRetries: app.defended('client_retry') ? 1 : 0,  // The grid twin states the code timeout as a signal of the run, below.  ...app.timeoutOption(undefined, app.defended('client_retry') ? 10_000 : undefined),  configuration: { baseURL: `${app.modelUrl}/v1` },});const refund = tool(  async (input) => {    const key = app.defended('idem_key') ? `refund-${input.charge}-${input.amount}` : undefined;    if (!app.defended('tool_try')) return JSON.stringify(await app.refund(input, key));    try {      return JSON.stringify(await app.refund(input, key));    } catch (error) {      app.handled('tool_try', error);      return `Error: ${error.message}`;    }  },  {    name: app.tool.name,    description: app.tool.description,    schema: z.object({      charge: z.string(),      amount: app.coerce ? z.coerce.number() : z.number(),      reason: z.enum(app.reasons),    }),  },);const lookup = tool(async (input) => JSON.stringify(await app.read(input)), {  name: app.readTool.name,  description: app.readTool.description,  schema: z.object({ charge: z.string() }),});const middleware = [  ...(app.defended('tool_error')    ? [        toolErrorMiddleware({          onError: (error) => {            app.handled('tool_error', error);            return `Error: ${error?.name ?? 'tool error'}`;          },        }),      ]    : []),  ...(app.defended('model_retry') ? [modelRetryMiddleware({ maxRetries: 1 })] : []),  ...(app.defended('tool_retry')    ? [toolRetryMiddleware({ maxRetries: 2, initialDelayMs: 0, jitter: false })]    : []),  ...(app.defended('call_limit')    ? [toolCallLimitMiddleware({ toolName: app.tool.name, runLimit: 1 })]    : []),];

"""LangChain `create_agent` with the defences of a developer.The developer wrote the defences below. Each one is a toggle: `ZC_DEFENCES` names the defencesthat are on (`all` by default). The two Provely lines are the same as in `apps/py`, and therunner removes them for the defended control. `app.handled` counts each run of a handler forproperty C4; it is test instrumentation, not a defence.- tool_try: the tool catches its own error and answers the error text.- validation_error: `handle_validation_error` answers a bad argument object with a text.- tool_error: `ToolErrorMiddleware` turns a tool error into an error message.- tool_retry: `ToolRetryMiddleware` runs a failed tool one more time.- model_retry: `ModelRetryMiddleware` retries a failed model call one time.- fallback: `ModelFallbackMiddleware` asks a second model when the first one fails.- call_limit: `ModelCallLimitMiddleware` and `ToolCallLimitMiddleware` bound the run.- client_retry: the chat client retries one time, with a timeout of 10 s.- recursion_limit: the graph stops after 12 steps, or after ZC_MAX_TURNS turns.- idem_key: the tool sends its own idempotency key to the provider."""

middleware = [    *([ToolErrorMiddleware(on_error=_tool_error)] if app.defended("tool_error") else []),    *(        [ToolRetryMiddleware(max_retries=1, initial_delay=0.01, jitter=False)]        if app.defended("tool_retry")        else []    ),    *(        [ModelRetryMiddleware(max_retries=1, initial_delay=0.01, jitter=False)]        if app.defended("model_retry")        else []    ),    *([ModelFallbackMiddleware(_model(0))] if app.defended("fallback") else []),    *(        [ModelCallLimitMiddleware(run_limit=8), ToolCallLimitMiddleware(run_limit=6)]        if app.defended("call_limit")        else []    ),]