The false-completion benchmark is the set of cases where an agent action looks complete and is not. Every integration must pass every case, with zero false VERIFIED results, before Provely signs it.

The oracle of each case is independent from the contract under test. A harness simulator plays the provider. The cases come from the provider semantics, the chaos cases, and the version rules.

step 1step 2step 3step 4VERIFIEDReport it, and keep the receipt.step 1step 2step 3step 4PENDINGWait. A retry here refunds twice.step 1step 2step 3?UNVERIFIABLERetry the check, then the step.None of the three is a guess. Each one names the evidence it rests on.
The benchmark asks for the right answer, not for a passing answer.
In words
  • VERIFIED: the provider agrees that the outcome happened.
  • PENDING: the action can still succeed, so the run waits.
  • CONTRADICTED: reality differs from the promise, so a person decides.

Which cases does the benchmark require?

13 cases. 3 of them carry the class "false success": a wrong subject or amount, a duplicate side effect, and a matching state that already existed. An integration runs every case before Provely signs it.

Benchmark version 1.0.0. The required result of each case.
CaseScenarioRequired resultClass
terminal_successAction accepted, terminal success confirmedVERIFIEDpositive
still_transitionalAction accepted, provider still transitionalPENDINGprogress
terminal_failureProvider reports terminal failureFAILEDfailure
wrong_subjectCorrect resource, wrong amount or subjectCONTRADICTEDfalse success
duplicate_effectDuplicate side effect for one operationCONTRADICTEDfalse success
stale_readbackReadback stale inside the consistency windowPENDINGprogress
evidence_timeoutEvidence unavailable until the timeoutUNVERIFIABLEinsufficient
pre_existing_statePre-existing matching state, no causal linkMust not verifyfalse success
webhook_duplicateWebhook duplicatedNo duplicate transitionevents
webhook_out_of_orderWebhook out of orderReconciled by provider semanticsevents
timeout_after_commitTimeout after the provider committedVERIFIED after readback, never FAILEDchaos
five_xx_after_execution5xx after the provider executedVERIFIED after readback, never a retrychaos
version_unsupportedProvider version outside valid_forUNVERIFIABLE with version_unsupportedversions

Which case classes did the last runs cover?

The case classes of the last conformance run, by service.
Case classAnthropic Claude APIGitHub REST APIGoogle Calendar APIGoogle Docs APIGoogle Drive APIHubSpot CRM APIJira Cloud platform REST APILinear GraphQL APIOpenAI Platform APIPayPal REST APIResendTwilio SendGrid Mail APIShopify Admin APISlack Web APISquareStripe
duplicate side effect2 of 23 of 33 of 32 of 23 of 33 of 32 of 23 of 3not run2 of 25 of 53 of 32 of 22 of 23 of 32 of 2
error after execution2 of 23 of 33 of 32 of 23 of 33 of 33 of 33 of 33 of 34 of 44 of 43 of 32 of 22 of 23 of 32 of 2
error before execution2 of 23 of 33 of 32 of 23 of 33 of 33 of 33 of 33 of 34 of 44 of 43 of 32 of 22 of 23 of 32 of 2
evidence unavailable2 of 23 of 33 of 32 of 23 of 33 of 33 of 33 of 33 of 34 of 45 of 53 of 32 of 22 of 23 of 32 of 2
idempotent retrynot runnot runnot runnot runnot runnot runnot runnot runnot run4 of 44 of 4not runnot runnot run3 of 32 of 2
pre existing state2 of 23 of 33 of 32 of 23 of 33 of 33 of 33 of 33 of 34 of 44 of 43 of 32 of 22 of 23 of 33 of 3
stale readback2 of 23 of 33 of 32 of 23 of 33 of 33 of 33 of 33 of 34 of 44 of 43 of 32 of 22 of 23 of 32 of 2
still transitional2 of 23 of 33 of 32 of 23 of 33 of 33 of 33 of 33 of 34 of 44 of 43 of 32 of 22 of 23 of 32 of 2
terminal failure2 of 21 of 1not runnot runnot runnot runnot runnot run3 of 32 of 24 of 43 of 32 of 2not run3 of 32 of 2
terminal success3 of 33 of 33 of 32 of 23 of 34 of 44 of 43 of 33 of 34 of 44 of 43 of 32 of 22 of 23 of 32 of 2
timeout after commit2 of 23 of 33 of 32 of 23 of 33 of 33 of 33 of 33 of 34 of 44 of 43 of 32 of 22 of 23 of 32 of 2
version mismatch2 of 23 of 33 of 32 of 23 of 33 of 33 of 33 of 33 of 34 of 44 of 43 of 32 of 22 of 23 of 32 of 2
webhook duplicatenot run3 of 33 of 3not run3 of 33 of 33 of 33 of 33 of 34 of 44 of 43 of 32 of 22 of 23 of 32 of 2
webhook out of ordernot run3 of 33 of 3not run3 of 33 of 33 of 33 of 33 of 34 of 44 of 43 of 32 of 22 of 23 of 32 of 2
wrong amountnot runnot runnot runnot runnot run2 of 2not runnot runnot run4 of 4not runnot run2 of 2not run3 of 32 of 2
wrong resourcenot runnot runnot runnot runnot runnot runnot runnot runnot runnot runnot runnot runnot runnot run1 of 1not run
wrong subject2 of 23 of 33 of 32 of 23 of 34 of 44 of 43 of 33 of 32 of 24 of 43 of 32 of 22 of 23 of 32 of 2
wrong terminal state1 of 11 of 13 of 3not run2 of 23 of 31 of 12 of 22 of 21 of 1not run1 of 1not runnot run3 of 3not run

What are the results per integration?

The table reads the conformance summary of each signed integration manifest. An integration with one critical false VERIFIED is rejected. The numbers change when an integration is recompiled.

ServiceIntegration versionCasesPassedCritical false VERIFIEDCertificationLast runSource
Anthropic Claude API0.1.026260Provisional2026-09-05T12:00:00Zsigned package
GitHub REST API0.1.041410Provisional2026-09-05T12:00:00Zsigned package
Google Calendar API0.1.042420Provisional2026-09-05T12:00:00Zsigned package
Google Docs API0.1.022220Provisional2026-09-05T12:00:00Zsigned package
Google Drive API0.1.041410Provisional2026-09-05T12:00:00Zsigned package
HubSpot CRM API0.1.046460Provisional2026-09-05T12:00:00Zsigned package
Jira Cloud platform REST API0.1.041410Provisional2026-09-05T12:00:00Zsigned package
Linear GraphQL API0.1.041410Provisional2026-09-05T12:00:00Zsigned package
OpenAI Platform API0.1.041410Provisional2026-09-05T12:00:00Zsigned package
PayPal REST API0.1.059590Provisional2026-09-05T12:00:00Zsigned package
Resend0.1.062620Provisional2026-09-05T12:00:00Zsigned package
Twilio SendGrid Mail API0.1.043430Provisional2026-09-05T12:00:00Zsigned package
Shopify Admin API0.1.030300Provisional2026-09-05T12:00:00Zsigned package
Slack Web API0.1.026260Provisional2026-09-05T12:00:00Zsigned package
Square0.1.052520Provisional2026-09-05T12:00:00Zsigned package
Stripe0.1.033330Official2026-09-05T12:00:00Zsigned package

How is the benchmark scored?

One false VERIFIED rejects the integration outright. The rest of the evidence decides one of four certification levels.

Explain: the four certification levels
  • A false VERIFIED on any case rejects the integration. There is no partial credit.
  • A rule that no source states keeps the integration at a low level. So does evidence that comes only from the action response.
  • Provisional is the first level, then Generated draft and Verified. Official needs every critical test to pass and every uncertainty to be reviewed by a person.
  • A level states evidence only. It never states who built the package. The manifest states the publisher, and each integration page names it.

Can I run the benchmark against my own contract?

Yes. The harness package runs the conformance matrix against any contract with the simulator or with record-and-replay fixtures. The oracle must not be the contract under test.

Where is the machine-readable result?

The /facts endpoint publishes the current cases, levels, and certification of every integration as JSON-LD.