Supercheck LogoSupercheck

Agent test

Test your own AI agent with repeated runs and a pass rateEdit

An agent test calls an AI agent you run (a chat model, support bot, A2A agent or MCP tool). Replies vary, so the test sends the same prompt several times and asserts on the pass rate instead of one answer.

Agent tests run in the cloud only. Run them in the Playground, schedule them in a job or watch them with a synthetic monitor.

Try it in 1 minute

Open Create → Agent Test. The Quick start template calls a tool on DeepWiki, a free public MCP server, so it needs no key or variable.

Select Run. The test calls the tool three times and passes when every reply mentions Playwright and arrives within 20 seconds.

Open the run. The Agent results card shows the pass rate, slowest call, tool called and any failures. Token counts appear for chat agents. MCP servers report none.

Select Templates and pick one for your own agent. Each template lists the variables and secrets to add first and marks the lines to change with ← edit.

How a run works

Loading diagram...

Every agent test imports @supercheck/agent-test: Playwright's test and expect plus an agent fixture that connects to your agent, sends messages, repeats checks and grades answers. It is built into the editor (with types and autocomplete) and cloud runs.

Templates

TemplateWhat it checksNeeds
Quick start: public MCP serverA public MCP tool answers, three times in a rowNothing
Chat agent (OpenAI-compatible)One answer states a fact, fast and within a token limitAGENT_URL, AGENT_MODEL, AGENT_KEY
Multi-turn memoryThe agent remembers an earlier turnSame
JSON reply contractReplies are valid JSON with the fields your app readsSame
Calls the right toolThe agent picks the right tool with the right argumentsSame
Tool guard with pass rateA dangerous tool the agent is offered is never called, over 10 triesSame
AI-graded answerYour own judge model grades the answer against a rubricSame plus JUDGE_MODEL
Speed and token budgetEvery answer stays within time and token limitsSame
Prompt-injection resistanceCommon attacks never reveal the hidden instructionsSame
Custom HTTP agentA bot behind your own JSON API answersAGENT_URL, AGENT_KEY
A2A agentAn A2A agent repliesAGENT_URL, AGENT_KEY
MCP tool with a keyOne tool on your own MCP server answersMCP_URL, MCP_TOKEN

Add variables and secrets on the Variables page. Templates read them with getVariable(..., { required: true }), so a missing one fails with its name instead of a confusing connection error.

Example

import { test, expect } from '@supercheck/agent-test';

test('support bot states the refund policy', async ({ agent }) => {
  const bot = agent.connect({
    kind: 'openai', // 'openai' | 'http' | 'a2a' | 'mcp'
    url: getVariable('AGENT_URL'),
    apiKey: getSecret('AGENT_KEY'),
    model: getVariable('AGENT_MODEL'),
  });

  const result = await agent.repeat(10, async () => {
    const run = await bot.send('What is your refund policy?');
    expect(run.text).toContain('30 days');
    expect(run.toolNames).not.toContain('delete_account');
    expect(run.durationMs).toBeLessThan(20_000);
    expect(run.usage.totalTokens).toBeLessThan(4_000);
  });

  expect(result.passRate).toBeGreaterThanOrEqual(0.8);
});

API

APIMeaning
agent.connect(options)Returns a client. kind: "openai": chat completions at url (model, optional system, tools). kind: "http": POST url with body(message) and reply: "json.path". kind: "a2a": A2A 1.0 SendMessage over JSON-RPC (legacy: true sends the 0.3 message/send). kind: "mcp": calls one tool of an MCP server, with the message as the named argument. Common options: apiKey, headers, timeoutMs (default 30,000, max 120,000)
client.send(message, { conversation? })One turn. Returns { text, json?, toolCalls: [{ name, arguments }], toolNames, usage: { promptTokens, completionTokens, totalTokens, cost? }, durationMs, raw }
agent.repeat(n, fn, { concurrency? })Runs fn n times (1 to 20, concurrency 1 to 5, default 1). A throw inside fn is one failed attempt. It has a 200-second budget, shorter when less run time is left: when that is used up it starts no further attempt and counts each one left as failed with the message "not run: time budget used". Returns { attempts, passed, failed, passRate, failures: [{ attempt, message }] }. It does not throw; assert on passRate
agent.judge(text, rubric, { url, apiKey, model })Asks your own OpenAI-compatible model to grade the text against the rubric. Returns { pass, reason }
expectPlaywright's expect, unchanged

The run page shows an Agent results card with the pass rate, attempts, tokens, slowest attempt, tools called and the failures. Failure messages are clipped to 200 characters and may quote the agent's reply. Prompts and replies are left out of the run record, the API and MCP, but the saved Playwright report keeps the full transcript (with your project secrets, API key and credential headers redacted, also when the agent echoes them URL- or base64-encoded; values under 4 characters are not redacted, and a common value such as true also blanks matching ordinary text), like any other report content.

Limits

  • Tools are only those the agent reports. For openai they are the tool_calls the agent returns; for a2a, what the task history exposes. A black-box HTTP agent reports none.
  • A run lasts at most 5 minutes, shared by every test in a job, so keep agent tests in their own job. One test may take at most 4 minutes and one repeat() at most 200 seconds. Each send() times out after timeoutMs.
  • You pay for the calls. Calls to your agent and to your judge model are billed by their provider, not by Supercheck. AI grading uses your own judge endpoint and key (agent.judge); it does not use Supercheck AI credits.
  • Public addresses only. Cloud runs reach public addresses only: an agent on a private or internal address is unreachable. Expose it on a public URL.
  • No redirects. A redirect (HTTP 3xx) fails and names the target, so requests cannot reach an internal address or carry your headers to another host. Connect to the final URL.
  • Replies up to 5 MiB. A larger reply fails the request.
  • Remote only. supercheck test run --local and job run --local refuse agent tests.

Watch an agent with a synthetic monitor

Pick an agent test in a synthetic monitor to check your agent on a schedule. The monitor shows each run's pass rate.

Plan calls before you pick an interval. A 5-minute interval is 8,640 runs a month (30 days), so a test that repeats 10 times sends about 86,400 calls a month to your agent for each location. A 15-minute interval cuts that to 28,800.

The monitor form defaults to a failure threshold of 3 and a sev3 incident. A single failed run sends a Single failed agent run warning and opens no incident. The incident opens after 3 failed runs in a row. See Alerts.

In repositories

With the CLI, an agent test is a *.pw.ts file whose first line is // @type agent. supercheck pull writes the line and deploy sends the test as an agent test.

// @type agent
import { test, expect } from '@supercheck/agent-test';

On this page