> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sempera.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Tests and evals

> Scripted test runs before deployment, evals over completed conversations.

## Tests

Tests are deterministic, scripted scenarios run before deployment on Sempera's own battery (the same one the Sempera App's Test tab runs).

* `POST /v1/tests` with `{"agent_id": "...", "scenarios": ["..."]}` (scenarios optional) starts a background run and answers `202` with a `test_run`.
* `GET /v1/tests/{id}` reports status and, when finished, the per-scenario verdicts and the evidence they wrote.
* `GET /v1/tests?agent_id=...` lists runs.

A published version's verification gates are on the Agent (`gates`). Tests never weaken those gates; they are the same battery under a simpler surface.

## Testbed

The Testbed takes one turn of a stored conversation and replays it against the step's current instruction and, if you give one, a candidate instruction, so you can see how a prompt change would have answered real callers before you make it. It is operator work under the same gate as standards: `run_local_test` on top of `tests:read` or `tests:write`, reads included. Nothing in the Testbed edits the agent; a candidate is returned as a proposal to apply as a reviewed change.

* `GET /v1/testbed/situation?agent_id=...&conversation_id=...&turn=...` answers the `testbed_situation` at that turn: the step the conversation stood on, the step's current instruction, the original reply and whether the step has an instruction to edit. Add `step` to name the step yourself. A text-message thread has no steps and answers `400`.
* `POST /v1/testbed/runs` starts a background run and answers `202` with a `testbed_run`. Body: `agent_id`, `conversation_id`, `turn`, optional `step`, optional `candidate_prompt` (up to 8000 characters), optional `expectations` (the same shape as a standard's) and optional `across_recent` (`true` for 5 situations, or a number from 1 to 20) to replay other recent conversations that stood on the same step. At most 20 expectations across all situations. One run per agent at a time: a simulation set or test battery already running answers `409`. Accepts `Idempotency-Key`.
* `GET /v1/testbed/runs/{run_id}?agent_id=...` reports the run: its `state`, the version it ran on, each situation with the original reply beside the current and candidate results, a `summary` and, when a candidate was given, the `proposal`.
* `POST /v1/testbed/standards` saves the situation at a turn as a standard and answers `201`. Body: `agent_id`, `conversation_id`, `turn`, optional `step`, `expectations` (1 to 8, as for any standard) and optional `name`. The new standard gates promotion like any other. Accepts `Idempotency-Key`.

Another workspace's agent or run is refused as not found. The turn must be a nonnegative integer. Invalid situations or candidate instructions answer `validation_failed`; a conflicting simulation or test run answers `conflict`. Saved standards retain the promotion gates described below.

## Synthetic concurrent runs

A synthetic concurrent run walks the built agent's own conversation graph with scripted routing decisions, several sessions at once, to show how the graph behaves under load. No model, voice, tool, webhook or business system is reached (`provider_calls` is always `0`), so a receipt proves graph behavior, not voice quality or model capacity. Same gate as the Testbed: `run_local_test` on top of `tests:write` to start and `tests:read` to read.

* `POST /v1/agent-testing/tornado` runs one and answers `201` with the finished `run`. Body: `agent_id`, `synthetic` (must be `true`), optional `concurrency` (1 to 4, default 1), optional `repeats` (1 to 100, default 1) and `scenarios`, each a `name` and 1 to 20 `turns` of `next_node`, `expected_node`, `condition_met` and `caller_said`. At most 100 sessions in total.
* `GET /v1/agent-testing/tornado/{run_id}?agent_id=...` reads a receipt again: `total`, `passed`, `failed`, the `service_version` it ran on, the `agent_sha256` of the build it walked and each session's `trace`.

Runs spend credits like simulations: one simulation attempt per session is held before any session starts, the sessions that completed are charged (`credited_sessions`) and a run that fails or is cancelled gives the hold back. Too few credits answers `402 credits_exhausted` with nothing started. A script naming a step the build does not have answers `validation_failed`, a workspace already running one answers `validation_failed` too, and an unreadable service version answers `503 service_unavailable`.

## Standards

A standard is a saved regression test for ONE step of an agent's conversation. It pins that step to a fixed situation (the caller's lines, and optionally the caller's number) and says what the step must do. It belongs to the agent and is the same standard on every version, so a change that breaks the step turns it red.

Standards are operator work: every route below needs the `run_local_test` capability (operator or owner) on top of the `tests:read` or `tests:write` scope, reads included, because step names and per-attempt conversations are internals an owner surface never shows. A standard or a run in another workspace answers exactly like one that does not exist.

* `GET /v1/standards?agent_id=...` lists the agent's standards. Each carries its `revision` (a short hash of what it asserts) and `last_result`, its newest result on the agent's CURRENT version, with `current: false` when the standard was edited after that result. The list also carries `version_number`, `blocks_promotion` and, when it is true, `gate`: the sentence that says which standards are red, stale or not yet run.
* `POST /v1/standards` saves one and answers `201`. Body: `agent_id`, `name`, `step` (the step it is bound to), `situation` (`script`: 1 to 12 caller lines of up to 400 characters; optional `caller_phone`), `expectations` (1 to 8), optional `repeats` (1 to 5, default 1) and `enabled` (default true). An agent holds at most 50 standards. Accepts `Idempotency-Key`.
* `PATCH /v1/standards/{standard_id}` changes one. The body is the whole standard again, with `agent_id`.
* `DELETE /v1/standards/{standard_id}?agent_id=...` deletes one.
* `POST /v1/standards/run` with `{"agent_id": "..."}` runs every enabled standard against the agent's current version as one background job and answers `202` with a `standards_run`. Add `standard_ids` to run only those (an unknown id answers `404`), and `timeout_s` to bound the run. A `version` other than the current one answers `409`: standards run on the current version only. One run holds at most 20 standards and 20 expectations; a larger agent names which to run. Accepts `Idempotency-Key`.
* `GET /v1/standards/runs?agent_id=...` lists the stored runs, newest first (the 30 most recent are kept), each with the version it ran on, its `state`, the standards it was asked for and a result per standard.

An expectation has a `kind`:

| `kind` | Needs | Passes when |
| - | - | - |
| `judged` (default) | `intent`: the rule, in words | a wording judge decides the step followed the rule |
| `says` | `pattern`: a regular expression, case-insensitive | what the agent said at the step matches |
| `never_says` | `pattern` | what the agent said at the step does not match |
| `routes_to` | `target`: a step name | the conversation goes on to that step |
| `captures` | `target`: a variable name | the step captures that variable |

`threshold` (above 0, at most 1, default 1) is the share of attempts that must meet the expectation; an attempt nobody could decide never counts as met. A standard's verdict is `passed`, `failed`, `unknown` (a rule nobody could decide), `error` (the attempt itself broke) or `incomplete` (the run stopped early). Only `passed` is green.

**The gate.** An agent with enabled standards cannot be approved or promoted on a version where any of them has not been run and passed. A result counts only for the version it ran on and the revision it ran as: edit a standard, or publish a new version, and it must run again. An agent with no enabled standards is not affected.

## Test analytics

* `GET /v1/test-analytics?agent_id=...` reads the stored runs and answers, per step of the conversation, how many attempts reached it (`visits`), how many failed there (`failures`), the `rate`, and whether there were enough attempts to trust it (`sufficient`). `weakest` is the step with the highest failure rate among those with enough attempts and at least one failure, or `null`.
* `days` (1 to 365, default 30) sets the window and `min_samples` (1 to 1000, default 5) the attempts a step needs. A value outside the range answers `400`; it is never clamped.
* `sources` says which runs were counted and which are not (evaluation runs and whole-scenario battery runs do not score individual steps). Same capability and scope as the standards reads.

## Evals

Evals judge completed conversations against owner-defined criteria with a language model. They are analytics, never a release authority.

* `POST /v1/evals` with `{"definition": {"name", "description", "kind": "binary"|"numeric", "instructions", "applies_to"}}`
* `GET /v1/evals`, `GET /v1/evals/{id}`, `PATCH /v1/evals/{id}` (with `revision`), `DELETE /v1/evals/{id}` (archive)
* `POST /v1/evals` and `PATCH` also take optional `pass_when` and `fail_when` (up to 1000 characters each), which the judge reads beside the instructions. Every agent named in `applies_to` must be one the caller can reach.
* `POST /v1/evals/run` with `{"agent_id": "..."}` scores the agent's completed conversations that are not yet scored. Add `eval_id` to run one criterion of this workspace, and/or `call_ids` (1 to 200) to score only those calls. A run with `eval_id` answers `503` when no judging model is reachable, rather than storing `unknown` verdicts.
* `GET /v1/evals/results?agent_id=...` reads the verdicts.
* `GET /v1/evals/agents` lists this workspace's criteria as judges, each with its pass/fail/unknown counts and latest run.
* `GET /v1/evals/test-cases` (optional `agent_id`) lists every call a judge can score (preview, simulator and scripted calls) with the verdicts this workspace's criteria gave it.
* `GET /v1/evals/runs` lists the evaluation runs `POST /v1/evals/run` recorded for this workspace's agents, newest first: status (`completed`, or `partial` when a judge returned unknown), pass rate (unknown excluded), the judges and agent versions covered, who started it and how long it took. It takes `agent_id`, `status`, `since` and `until` (days, `2026-10-29`) and the standard `limit` and `cursor`. A run before this existed has no record and is not listed.
* `GET /v1/evals/runs/{run_id}` returns one run with a result per call and judge (verdict, the judge's reasoning, whether it was scored in this run or carried over from an earlier verdict), how each result differs from the previous run of the same suite (the same agent and judges), and the body to start it again with `POST /v1/evals/run`. A run of another workspace answers `404`.
* `GET /v1/evals/agent-templates` lists the judge templates Sempera ships (`sempera/judge-templates.json`).
* `POST /v1/evals/quick-run` with `{"agent_id", "template_id", "limit"}` (limit 1 to 20, default 5) scores one template against the agent's most recent calls and saves nothing; `503` when no judging model is reachable.
* `GET /v1/evals/user-templates` lists this workspace's saved suites; `POST /v1/evals/user-templates` with `{"name", "agent_id", "template_ids"}` saves one.

These runs read the call transcript. Audio judges score only production calls, below.

The console's Runs page offers Create run to owners and operators. Choose a workspace judge, then an agent it covers and the recorded calls to score. This uses `POST /v1/evals/run?workspace=<id>` with explicit `agent_id`, `eval_id` and `call_ids`, then opens the returned `run_id`. It never places a phone call. If judging, admission or permissions refuse the request, the dialog retains the refusal and no successful result is invented. Recorded versions lists the selected agent's saved versions. These are not replays: each evaluation result retains the version of the call it scored.

### Production-call judging

A judge can also be checked against stored production calls without saving a second copy of any call:

* `GET /v1/evals/{eval_id}/production-calls?agent_id=...` (`evals:read` and `calls:read`) lists up to 200 recent completed calls of an agent the judge covers, with `truncated` when more exist.
* `POST /v1/evals/{eval_id}/production-run` (`evals:write`, operator or owner) with `{"call_ids": [...]}` (1 to 10 completed calls) returns a verdict, score, rationale and reason per call. Nothing is saved and the result never satisfies a release gate.
* `POST /v1/evals/{eval_id}/align` (`evals:write`, owner) with `{"revision", "corrections": [{"call_id", "verdict": "pass"|"fail", "explanation"}]}` proposes new wording (transcript judges) or a new threshold (audio judges) at the revision you read. Nothing is saved; apply a proposal with `PATCH /v1/evals/{id}`.

Transcript judges read an imported corrected transcript when one exists, otherwise the original, and answer `503` when no independent judge is reachable. Audio judges (`mode: audio`) measure signal levels from the stored recording (16-bit PCM WAV, at most 30 minutes and 32 MiB): duration, energy, low-energy frames and clipping. They never infer meaning, and an unreadable, deleted, expired or altered recording is `unknown` with a reason. Each recording read is audited with ids only. A call recorded under Exclude from history is `404` and is never listed.

Both runs and alignment spend Eval credits: one per selected call is held before any work, a run is charged only for calls that received a pass or fail verdict, and a failed alignment is refunded. Too few credits is `402 credits_exhausted` and nothing is scored.

## Experiments

Experiments compare the published version of one agent with another verified, already-published version of that same agent. They use the existing conversation router and count the records it writes, not simulated traffic or projected results.

Reads require `agents:read`. Creating, updating and stopping require `agents:write` and the owner-only `manage_experiments` capability. Every route is workspace and agent scoped; an unknown or out-of-scope agent or experiment returns `404`.

* `GET /v2/agents/{uuid}/experiments` returns `data.experiments`, newest first.
* `POST /v2/agents/{uuid}/experiments` starts an experiment and returns `201` with `data.experiment`. Send `variant_version_id` and `traffic_percentage` (an integer from 1 through 99). Optional `name` is 1 to 80 characters, defaulting to `API experiment`. `variant_percent` is an alias for `traffic_percentage`; do not send both. The current published version, a draft, another agent's version, or a version without its verified saved artifact is refused.
* `GET /v2/agents/{uuid}/experiments/{experiment_id}` returns `data.experiment`, including its status, version identities, share and lifecycle timestamps.
* `PATCH /v2/agents/{uuid}/experiments/{experiment_id}` changes `name` and/or `traffic_percentage` (or its alias). A changed share applies to new conversations. The compared versions cannot be changed. A stopped experiment is immutable.
* `POST /v2/agents/{uuid}/experiments/{experiment_id}/stop` accepts no options and returns the stopped experiment. Repeating it is safe. New conversations return to baseline; conversations already open keep the version they started with.
* `GET /v2/agents/{uuid}/experiments/{experiment_id}/results` returns `data` containing `experiment`, `arms`, `reading`, `measure`, `dispositions`, `served_on` and `not_served`. Each arm has conversation and outcome counts, `reached_result`, `result_rate` (percent), `average_turns` and `flagged`. Empty arms have null rates and averages, not made-up scores. `reading` explains the sample size and uncertainty; it does not pick a winner. Disposition scores are not counted.

Only one experiment can run per agent (`409` for a second). A workspace can retain up to 100 experiments. Creation starts immediately. Scheduling (`run_hours`, `starts_at`, `ends_at`) and variant call quotas are not supported; these options and other unknown fields are rejected with `400`, never silently ignored. Without the customer database the routes return `503`, not an empty successful list.

The `served_on` and `not_served` fields describe the server's current channel configuration. By default only new typed conversations in the App's Try it live participate. Phone routing participates only where Sempera's existing pinned-version call service is explicitly enabled; creating an experiment does not enable it. Calls on a hosted call service, SMS and website chat do not participate. The public agent preview endpoint does not opt into experiment assignment.

Publishing a new baseline suspends assignment until the experiment is stopped. Assignment failures fall back to the published version. Experiments do not place calls, send messages, promote versions, change consent, or bypass safety checks, call caps or pinned-engine limits. The variant pins its prompt, model and settings; tools and knowledge remain current. Results remain readable after stopping and continue to reflect conversations that were already open.

## Production guardrails

Guardrails compiled into a published version are read on the Agent detail (`guardrails` in the App; the same rules the compiler enforces). Continuous monitoring of live behaviour is the evals road, run on a schedule from your side.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.