Skip to main content

​
Tests

Tests are deterministic, scripted scenarios run before deployment on Sempera’s own battery (the same one the Sempera App’s Test tab runs).
  • POST /v1/tests with {"agent_id": "...", "scenarios": ["..."]} (scenarios optional) starts a background run and answers 202 with a test_run.
  • GET /v1/tests/{id} reports status and, when finished, the per-scenario verdicts and the evidence they wrote.
  • GET /v1/tests?agent_id=... lists runs.
A published version’s verification gates are on the Agent (gates). Tests never weaken those gates; they are the same battery under a simpler surface.

​
Testbed

The Testbed takes one turn of a stored conversation and replays it against the step’s current instruction and, if you give one, a candidate instruction, so you can see how a prompt change would have answered real callers before you make it. It is operator work under the same gate as standards: run_local_test on top of tests:read or tests:write, reads included. Nothing in the Testbed edits the agent; a candidate is returned as a proposal to apply as a reviewed change.
  • GET /v1/testbed/situation?agent_id=...&conversation_id=...&turn=... answers the testbed_situation at that turn: the step the conversation stood on, the step’s current instruction, the original reply and whether the step has an instruction to edit. Add step to name the step yourself. A text-message thread has no steps and answers 400.
  • POST /v1/testbed/runs starts a background run and answers 202 with a testbed_run. Body: agent_id, conversation_id, turn, optional step, optional candidate_prompt (up to 8000 characters), optional expectations (the same shape as a standard’s) and optional across_recent (true for 5 situations, or a number from 1 to 20) to replay other recent conversations that stood on the same step. At most 20 expectations across all situations. One run per agent at a time: a simulation set or test battery already running answers 409. Accepts Idempotency-Key.
  • GET /v1/testbed/runs/{run_id}?agent_id=... reports the run: its state, the version it ran on, each situation with the original reply beside the current and candidate results, a summary and, when a candidate was given, the proposal.
  • POST /v1/testbed/standards saves the situation at a turn as a standard and answers 201. Body: agent_id, conversation_id, turn, optional step, expectations (1 to 8, as for any standard) and optional name. The new standard gates promotion like any other. Accepts Idempotency-Key.
Another workspace’s agent or run is refused as not found. The turn must be a nonnegative integer. Invalid situations or candidate instructions answer validation_failed; a conflicting simulation or test run answers conflict. Saved standards retain the promotion gates described below.

​
Synthetic concurrent runs

A synthetic concurrent run walks the built agent’s own conversation graph with scripted routing decisions, several sessions at once, to show how the graph behaves under load. No model, voice, tool, webhook or business system is reached (provider_calls is always 0), so a receipt proves graph behavior, not voice quality or model capacity. Same gate as the Testbed: run_local_test on top of tests:write to start and tests:read to read.
  • POST /v1/agent-testing/tornado runs one and answers 201 with the finished run. Body: agent_id, synthetic (must be true), optional concurrency (1 to 4, default 1), optional repeats (1 to 100, default 1) and scenarios, each a name and 1 to 20 turns of next_node, expected_node, condition_met and caller_said. At most 100 sessions in total.
  • GET /v1/agent-testing/tornado/{run_id}?agent_id=... reads a receipt again: total, passed, failed, the service_version it ran on, the agent_sha256 of the build it walked and each session’s trace.
Runs spend credits like simulations: one simulation attempt per session is held before any session starts, the sessions that completed are charged (credited_sessions) and a run that fails or is cancelled gives the hold back. Too few credits answers 402 credits_exhausted with nothing started. A script naming a step the build does not have answers validation_failed, a workspace already running one answers validation_failed too, and an unreadable service version answers 503 service_unavailable.

​
Standards

A standard is a saved regression test for ONE step of an agent’s conversation. It pins that step to a fixed situation (the caller’s lines, and optionally the caller’s number) and says what the step must do. It belongs to the agent and is the same standard on every version, so a change that breaks the step turns it red. Standards are operator work: every route below needs the run_local_test capability (operator or owner) on top of the tests:read or tests:write scope, reads included, because step names and per-attempt conversations are internals an owner surface never shows. A standard or a run in another workspace answers exactly like one that does not exist.
  • GET /v1/standards?agent_id=... lists the agent’s standards. Each carries its revision (a short hash of what it asserts) and last_result, its newest result on the agent’s CURRENT version, with current: false when the standard was edited after that result. The list also carries version_number, blocks_promotion and, when it is true, gate: the sentence that says which standards are red, stale or not yet run.
  • POST /v1/standards saves one and answers 201. Body: agent_id, name, step (the step it is bound to), situation (script: 1 to 12 caller lines of up to 400 characters; optional caller_phone), expectations (1 to 8), optional repeats (1 to 5, default 1) and enabled (default true). An agent holds at most 50 standards. Accepts Idempotency-Key.
  • PATCH /v1/standards/{standard_id} changes one. The body is the whole standard again, with agent_id.
  • DELETE /v1/standards/{standard_id}?agent_id=... deletes one.
  • POST /v1/standards/run with {"agent_id": "..."} runs every enabled standard against the agent’s current version as one background job and answers 202 with a standards_run. Add standard_ids to run only those (an unknown id answers 404), and timeout_s to bound the run. A version other than the current one answers 409: standards run on the current version only. One run holds at most 20 standards and 20 expectations; a larger agent names which to run. Accepts Idempotency-Key.
  • GET /v1/standards/runs?agent_id=... lists the stored runs, newest first (the 30 most recent are kept), each with the version it ran on, its state, the standards it was asked for and a result per standard.
An expectation has a kind: threshold (above 0, at most 1, default 1) is the share of attempts that must meet the expectation; an attempt nobody could decide never counts as met. A standard’s verdict is passed, failed, unknown (a rule nobody could decide), error (the attempt itself broke) or incomplete (the run stopped early). Only passed is green. The gate. An agent with enabled standards cannot be approved or promoted on a version where any of them has not been run and passed. A result counts only for the version it ran on and the revision it ran as: edit a standard, or publish a new version, and it must run again. An agent with no enabled standards is not affected.

​
Test analytics

  • GET /v1/test-analytics?agent_id=... reads the stored runs and answers, per step of the conversation, how many attempts reached it (visits), how many failed there (failures), the rate, and whether there were enough attempts to trust it (sufficient). weakest is the step with the highest failure rate among those with enough attempts and at least one failure, or null.
  • days (1 to 365, default 30) sets the window and min_samples (1 to 1000, default 5) the attempts a step needs. A value outside the range answers 400; it is never clamped.
  • sources says which runs were counted and which are not (evaluation runs and whole-scenario battery runs do not score individual steps). Same capability and scope as the standards reads.

​
Evals

Evals judge completed conversations against owner-defined criteria with a language model. They are analytics, never a release authority.
  • POST /v1/evals with {"definition": {"name", "description", "kind": "binary"|"numeric", "instructions", "applies_to"}}
  • GET /v1/evals, GET /v1/evals/{id}, PATCH /v1/evals/{id} (with revision), DELETE /v1/evals/{id} (archive)
  • POST /v1/evals and PATCH also take optional pass_when and fail_when (up to 1000 characters each), which the judge reads beside the instructions. Every agent named in applies_to must be one the caller can reach.
  • POST /v1/evals/run with {"agent_id": "..."} scores the agent’s completed conversations that are not yet scored. Add eval_id to run one criterion of this workspace, and/or call_ids (1 to 200) to score only those calls. A run with eval_id answers 503 when no judging model is reachable, rather than storing unknown verdicts.
  • GET /v1/evals/results?agent_id=... reads the verdicts.
  • GET /v1/evals/agents lists this workspace’s criteria as judges, each with its pass/fail/unknown counts and latest run.
  • GET /v1/evals/test-cases (optional agent_id) lists every call a judge can score (preview, simulator and scripted calls) with the verdicts this workspace’s criteria gave it.
  • GET /v1/evals/runs lists the evaluation runs POST /v1/evals/run recorded for this workspace’s agents, newest first: status (completed, or partial when a judge returned unknown), pass rate (unknown excluded), the judges and agent versions covered, who started it and how long it took. It takes agent_id, status, since and until (days, 2026-10-29) and the standard limit and cursor. A run before this existed has no record and is not listed.
  • GET /v1/evals/runs/{run_id} returns one run with a result per call and judge (verdict, the judge’s reasoning, whether it was scored in this run or carried over from an earlier verdict), how each result differs from the previous run of the same suite (the same agent and judges), and the body to start it again with POST /v1/evals/run. A run of another workspace answers 404.
  • GET /v1/evals/agent-templates lists the judge templates Sempera ships (sempera/judge-templates.json).
  • POST /v1/evals/quick-run with {"agent_id", "template_id", "limit"} (limit 1 to 20, default 5) scores one template against the agent’s most recent calls and saves nothing; 503 when no judging model is reachable.
  • GET /v1/evals/user-templates lists this workspace’s saved suites; POST /v1/evals/user-templates with {"name", "agent_id", "template_ids"} saves one.
These runs read the call transcript. Audio judges score only production calls, below. The console’s Runs page offers Create run to owners and operators. Choose a workspace judge, then an agent it covers and the recorded calls to score. This uses POST /v1/evals/run?workspace=<id> with explicit agent_id, eval_id and call_ids, then opens the returned run_id. It never places a phone call. If judging, admission or permissions refuse the request, the dialog retains the refusal and no successful result is invented. Recorded versions lists the selected agent’s saved versions. These are not replays: each evaluation result retains the version of the call it scored.

​
Production-call judging

A judge can also be checked against stored production calls without saving a second copy of any call:
  • GET /v1/evals/{eval_id}/production-calls?agent_id=... (evals:read and calls:read) lists up to 200 recent completed calls of an agent the judge covers, with truncated when more exist.
  • POST /v1/evals/{eval_id}/production-run (evals:write, operator or owner) with {"call_ids": [...]} (1 to 10 completed calls) returns a verdict, score, rationale and reason per call. Nothing is saved and the result never satisfies a release gate.
  • POST /v1/evals/{eval_id}/align (evals:write, owner) with {"revision", "corrections": [{"call_id", "verdict": "pass"|"fail", "explanation"}]} proposes new wording (transcript judges) or a new threshold (audio judges) at the revision you read. Nothing is saved; apply a proposal with PATCH /v1/evals/{id}.
Transcript judges read an imported corrected transcript when one exists, otherwise the original, and answer 503 when no independent judge is reachable. Audio judges (mode: audio) measure signal levels from the stored recording (16-bit PCM WAV, at most 30 minutes and 32 MiB): duration, energy, low-energy frames and clipping. They never infer meaning, and an unreadable, deleted, expired or altered recording is unknown with a reason. Each recording read is audited with ids only. A call recorded under Exclude from history is 404 and is never listed. Both runs and alignment spend Eval credits: one per selected call is held before any work, a run is charged only for calls that received a pass or fail verdict, and a failed alignment is refunded. Too few credits is 402 credits_exhausted and nothing is scored.

​
Experiments

Experiments compare the published version of one agent with another verified, already-published version of that same agent. They use the existing conversation router and count the records it writes, not simulated traffic or projected results. Reads require agents:read. Creating, updating and stopping require agents:write and the owner-only manage_experiments capability. Every route is workspace and agent scoped; an unknown or out-of-scope agent or experiment returns 404.
  • GET /v2/agents/{uuid}/experiments returns data.experiments, newest first.
  • POST /v2/agents/{uuid}/experiments starts an experiment and returns 201 with data.experiment. Send variant_version_id and traffic_percentage (an integer from 1 through 99). Optional name is 1 to 80 characters, defaulting to API experiment. variant_percent is an alias for traffic_percentage; do not send both. The current published version, a draft, another agent’s version, or a version without its verified saved artifact is refused.
  • GET /v2/agents/{uuid}/experiments/{experiment_id} returns data.experiment, including its status, version identities, share and lifecycle timestamps.
  • PATCH /v2/agents/{uuid}/experiments/{experiment_id} changes name and/or traffic_percentage (or its alias). A changed share applies to new conversations. The compared versions cannot be changed. A stopped experiment is immutable.
  • POST /v2/agents/{uuid}/experiments/{experiment_id}/stop accepts no options and returns the stopped experiment. Repeating it is safe. New conversations return to baseline; conversations already open keep the version they started with.
  • GET /v2/agents/{uuid}/experiments/{experiment_id}/results returns data containing experiment, arms, reading, measure, dispositions, served_on and not_served. Each arm has conversation and outcome counts, reached_result, result_rate (percent), average_turns and flagged. Empty arms have null rates and averages, not made-up scores. reading explains the sample size and uncertainty; it does not pick a winner. Disposition scores are not counted.
Only one experiment can run per agent (409 for a second). A workspace can retain up to 100 experiments. Creation starts immediately. Scheduling (run_hours, starts_at, ends_at) and variant call quotas are not supported; these options and other unknown fields are rejected with 400, never silently ignored. Without the customer database the routes return 503, not an empty successful list. The served_on and not_served fields describe the server’s current channel configuration. By default only new typed conversations in the App’s Try it live participate. Phone routing participates only where Sempera’s existing pinned-version call service is explicitly enabled; creating an experiment does not enable it. Calls on a hosted call service, SMS and website chat do not participate. The public agent preview endpoint does not opt into experiment assignment. Publishing a new baseline suspends assignment until the experiment is stopped. Assignment failures fall back to the published version. Experiments do not place calls, send messages, promote versions, change consent, or bypass safety checks, call caps or pinned-engine limits. The variant pins its prompt, model and settings; tools and knowledge remain current. Results remain readable after stopping and continue to reflect conversations that were already open.

​
Production guardrails

Guardrails compiled into a published version are read on the Agent detail (guardrails in the App; the same rules the compiler enforces). Continuous monitoring of live behaviour is the evals road, run on a schedule from your side.