- add RunEventsService and emit run.started / run.finished from the scheduler - stream run events over SSE at GET /api/v1/scenarios/runs/events with a heartbeat - add useRunNotifications hook and NotificationToggle (opt-in bell in the side panel footer) - refresh runs lists when a run finishes - document the events stream and notifications Closes #12
13 KiB
Scenario execution
API paths below are relative to the REST prefix /api/v1 (e.g. POST /api/v1/scenarios/:id/run).
Runs
Creating a run (POST /scenarios/:id/run, the Run button, or the run_scenario MCP tool) takes an environment, and optionally a scenario version (latest by default), saveSession and a trace mode.
- The run gets one step run per step of that scenario version, each with its own copy of the step's title, code and timeout. Editing or deleting steps later doesn't change existing runs.
- The first step run starts as
pending, the rest aswaiting. - A scheduler tick (
@Interval(1000)) picks up pending runs (up to 10 at a time) and executes their steps in order. All steps of a run share one Playwright browser, context and page. - A step passes when its code returns, and fails when it throws or times out. Its return value is stored as the step output (JSON) and is available to later steps via
context.getStepOutput(). - When a step fails, the run is marked
fail, the remaining steps arecancelled, and a screenshot and the page HTML are saved to the run's files (step-<order>-failure.png/.html). - Timeouts: each step gets its own
timeoutSeconds, otherwise the scenario'stimeoutSeconds, otherwise 60 s. The run as a whole fails once it has run longer than the scenario'stimeoutSeconds(default 600 s); this is checked before each step. - With
saveSession, the run's browser is kept after the run as a saved session namedrun-<runId>(see Sessions in the UI), instead of being closed. - Runs left
in_progresswhen the server stops are markedfailon the next start.
Script runtime
Step code (execCode) and snippet code run as an async JavaScript function body with a single context argument:
async (context) => {
// your code here
}
Available on context:
context.page: PlaywrightPagecontext.browser: PlaywrightBrowserContextcontext.env: shallow copy of the environment's key/value mapcontext.getEnv(key): environment value lookup (throws if missing)context.getCredential(alias): payload of the credential assigned to the scenario underalias(parsed JSON when the data is JSON; throws if missing)context.getStepOutput(order): async (awaitit); output of a previous step by absolute order (1,2, ...) or relative (-1= previous step);nullif there is nonecontext.dumpDom(selector?): simplified DOM snapshot of the page (or of one element)context.log(...args),context.warn(...args),context.error(...args),context.debug(...args): run log entriescontext.assert(() => boolean, description, details?): runs the arrow function and requires it to returntrue. Logsdescription(as alogentry on success, as anerrorentry on failure or exception), then throws to stop the step if the function returned anything other thantrueor threw. On failure, the log entry's data holdsdetails(if given) and the exception message (aserror, if the function threw)context.runSnippet(name, ...args): runs stored snippet code with the same context; the snippet receives the extra arguments as...args.nameis the snippet alias for its latest version, oralias@<version>(e.g.login@3) to pin a specific version. Snippet start/finish/failure is loggedcontext.getScenarioFiles(opts?): list files uploaded to the scenario (limit,offset), including their path on diskcontext.downloadFile(url, opts?): fetchurland save the result as a run file (requires a real scenario run — throws when invoked ad hoc, e.g. via theexec_codeMCP tool).optsmay includemethod,headers,body,filename.urlalso accepts adata:URI (data:<mediaType>;base64,<data>ordata:<mediaType>,<percent-encoded data>) to save content generated in-script (e.g. a credential JWT) directly, without an actual network fetch.
Also in scope:
console: proxied to run logs (log/info→log,warn,error,debug)expect: Playwright'sexpect(e.g.await expect(context.page.locator("h1")).toHaveText("Done"))
Code is syntax-checked before it runs, and executes in the server process: it's trusted code, not sandboxed.
The step and snippet editors have a (?) button next to the Code label that opens a reference of this runtime in the UI. Its content is client/src/components/modals/code-help.md: keep it in sync with this section when the context API changes.
Run logs
Each run log entry (level, message, createdAt) also carries a context: where the message was generated. Contexts:
step:<N>: the step a log entry belongs to, including the automatic "Step N started / passed / failed" entries ("Step N passed" carries the step's return value asdata; "Step N failed" for an assertion failure names only the assertion, whose details are in theassertion:failentry)snippet:<alias>: entries logged whilecontext.runSnippet(alias, ...)is running, including its own automatic "started"/"finished"/"failed" entries. A snippet calling another snippet tags entries with the innermost running snippet's aliasassertion:pass/assertion:fail:context.assert()'s own log entryrun: run-level entries not tied to a step (e.g. a scenario timeout, or the Playwright trace upload notice)
context.log/warn/error (and console.log/etc.) split their arguments: strings/numbers/booleans are joined into message, while object/array arguments are kept out of it and stored as data (JSON), so message stays a stable string regardless of an object's shape. With one object argument data is that object; with more than one, data is an array of them. E.g. context.log("Saved credential as run file:", fileMeta) logs message: "Saved credential as run file:" with fileMeta as data, shown pretty-printed and separately from the message in the UI.
On the run page, a data block whose pretty-printed JSON is over 256 bytes or 5 lines starts collapsed behind a "Show data" toggle (set by RUN_LOG_DATA_COLLAPSE_BYTES / RUN_LOG_DATA_COLLAPSE_LINES and passed to the UI via /manifest.json). The "Expand data" / "Collapse data" button next to the Logs heading opens or closes all of them. "Copy logs" copies the currently filtered logs to the clipboard as plain text, with every data block expanded.
GET /scenarios/:id/runs/:runId?q=... and the exported CSV/Markdown logs (GET /scenarios/:id/runs/:runId/export) match q against both message and context.
Versioning
Scenarios and snippets are versioned, starting at version 1.
- A scenario's version goes up when its content changes: name, description, default environment, timeout, or any step change (add, edit, delete, reorder). Credential assignments and uploaded files are not versioned. Saving without changes does not create a new version.
- A snippet's version goes up when its alias, title, description or code changes.
- The scenario and snippet pages always show the latest version. Older versions are available via
GET /scenarios/:id/versions[/:version]andGET /snippets/:id/versions[/:version]. - A run executes one scenario version, chosen in the run modal (latest by default) or via
versioninPOST /scenarios/:id/run. The run keeps its own copy of that version's steps, so deleting or editing steps later doesn't change what the run shows. "Repeat run" reuses the run's version. - Step code uses the latest version of a snippet by default.
context.runSnippet("alias@3")pins version 3 instead. The version is looked up through the snippet's current alias. Pinned versions are listed under "Used Snippets" on the scenario page.
Export / import
Exports hold exactly one version and always include its version number (the latest by default, or ?version=N). Imports require version, and restore that exact version by id + version:
- If that version already exists, it is left untouched.
- Otherwise it is stored under that number, and becomes the latest if it is higher than the current latest version.
- Without an
id, a new scenario or snippet is created at the given version.
Import
POST /import (the Import page in the UI, and the Import buttons on the list pages; the import_yaml MCP tool) imports any mix of environments, credentials, snippets and scenarios.
- Send YAML with
Content-Type: application/yaml. A file is a stream of----separated documents; each document holds one exported item or a list of items. JSON is valid YAML, and a JSON body (one item or an array) is accepted too. - Every item needs a
kind:environment,credential,snippetorscenario. Snippets and scenarios also needversion(see above). - All items are validated before anything is imported, so one invalid item means nothing from that request is imported. The error lists every invalid item by its position.
- Items are imported in dependency order - environments, credentials, snippets, scenarios - regardless of their order in the file, so a scenario can reference an environment or credential from the same file.
- The response lists what was imported:
{ imported: [{ kind, id, name, version? }] }.
A scenario export (GET /scenarios/:id/export, optionally with its environment, credentials and used snippets) is in this format, one document per entity, and can be imported back as is. Query parameters: includeEnvironment=true, credentialIds and snippetIds (repeat the parameter or separate ids with commas) and version. A selected snippet is exported once per version the scenario uses: the latest for plain runSnippet("alias") calls, plus each version pinned with alias@N. Snippets used only through other snippets count as used.
---
# Environment: staging
kind: environment
id: 5b0c…
name: staging
data:
base_url: https://staging.example
---
# Scenario: Login flow
kind: scenario
id: 9e1f…
version: 3
name: Login flow
environmentId: 5b0c…
steps:
- title: Open login page
execCode: await context.page.goto(context.getEnv("base_url"));
Playwright traces
A run can record a Playwright trace of its browser context (screenshots and DOM snapshots). When the run finishes, the trace is uploaded to the run's files as trace.zip. Download it from the run detail page, then open it with npx playwright show-trace trace.zip or at https://trace.playwright.dev.
The trace mode is chosen per run: the run modal in the UI, the traceMode field of POST /scenarios/:id/run, and the traceMode argument of the run_scenario MCP tool all accept one of the values below. When it is omitted, the server's PLAYWRIGHT_TRACE_MODE setting is used (exposed via GET /scenarios/run-defaults). "Repeat run" reuses the original run's mode.
on: keep the trace of every runretain-on-failure: keep traces only for runs that failedoff(default): do not record traces
Traces are off by default because they often reach ~10 MB per run.
Traces are run artifacts, so they expire after FILE_RUN_EXPIRATION_DAYS like other run files.
Run events and browser notifications
The server publishes run lifecycle events on a server-sent events (SSE) stream, GET /api/v1/scenarios/runs/events, regardless of how the run was started (UI, REST API, MCP or the scheduler). Each message is named after its type and carries a JSON body:
{ "type": "run.finished", "runId": "...", "scenarioId": "...", "scenarioName": "Login", "status": "pass", "at": "2026-01-01T12:00:00.000Z" }
run.startedis sent when a run becomesin_progress(statusisin_progress).run.finishedis sent exactly once per run, with the finalstatus(passorfail), including runs that fail during setup, time out, or are failed on server restart.- A
pingevent is sent every 25 s to keep idle connections open through proxies.
The UI uses this stream for opt-in browser notifications: the bell button in the side panel footer asks for the notification permission and then shows "Run started / passed / failed: " notifications while the UI is open in a tab; clicking one opens the run. The choice is stored in the browser's localStorage. Browsers only offer notifications in a secure context (HTTPS or localhost), so over plain HTTP on another host the bell has no effect. When the permission is blocked, the button is disabled; re-allow it in the browser's site settings. Finished runs also refresh the runs lists immediately.
Statuses
Run status:
pendingin_progresspassfail
Run-step status:
waitingpendingin_progresspassfailcancelled
Example workflow
- Create environment
- Create credentials and snippets (optional but common)
- Create scenario
- Add ordered steps (
execCode, optionaltitleandtimeoutSeconds) - Assign credentials to scenario aliases when needed
- Run scenario
- Wait for run completion and inspect run logs/outputs
The self-test example follows these steps through the REST API, and can be imported as a complete, working scenario.