Files
liqa/docs/scenario.md
ars9 81164f8494
CI / changes (pull_request) Successful in 14s
CI / server (pull_request) Successful in 2m25s
CI / client (pull_request) Successful in 1m35s
CI / image (pull_request) Skipped
feat: export e2e tests for a chosen scenario version
- GET /scenarios/:id/export-e2e and the export_e2e_test MCP tool take
  an optional version, defaulting to the latest
- build the test project from that version's steps, name and timeout
- put the version in the zip name (<scenario>-vN-e2e-test.zip), the
  generated package name and the README heading
- the Export e2e test button exports the version being viewed
- document the e2e export in scenario.md and mcp.md
2026-10-08 18:37:33 +03:00

14 KiB

Scenario execution

API paths below are relative to the REST prefix /api/v1 (e.g. POST /api/v1/scenarios/:id/run).

Runs

Creating a run (POST /scenarios/:id/run, the Run button, or the run_scenario MCP tool) takes an environment, and optionally a scenario version (latest by default), saveSession and a trace mode.

  • The run gets one step run per step of that scenario version, each with its own copy of the step's title, code and timeout. Editing or deleting steps later doesn't change existing runs.
  • The first step run starts as pending, the rest as waiting.
  • A scheduler tick (@Interval(1000)) picks up pending runs (up to 10 at a time) and executes their steps in order. All steps of a run share one Playwright browser, context and page.
  • A step passes when its code returns, and fails when it throws or times out. Its return value is stored as the step output (JSON) and is available to later steps via context.getStepOutput().
  • When a step fails, the run is marked fail, the remaining steps are cancelled, and a screenshot and the page HTML are saved to the run's files (step-<order>-failure.png / .html).
  • Timeouts: each step gets its own timeoutSeconds, otherwise the scenario's timeoutSeconds, otherwise 60 s. The run as a whole fails once it has run longer than the scenario's timeoutSeconds (default 600 s); this is checked before each step.
  • With saveSession, the run's browser is kept after the run as a saved session named run-<runId> (see Sessions in the UI), instead of being closed.
  • Runs left in_progress when the server stops are marked fail on the next start.

Script runtime

Step code (execCode) and snippet code run as an async JavaScript function body with a single context argument:

async (context) => {
  // your code here
}

Available on context:

  • context.page: Playwright Page
  • context.browser: Playwright BrowserContext
  • context.env: shallow copy of the environment's key/value map
  • context.getEnv(key): environment value lookup (throws if missing)
  • context.getCredential(alias): payload of the credential assigned to the scenario under alias (parsed JSON when the data is JSON; throws if missing)
  • context.getStepOutput(order): async (await it); output of a previous step by absolute order (1, 2, ...) or relative (-1 = previous step); null if there is none
  • context.dumpDom(selector?): simplified DOM snapshot of the page (or of one element)
  • context.log(...args), context.warn(...args), context.error(...args), context.debug(...args): run log entries
  • context.assert(() => boolean, description, details?): runs the arrow function and requires it to return true. Logs description (as a log entry on success, as an error entry on failure or exception), then throws to stop the step if the function returned anything other than true or threw. On failure, the log entry's data holds details (if given) and the exception message (as error, if the function threw)
  • context.runSnippet(name, ...args): runs stored snippet code with the same context; the snippet receives the extra arguments as ...args. name is the snippet alias for its latest version, or alias@<version> (e.g. login@3) to pin a specific version. Snippet start/finish/failure is logged
  • context.getScenarioFiles(opts?): list files uploaded to the scenario (limit, offset), including their path on disk
  • context.downloadFile(url, opts?): fetch url and save the result as a run file (requires a real scenario run — throws when invoked ad hoc, e.g. via the exec_code MCP tool). opts may include method, headers, body, filename. url also accepts a data: URI (data:<mediaType>;base64,<data> or data:<mediaType>,<percent-encoded data>) to save content generated in-script (e.g. a credential JWT) directly, without an actual network fetch.

Also in scope:

  • console: proxied to run logs (log/info → log, warn, error, debug)
  • expect: Playwright's expect (e.g. await expect(context.page.locator("h1")).toHaveText("Done"))

Code is syntax-checked before it runs, and executes in the server process: it's trusted code, not sandboxed.

The step and snippet editors have a (?) button next to the Code label that opens a reference of this runtime in the UI. Its content is client/src/components/modals/code-help.md: keep it in sync with this section when the context API changes.

Run logs

Each run log entry (level, message, createdAt) also carries a context: where the message was generated. Contexts:

  • step:<N>: the step a log entry belongs to, including the automatic "Step N started / passed / failed" entries ("Step N passed" carries the step's return value as data; "Step N failed" for an assertion failure names only the assertion, whose details are in the assertion:fail entry)
  • snippet:<alias>: entries logged while context.runSnippet(alias, ...) is running, including its own automatic "started"/"finished"/"failed" entries. A snippet calling another snippet tags entries with the innermost running snippet's alias
  • assertion:pass / assertion:fail: context.assert()'s own log entry
  • run: run-level entries not tied to a step (e.g. a scenario timeout, or the Playwright trace upload notice)

context.log/warn/error (and console.log/etc.) split their arguments: strings/numbers/booleans are joined into message, while object/array arguments are kept out of it and stored as data (JSON), so message stays a stable string regardless of an object's shape. With one object argument data is that object; with more than one, data is an array of them. E.g. context.log("Saved credential as run file:", fileMeta) logs message: "Saved credential as run file:" with fileMeta as data, shown pretty-printed and separately from the message in the UI.

On the run page, a data block whose pretty-printed JSON is over 256 bytes or 5 lines starts collapsed behind a "Show data" toggle (set by RUN_LOG_DATA_COLLAPSE_BYTES / RUN_LOG_DATA_COLLAPSE_LINES and passed to the UI via /manifest.json). The "Expand data" / "Collapse data" button next to the Logs heading opens or closes all of them. "Copy logs" copies the currently filtered logs to the clipboard as plain text, with every data block expanded.

GET /scenarios/:id/runs/:runId?q=... and the exported CSV/Markdown logs (GET /scenarios/:id/runs/:runId/export) match q against both message and context.

Versioning

Scenarios and snippets are versioned, starting at version 1.

  • A scenario's version goes up when its content changes: name, description, default environment, timeout, or any step change (add, edit, delete, reorder). Credential assignments and uploaded files are not versioned. Saving without changes does not create a new version.
  • A snippet's version goes up when its alias, title, description or code changes.
  • The scenario and snippet pages always show the latest version. Older versions are available via GET /scenarios/:id/versions[/:version] and GET /snippets/:id/versions[/:version].
  • A run executes one scenario version, chosen in the run modal (latest by default) or via version in POST /scenarios/:id/run. The run keeps its own copy of that version's steps, so deleting or editing steps later doesn't change what the run shows. "Repeat run" reuses the run's version.
  • Step code uses the latest version of a snippet by default. context.runSnippet("alias@3") pins version 3 instead. The version is looked up through the snippet's current alias. Pinned versions are listed under "Used Snippets" on the scenario page.

Export / import

Exports hold exactly one version and always include its version number (the latest by default, or ?version=N). Imports require version, and restore that exact version by id + version:

  • If that version already exists, it is left untouched.
  • Otherwise it is stored under that number, and becomes the latest if it is higher than the current latest version.
  • Without an id, a new scenario or snippet is created at the given version.

Import

POST /import (the Import page in the UI, and the Import buttons on the list pages; the import_yaml MCP tool) imports any mix of environments, credentials, snippets and scenarios.

  • Send YAML with Content-Type: application/yaml. A file is a stream of ----separated documents; each document holds one exported item or a list of items. JSON is valid YAML, and a JSON body (one item or an array) is accepted too.
  • Every item needs a kind: environment, credential, snippet or scenario. Snippets and scenarios also need version (see above).
  • All items are validated before anything is imported, so one invalid item means nothing from that request is imported. The error lists every invalid item by its position.
  • Items are imported in dependency order - environments, credentials, snippets, scenarios - regardless of their order in the file, so a scenario can reference an environment or credential from the same file.
  • The response lists what was imported: { imported: [{ kind, id, name, version? }] }.

A scenario export (GET /scenarios/:id/export, optionally with its environment, credentials and used snippets) is in this format, one document per entity, and can be imported back as is. Query parameters: includeEnvironment=true, credentialIds and snippetIds (repeat the parameter or separate ids with commas) and version. A selected snippet is exported once per version the scenario uses: the latest for plain runSnippet("alias") calls, plus each version pinned with alias@N. Snippets used only through other snippets count as used.

---
# Environment: staging
kind: environment
id: 5b0c…
name: staging
data:
  base_url: https://staging.example
---
# Scenario: Login flow
kind: scenario
id: 9e1f…
version: 3
name: Login flow
environmentId: 5b0c…
steps:
  - title: Open login page
    execCode: await context.page.goto(context.getEnv("base_url"));

E2e test export

GET /scenarios/:id/export-e2e (the "Export e2e test" button, the export_e2e_test MCP tool) exports one scenario version as a standalone Playwright test project, zipped as <scenario>-v<N>-e2e-test.zip. It takes an optional version (default: the latest). Snippets are bundled the same way as in the scenario export: the latest version for plain runSnippet("alias") calls, plus each version pinned with alias@N. The environment, credentials and files are the scenario's current ones; they are not versioned.

Playwright traces

A run can record a Playwright trace of its browser context (screenshots and DOM snapshots). When the run finishes, the trace is uploaded to the run's files as trace.zip. Download it from the run detail page, then open it with npx playwright show-trace trace.zip or at https://trace.playwright.dev.

The trace mode is chosen per run: the run modal in the UI, the traceMode field of POST /scenarios/:id/run, and the traceMode argument of the run_scenario MCP tool all accept one of the values below. When it is omitted, the server's PLAYWRIGHT_TRACE_MODE setting is used (exposed via GET /scenarios/run-defaults). "Repeat run" reuses the original run's mode.

  • on: keep the trace of every run
  • retain-on-failure: keep traces only for runs that failed
  • off (default): do not record traces

Traces are off by default because they often reach ~10 MB per run.

Traces are run artifacts, so they expire after FILE_RUN_EXPIRATION_DAYS like other run files.

Run events and browser notifications

The server publishes run lifecycle events on a server-sent events (SSE) stream, GET /api/v1/scenarios/runs/events, regardless of how the run was started (UI, REST API, MCP or the scheduler). Each message is named after its type and carries a JSON body:

{ "type": "run.finished", "runId": "...", "scenarioId": "...", "scenarioName": "Login", "status": "pass", "at": "2026-01-01T12:00:00.000Z" }
  • run.started is sent when a run becomes in_progress (status is in_progress).
  • run.finished is sent exactly once per run, with the final status (pass or fail), including runs that fail during setup, time out, or are failed on server restart.
  • A ping event is sent every 25 s to keep idle connections open through proxies.

The UI uses this stream for opt-in browser notifications: the bell button in the side panel footer asks for the notification permission (granting it from the browser's own prompt or site settings works too) and then shows "Run started / passed / failed: " notifications while the UI is open in a tab; clicking one opens the run. The bell is highlighted while notifications are on; turning them off is stored in the browser's localStorage. Browsers only offer notifications in a secure context (HTTPS or localhost), so over plain HTTP on another host the bell has no effect. When the permission is blocked, the button is disabled; re-allow it in the browser's site settings. Finished runs also refresh the runs lists immediately.

Statuses

Run status:

  • pending
  • in_progress
  • pass
  • fail

Run-step status:

  • waiting
  • pending
  • in_progress
  • pass
  • fail
  • cancelled

Example workflow

  1. Create environment
  2. Create credentials and snippets (optional but common)
  3. Create scenario
  4. Add ordered steps (execCode, optional title and timeoutSeconds)
  5. Assign credentials to scenario aliases when needed
  6. Run scenario
  7. Wait for run completion and inspect run logs/outputs

The self-test example follows these steps through the REST API, and can be imported as a complete, working scenario.