- collapse run log data blocks whose pretty-printed JSON exceeds a byte or line limit, with a per-block toggle and an expand/collapse-all button next to the Logs heading - align data blocks and toggles with the context/message column - add a "Copy logs" button that copies the filtered logs as plain text with every data block expanded - add GET /manifest.json with the app name, version and run log data limits (RUN_LOG_DATA_COLLAPSE_BYTES / RUN_LOG_DATA_COLLAPSE_LINES) - load the manifest in the client stale-while-revalidate: localStorage copy first, refetched per page load and on tab focus after a while - show the server version in the sidebar footer, with a reload hint when it differs from the client build Closes #3
158 lines
11 KiB
Markdown
158 lines
11 KiB
Markdown
# Scenario execution
|
|
|
|
API paths below are relative to the REST prefix `/api/v1` (e.g. `POST /api/v1/scenarios/:id/run`).
|
|
|
|
## Runs
|
|
|
|
Creating a run (`POST /scenarios/:id/run`, the Run button, or the `run_scenario` MCP tool) takes an environment, and optionally a scenario version (latest by default), `saveSession` and a trace mode.
|
|
|
|
- The run gets one step run per step of that scenario version, each with its own copy of the step's title, code and timeout. Editing or deleting steps later doesn't change existing runs.
|
|
- The first step run starts as `pending`, the rest as `waiting`.
|
|
- A scheduler tick (`@Interval(1000)`) picks up pending runs (up to 10 at a time) and executes their steps in order. All steps of a run share one Playwright browser, context and page.
|
|
- A step passes when its code returns, and fails when it throws or times out. Its return value is stored as the step output (JSON) and is available to later steps via `context.getStepOutput()`.
|
|
- When a step fails, the run is marked `fail`, the remaining steps are `cancelled`, and a screenshot and the page HTML are saved to the run's files (`step-<order>-failure.png` / `.html`).
|
|
- Timeouts: each step gets its own `timeoutSeconds`, otherwise the scenario's `timeoutSeconds`, otherwise 60 s. The run as a whole fails once it has run longer than the scenario's `timeoutSeconds` (default 600 s); this is checked before each step.
|
|
- With `saveSession`, the run's browser is kept after the run as a saved session named `run-<runId>` (see Sessions in the UI), instead of being closed.
|
|
- Runs left `in_progress` when the server stops are marked `fail` on the next start.
|
|
|
|
## Script runtime
|
|
|
|
Step code (`execCode`) and snippet code run as an async JavaScript function body with a single `context` argument:
|
|
|
|
```js
|
|
async (context) => {
|
|
// your code here
|
|
}
|
|
```
|
|
|
|
Available on `context`:
|
|
|
|
- `context.page`: Playwright `Page`
|
|
- `context.browser`: Playwright `BrowserContext`
|
|
- `context.env`: shallow copy of the environment's key/value map
|
|
- `context.getEnv(key)`: environment value lookup (throws if missing)
|
|
- `context.getCredential(alias)`: payload of the credential assigned to the scenario under `alias` (parsed JSON when the data is JSON; throws if missing)
|
|
- `context.getStepOutput(order)`: async (`await` it); output of a previous step by absolute order (`1`, `2`, ...) or relative (`-1` = previous step); `null` if there is none
|
|
- `context.dumpDom(selector?)`: simplified DOM snapshot of the page (or of one element)
|
|
- `context.log(...args)`, `context.warn(...args)`, `context.error(...args)`, `context.debug(...args)`: run log entries
|
|
- `context.assert(() => boolean, description, details?)`: runs the arrow function and requires it to return `true`. Logs `description` (as a `log` entry on success, as an `error` entry on failure or exception), then throws to stop the step if the function returned anything other than `true` or threw. On failure, the log entry's data holds `details` (if given) and the exception message (as `error`, if the function threw)
|
|
- `context.runSnippet(name, ...args)`: runs stored snippet code with the same context; the snippet receives the extra arguments as `...args`. `name` is the snippet alias for its latest version, or `alias@<version>` (e.g. `login@3`) to pin a specific version. Snippet start/finish/failure is logged
|
|
- `context.getScenarioFiles(opts?)`: list files uploaded to the scenario (`limit`, `offset`), including their path on disk
|
|
- `context.downloadFile(url, opts?)`: fetch `url` and save the result as a run file (requires a real scenario run — throws when invoked ad hoc, e.g. via the `exec_code` MCP tool). `opts` may include `method`, `headers`, `body`, `filename`. `url` also accepts a `data:` URI (`data:<mediaType>;base64,<data>` or `data:<mediaType>,<percent-encoded data>`) to save content generated in-script (e.g. a credential JWT) directly, without an actual network fetch.
|
|
|
|
Also in scope:
|
|
|
|
- `console`: proxied to run logs (`log`/`info` → `log`, `warn`, `error`, `debug`)
|
|
- `expect`: Playwright's `expect` (e.g. `await expect(context.page.locator("h1")).toHaveText("Done")`)
|
|
|
|
Code is syntax-checked before it runs, and executes in the server process: it's trusted code, not sandboxed.
|
|
|
|
The step and snippet editors have a (?) button next to the Code label that opens a reference of this runtime in the UI. Its content is [client/src/components/modals/code-help.md](../client/src/components/modals/code-help.md): keep it in sync with this section when the `context` API changes.
|
|
|
|
## Run logs
|
|
|
|
Each run log entry (`level`, `message`, `createdAt`) also carries a `context`: where the message was generated. Contexts:
|
|
|
|
- `step:<N>`: the step a log entry belongs to, including the automatic "Step N started / passed / failed" entries ("Step N passed" carries the step's return value as `data`; "Step N failed" for an assertion failure names only the assertion, whose details are in the `assertion:fail` entry)
|
|
- `snippet:<alias>`: entries logged while `context.runSnippet(alias, ...)` is running, including its own automatic "started"/"finished"/"failed" entries. A snippet calling another snippet tags entries with the innermost running snippet's alias
|
|
- `assertion:pass` / `assertion:fail`: `context.assert()`'s own log entry
|
|
- `run`: run-level entries not tied to a step (e.g. a scenario timeout, or the Playwright trace upload notice)
|
|
|
|
`context.log`/`warn`/`error` (and `console.log`/etc.) split their arguments: strings/numbers/booleans are joined into `message`, while object/array arguments are kept out of it and stored as `data` (JSON), so `message` stays a stable string regardless of an object's shape. With one object argument `data` is that object; with more than one, `data` is an array of them. E.g. `context.log("Saved credential as run file:", fileMeta)` logs `message: "Saved credential as run file:"` with `fileMeta` as `data`, shown pretty-printed and separately from the message in the UI.
|
|
|
|
On the run page, a `data` block whose pretty-printed JSON is over 256 bytes or 5 lines starts collapsed behind a "Show data" toggle (set by `RUN_LOG_DATA_COLLAPSE_BYTES` / `RUN_LOG_DATA_COLLAPSE_LINES` and passed to the UI via `/manifest.json`). The "Expand data" / "Collapse data" button next to the Logs heading opens or closes all of them. "Copy logs" copies the currently filtered logs to the clipboard as plain text, with every `data` block expanded.
|
|
|
|
`GET /scenarios/:id/runs/:runId?q=...` and the exported CSV/Markdown logs (`GET /scenarios/:id/runs/:runId/export`) match `q` against both `message` and `context`.
|
|
|
|
## Versioning
|
|
|
|
Scenarios and snippets are versioned, starting at version 1.
|
|
|
|
- A scenario's version goes up when its content changes: name, description, default environment, timeout, or any step change (add, edit, delete, reorder). Credential assignments and uploaded files are not versioned. Saving without changes does not create a new version.
|
|
- A snippet's version goes up when its alias, title, description or code changes.
|
|
- The scenario and snippet pages always show the latest version. Older versions are available via `GET /scenarios/:id/versions[/:version]` and `GET /snippets/:id/versions[/:version]`.
|
|
- A run executes one scenario version, chosen in the run modal (latest by default) or via `version` in `POST /scenarios/:id/run`. The run keeps its own copy of that version's steps, so deleting or editing steps later doesn't change what the run shows. "Repeat run" reuses the run's version.
|
|
- Step code uses the latest version of a snippet by default. `context.runSnippet("alias@3")` pins version 3 instead. The version is looked up through the snippet's current alias. Pinned versions are listed under "Used Snippets" on the scenario page.
|
|
|
|
### Export / import
|
|
|
|
Exports hold exactly one version and always include its `version` number (the latest by default, or `?version=N`). Imports require `version`, and restore that exact version by id + version:
|
|
|
|
- If that version already exists, it is left untouched.
|
|
- Otherwise it is stored under that number, and becomes the latest if it is higher than the current latest version.
|
|
- Without an `id`, a new scenario or snippet is created at the given version.
|
|
|
|
## Import
|
|
|
|
`POST /import` (the Import page in the UI, and the Import buttons on the list pages; the `import_yaml` MCP tool) imports any mix of environments, credentials, snippets and scenarios.
|
|
|
|
- Send YAML with `Content-Type: application/yaml`. A file is a stream of `---`-separated documents; each document holds one exported item or a list of items. JSON is valid YAML, and a JSON body (one item or an array) is accepted too.
|
|
- Every item needs a `kind`: `environment`, `credential`, `snippet` or `scenario`. Snippets and scenarios also need `version` (see above).
|
|
- All items are validated before anything is imported, so one invalid item means nothing from that request is imported. The error lists every invalid item by its position.
|
|
- Items are imported in dependency order - environments, credentials, snippets, scenarios - regardless of their order in the file, so a scenario can reference an environment or credential from the same file.
|
|
- The response lists what was imported: `{ imported: [{ kind, id, name, version? }] }`.
|
|
|
|
A scenario export (`GET /scenarios/:id/export`, optionally with its environment, credentials and used snippets) is in this format, one document per entity, and can be imported back as is. Query parameters: `includeEnvironment=true`, `credentialIds` and `snippetIds` (repeat the parameter or separate ids with commas) and `version`. A selected snippet is exported once per version the scenario uses: the latest for plain `runSnippet("alias")` calls, plus each version pinned with `alias@N`. Snippets used only through other snippets count as used.
|
|
|
|
```yaml
|
|
---
|
|
# Environment: staging
|
|
kind: environment
|
|
id: 5b0c…
|
|
name: staging
|
|
data:
|
|
base_url: https://staging.example
|
|
---
|
|
# Scenario: Login flow
|
|
kind: scenario
|
|
id: 9e1f…
|
|
version: 3
|
|
name: Login flow
|
|
environmentId: 5b0c…
|
|
steps:
|
|
- title: Open login page
|
|
execCode: await context.page.goto(context.getEnv("base_url"));
|
|
```
|
|
|
|
## Playwright traces
|
|
|
|
A run can record a [Playwright trace](https://playwright.dev/docs/trace-viewer) of its browser context (screenshots and DOM snapshots). When the run finishes, the trace is uploaded to the run's files as `trace.zip`. Download it from the run detail page, then open it with `npx playwright show-trace trace.zip` or at https://trace.playwright.dev.
|
|
|
|
The trace mode is chosen per run: the run modal in the UI, the `traceMode` field of `POST /scenarios/:id/run`, and the `traceMode` argument of the `run_scenario` MCP tool all accept one of the values below. When it is omitted, the server's `PLAYWRIGHT_TRACE_MODE` setting is used (exposed via `GET /scenarios/run-defaults`). "Repeat run" reuses the original run's mode.
|
|
|
|
- `on`: keep the trace of every run
|
|
- `retain-on-failure`: keep traces only for runs that failed
|
|
- `off` (default): do not record traces
|
|
|
|
Traces are off by default because they often reach ~10 MB per run.
|
|
|
|
Traces are run artifacts, so they expire after `FILE_RUN_EXPIRATION_DAYS` like other run files.
|
|
|
|
## Statuses
|
|
|
|
Run status:
|
|
|
|
- `pending`
|
|
- `in_progress`
|
|
- `pass`
|
|
- `fail`
|
|
|
|
Run-step status:
|
|
|
|
- `waiting`
|
|
- `pending`
|
|
- `in_progress`
|
|
- `pass`
|
|
- `fail`
|
|
- `cancelled`
|
|
|
|
## Example workflow
|
|
|
|
1. Create environment
|
|
2. Create credentials and snippets (optional but common)
|
|
3. Create scenario
|
|
4. Add ordered steps (`execCode`, optional `title` and `timeoutSeconds`)
|
|
5. Assign credentials to scenario aliases when needed
|
|
6. Run scenario
|
|
7. Wait for run completion and inspect run logs/outputs
|