- snapshot scenario and snippet content into version tables on every
content change; live tables keep holding the latest version
- record the scenario version on each run and copy its steps into the
run steps, so runs execute and show the version they were created with
- keep run step history when a step is deleted (FK set null)
- pin a snippet version with runSnippet('alias@N')
- add version picker to the run modal and version info across the UI
- add version list/get endpoints and MCP tools
- include version in exports and require it on import, restoring the
exact version by id + version
120 lines
5.9 KiB
Markdown
120 lines
5.9 KiB
Markdown
# Scenario execution
|
|
|
|
When a scenario run is created, the server creates one `ScenarioRunStep` per scenario step.
|
|
|
|
- Step runs are ordered by `step.order`.
|
|
- First step run starts as `pending`; the rest start as `waiting`.
|
|
- Scheduler tick (`@Interval(1000)`) picks pending runs and processes steps in order.
|
|
- Steps in the same run share one Playwright browser/context/page.
|
|
|
|
## Execution model
|
|
|
|
Current scheduler behavior is exec-centric:
|
|
|
|
- `execCode` is required for a step to pass.
|
|
- `validateCode` is optional and runs after `execCode`.
|
|
- Legacy step type metadata may still appear in MCP payloads, but run execution is not branched by step type.
|
|
|
|
## Script runtime
|
|
|
|
`execCode` and `validateCode` execute as async JavaScript with a single `context` argument:
|
|
|
|
```js
|
|
async (context) => {
|
|
// your code here
|
|
}
|
|
```
|
|
|
|
Available in scope:
|
|
|
|
- `context.page`: Playwright `Page`
|
|
- `context.browser`: Playwright `BrowserContext`
|
|
- `context.env`: shallow copy of environment key/value map
|
|
- `context.getEnv(key)`: required environment value lookup (throws if missing)
|
|
- `context.getCredential(alias)`: credential payload assigned to the scenario alias
|
|
- `context.getStepOutput(order)`: prior step output by absolute order (`1`, `2`, ...) or relative (`-1` = previous step)
|
|
- `context.dumpDom(selector?)`: simplified DOM snapshot
|
|
- `context.log(...args)`, `context.warn(...args)`, `context.error(...args)`: structured step logs
|
|
- `context.assert(() => boolean, description)`: runs the arrow function and requires it to return `true`. Logs `description` (as a `log` entry on success, as an `error` entry on failure or exception), then throws to stop the step if the function returned anything other than `true` or threw
|
|
- `context.runSnippet(name, ...args)`: execute stored snippet code with the same context. `name` is the snippet alias for its latest version, or `alias@<version>` (e.g. `login@3`) to pin a specific version
|
|
- `context.getScenarioFiles(opts?)`: list files uploaded to the scenario (`limit`, `offset`)
|
|
- `context.downloadFile(url, opts?)`: fetch `url` and save the result as a run artifact (requires a real scenario run — throws when invoked ad hoc, e.g. via the `exec_code` MCP tool). `opts` may include `method`, `headers`, `body`, `filename`. `url` also accepts a `data:` URI (`data:<mediaType>;base64,<data>` or `data:<mediaType>,<percent-encoded data>`) to save content generated in-script (e.g. a credential JWT) directly, without an actual network fetch.
|
|
- `console`: proxied to run logs (`log`, `warn`, `error`, etc.)
|
|
|
|
`validateCode` additionally receives `result` in scope, which is the value returned by `execCode`.
|
|
|
|
## Versioning
|
|
|
|
Scenarios and snippets are versioned, starting at version 1.
|
|
|
|
- A scenario's version goes up when its content changes: name, description, default environment, timeout, or any step change (add, edit, delete, reorder). Credential assignments and uploaded files are not versioned. Saving without changes does not create a new version.
|
|
- A snippet's version goes up when its alias, title, description or code changes.
|
|
- The scenario and snippet pages always show the latest version. Older versions are available via `GET /scenarios/:id/versions[/:version]` and `GET /snippets/:id/versions[/:version]`.
|
|
- A run executes one scenario version, chosen in the run modal (latest by default) or via `version` in `POST /scenarios/:id/run`. The run keeps its own copy of that version's steps, so deleting or editing steps later doesn't change what the run shows. "Repeat run" reuses the run's version.
|
|
- Step code uses the latest version of a snippet by default. `context.runSnippet("alias@3")` pins version 3 instead. The version is looked up through the snippet's current alias. Pinned versions are listed under "Used Snippets" on the scenario page.
|
|
|
|
### Export / import
|
|
|
|
Exports hold exactly one version and always include its `version` number (the latest by default, or `?version=N`). Imports require `version`, and restore that exact version by id + version:
|
|
|
|
- If that version already exists, it is left untouched.
|
|
- Otherwise it is stored under that number, and becomes the latest if it is higher than the current latest version.
|
|
- Without an `id`, a new scenario or snippet is created at the given version.
|
|
|
|
## Playwright traces
|
|
|
|
Each run records a [Playwright trace](https://playwright.dev/docs/trace-viewer) of its browser context (screenshots and DOM snapshots). When the run finishes, the trace is uploaded to the run's files as `trace.zip`. Download it from the run detail page, then open it with `npx playwright show-trace trace.zip` or at https://trace.playwright.dev.
|
|
|
|
The trace mode is chosen per run: the run modal in the UI, the `traceMode` field of `POST /scenarios/:id/run`, and the `traceMode` argument of the `run_scenario` MCP tool all accept one of the values below. When it is omitted, the server's `PLAYWRIGHT_TRACE_MODE` setting is used (exposed via `GET /scenarios/run-defaults`). "Repeat run" reuses the original run's mode.
|
|
|
|
- `on` (default): keep the trace of every run
|
|
- `retain-on-failure`: keep traces only for runs that failed
|
|
- `off`: do not record traces
|
|
|
|
Traces are run artifacts, so they expire after `FILE_RUN_EXPIRATION_DAYS` like other run files.
|
|
|
|
## Validation contract
|
|
|
|
`validateCode` may return:
|
|
|
|
```js
|
|
return { success: true, description: 'ok' };
|
|
// or
|
|
return false;
|
|
```
|
|
|
|
Interpretation:
|
|
|
|
- Boolean: `true` = pass, `false` = fail
|
|
- Object: `success` controls pass/fail; `description` is persisted as step description
|
|
|
|
On validation failure, the step fails and the run is marked `fail`.
|
|
|
|
## Statuses
|
|
|
|
Run status:
|
|
|
|
- `pending`
|
|
- `in_progress`
|
|
- `pass`
|
|
- `fail`
|
|
|
|
Run-step status:
|
|
|
|
- `waiting`
|
|
- `pending`
|
|
- `in_progress`
|
|
- `pass`
|
|
- `fail`
|
|
- `cancelled`
|
|
|
|
## Example workflow
|
|
|
|
1. Create environment
|
|
2. Create credentials and snippets (optional but common)
|
|
3. Create scenario
|
|
4. Add ordered steps (`execCode`, optional `validateCode`)
|
|
5. Assign credentials to scenario aliases when needed
|
|
6. Run scenario
|
|
7. Wait for run completion and inspect run logs/outputs
|