Compare commits

..
6 Commits
Author SHA1 Message Date
ars9 4e55d14599 chore: step output and assertion log data
CI / changes (push) Successful in 14s
CI / version (push) Successful in 12s
CI / server (push) Successful in 1m17s
CI / client (push) Successful in 1m18s
CI / image (push) Successful in 1m49s
- "Step N passed" log entries carry the step's return value as data
- context.assert takes an optional details argument stored as log data
  on failure; step and snippet failure entries name only the assertion
- log output shows elapsed time in whole seconds
- run artifact and scenario file names open the file in a new tab
- bump version to 1.24.1
2026-10-02 11:12:12 +03:00
ars9 0f84db57f1 Merge pull request 'feat(scenario): attach step output and assertion details to run log' (#7) from feat/step-assert-log-data into master
CI / changes (push) Successful in 6s
CI / version (push) Successful in 8s
CI / server (push) Successful in 1m25s
CI / client (push) Successful in 1m16s
CI / image (push) Skipped
Reviewed-on: #7
2026-10-02 11:09:52 +03:00
ars9 959b3b70c9 feat(scenario): attach step output and assertion details to run log
CI / changes (pull_request) Successful in 9s
CI / version (pull_request) Skipped
CI / server (pull_request) Successful in 1m44s
CI / client (pull_request) Successful in 1m36s
CI / image (pull_request) Skipped
- "Step N passed" log entry carries the step's return value as data
- context.assert accepts an optional details argument; on failure the
  assertion:fail entry stores { details, error } as data
- assertion failures throw AssertionFailedError so "Step N failed" and
  "Snippet X failed" entries name only the assertion, without repeating
  the details; the step run description keeps the full message
- exported e2e runtime accepts and prints assert details
- log output shows elapsed time in whole seconds, matching the stored
  timestamp resolution
- run artifact and scenario file names link to the inline file content
  in a new tab
2026-10-02 11:05:29 +03:00
ars9 2281a8729b chore: code reference modal
CI / changes (push) Successful in 6s
CI / version (push) Successful in 6s
CI / server (push) Successful in 1m24s
CI / client (push) Successful in 1m34s
CI / image (push) Successful in 1m57s
- step and snippet editors open a reference of the script runtime and
  its context API from a (?) button next to the Code label
- bump version to 1.24.0
2026-10-01 14:50:23 +03:00
ars9 e3c2c0cebb Merge pull request 'feat(client): add code reference modal to step and snippet editors' (#4) from feat/code-context-help into master
CI / changes (push) Successful in 5s
CI / server (push) Skipped
CI / version (push) Successful in 5s
CI / client (push) Successful in 1m18s
CI / image (push) Skipped
Reviewed-on: #4
2026-10-01 14:48:35 +03:00
ars9 daecf3d539 feat(client): add code reference modal to step and snippet editors
CI / changes (pull_request) Successful in 7s
CI / version (pull_request) Skipped
CI / server (pull_request) Skipped
CI / client (pull_request) Successful in 1m13s
CI / image (pull_request) Skipped
- a (?) button next to the Code label on the step and snippet
  create/edit pages opens a reference of the script runtime: globals,
  every context member with examples, and pitfalls such as awaiting
  getStepOutput and assert
- the reference lives in components/modals/code-help.md and is
  rendered as markdown
- Modal takes size="lg" for document-like content; the dialog is
  capped to the viewport and its body scrolls, focusable when lg so
  the keyboard can scroll it
- docs/scenario.md notes that getStepOutput is async and points to
  the in-app reference to keep in sync
2026-10-01 14:45:03 +03:00
24 changed files with 577 additions and 33 deletions

No files matched your search

+27
View File
@@ -2,6 +2,33 @@
All notable changes to this project will be documented in this file.
## 1.24.1 - 2026-10-02
Changes since 1.24.0:
### Added
- `context.assert(fn, description, details?)`: optional `details` argument. On failure, the `assertion:fail` log entry stores `{ details, error }` as data (`error` is the exception message when `fn` threw), so large diagnostics such as a response body can stay out of the message. The exported e2e runtime accepts and prints it too.
- The "Step N passed" log entry carries the step's return value as data.
- File names in the run Artifacts and scenario Files tables link to the file content, opening it in a new tab (shown inline when the browser can display it).
### Changed
- For assertion failures, the "Step N failed" and "Snippet X failed" log entries name only the assertion (`Assertion failed: <description>`) instead of repeating the failure details. The step run's description keeps the full message.
- Log output shows elapsed time in whole seconds (`+14s`), matching the stored timestamp resolution.
## 1.24.0 - 2026-10-01
Changes since 1.23.0:
### Added
- Code reference in the UI: a (?) button next to the Code label on the step and snippet create/edit pages opens a modal documenting the script runtime - the globals (`context`, `expect`, `console`, and `args` in snippets) and every `context` member with examples, including pitfalls such as awaiting `getStepOutput()` and `assert()`, and `downloadFile()` not sending the browser's cookies.
### Changed
- `Modal` UI component: new `size="lg"` for long, document-like content. Dialogs are now capped to the viewport height and their body scrolls, instead of overflowing the screen.
## 1.23.0 - 2026-10-01
Changes since 1.22.2:
+36 -1
View File
@@ -4,13 +4,22 @@ import { expect, userEvent, within } from 'storybook/test';
import { Modal } from '../../src/ui/Modal/Modal';
import { Button } from '../../src/ui/Button/Button';
function ModalDemo({ title, body }: { title: string; body: string }) {
function ModalDemo({
title,
body,
size,
}: {
title: string;
body: React.ReactNode;
size?: 'md' | 'lg';
}) {
const [open, setOpen] = useState(false);
return (
<>
<Button onClick={() => setOpen(true)}>Open modal</Button>
<Modal
open={open}
size={size}
title={title}
onClose={() => setOpen(false)}
footer={
@@ -61,3 +70,29 @@ export const ClosesOnEscapeAndBackdrop: Story = {
await expect(canvas.queryByRole('dialog')).not.toBeInTheDocument();
},
};
export const Large: Story = {
args: {
title: 'Code reference',
size: 'lg',
body: (
<>
{Array.from({ length: 60 }, (_, i) => (
<p key={i}>Paragraph {i + 1} of a long, document-like modal body.</p>
))}
</>
),
},
play: async ({ canvasElement }) => {
const canvas = within(canvasElement);
await userEvent.click(canvas.getByRole('button', { name: 'Open modal' }));
const dialog = canvas.getByRole('dialog', { name: 'Code reference' });
// The dialog stays within the viewport; its body scrolls instead.
await expect(dialog.getBoundingClientRect().height).toBeLessThanOrEqual(window.innerHeight);
const body = within(dialog).getByText(
'Paragraph 1 of a long, document-like modal body.',
).parentElement!;
await expect(body.scrollHeight).toBeGreaterThan(body.clientHeight);
await expect(within(dialog).getByRole('button', { name: 'Confirm' })).toBeVisible();
},
};
@@ -0,0 +1,20 @@
.helpButton {
display: inline-flex;
align-items: center;
justify-content: center;
padding: 2px;
border: none;
border-radius: var(--radius-sm);
background: none;
color: var(--color-text-muted);
cursor: pointer;
}
.helpButton:hover {
color: var(--color-primary);
}
.helpButton:focus-visible {
outline: none;
box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.35);
}
@@ -0,0 +1,54 @@
import { useState } from 'react';
import { useTranslation } from 'react-i18next';
import { CircleHelp, X } from 'lucide-react';
import { Modal, Button, MarkdownContent } from '../../ui';
import content from './code-help.md?raw';
import styles from './CodeHelpModal.module.css';
export interface CodeHelpModalProps {
open: boolean;
onClose: () => void;
}
/** Reference for the script runtime (`context`, `expect`, ...) available to step and snippet code. */
export function CodeHelpModal({ open, onClose }: CodeHelpModalProps) {
const { t } = useTranslation();
return (
<Modal
open={open}
size="lg"
title={t('code_help.title')}
onClose={onClose}
footer={
<Button variant="secondary" onClick={onClose}>
<X size={14} />
{t('code_help.close')}
</Button>
}
>
<MarkdownContent content={content} />
</Modal>
);
}
/** A (?) button that opens the {@link CodeHelpModal}; place it next to a code field label. */
export function CodeHelpButton() {
const { t } = useTranslation();
const [open, setOpen] = useState(false);
return (
<>
<button
type="button"
className={styles.helpButton}
onClick={() => setOpen(true)}
aria-label={t('code_help.open')}
title={t('code_help.open')}
>
<CircleHelp size={16} />
</button>
<CodeHelpModal open={open} onClose={() => setOpen(false)} />
</>
);
}
+195
View File
@@ -0,0 +1,195 @@
Step code and snippet code are the **body of an async function**. Use `await` freely; whatever you `return` becomes the step (or snippet) output.
```js
async (context) => {
// your code here
}
```
All steps of a run share one browser, one browser context and one page, so state carries over from step to step: the current URL, cookies, local storage, open dialogs.
A step **passes** when its code returns, and **fails** when it throws or exceeds its timeout. When a step fails, the remaining steps are cancelled, and a screenshot and the page HTML are saved to the run's files.
## Globals
| Name | Description |
| --- | --- |
| `context` | The run context, described below. |
| `expect` | Playwright's [`expect`](https://playwright.dev/docs/test-assertions), with auto-retrying web assertions. |
| `console` | Writes to the run log: `log`/`info` → `log`, plus `warn`, `error` and `debug`. |
| `args` | **Snippets only.** The extra arguments passed to `context.runSnippet(name, ...args)`. |
Code runs in the server process with Node.js. It is trusted code, not sandboxed.
## Browser
### `context.page`
The Playwright [`Page`](https://playwright.dev/docs/api/class-page) of the run.
```js
await context.page.goto(context.getEnv("base_url"));
await context.page.getByRole("button", { name: "Sign in" }).click();
await expect(context.page.locator("h1")).toHaveText("Dashboard");
```
### `context.browser`
The Playwright [`BrowserContext`](https://playwright.dev/docs/api/class-browsercontext) the page belongs to. Use it for cookies, storage state, extra pages and HTTP requests that share the browser's cookies.
```js
const cookies = await context.browser.cookies();
const res = await context.browser.request.get(context.getEnv("api_url") + "/me");
```
### `context.dumpDom(selector?)` → `Promise<DomNode>`
Returns a simplified tree of the live DOM, starting at `selector` (default `"body"`). Decorative and layout elements are filtered out. The tree keeps landmarks, interactive elements, and elements with `role`, `data-testid`, `data-qa`, `data-action` or `data-element-id`. Each node has `tag` and `children`, plus `role`, `testid`, `id`, `name`, `type`, `text`, `href`, `checked`, `disabled`, etc. when they are present. This is useful for finding selectors without a screenshot.
If nothing matches `selector`, it returns a node with `tag: "ERROR"` instead of throwing.
```js
context.log("Login form:", await context.dumpDom("form"));
```
## Environment
### `context.env`
A copy of the selected environment's key/value map, `{}` when there is none. A missing key reads as `undefined`.
### `context.getEnv(key)` → `string`
Returns the environment value for `key`, and **throws** if it is not defined. Prefer it over `context.env[key]` for required settings: the step then fails with a clear message.
```js
const baseUrl = context.getEnv("base_url");
const locale = context.env.locale ?? "en";
```
## Credentials
### `context.getCredential(alias)` → `unknown`
Returns the payload of the credential assigned to this scenario under `alias` (see **Credentials** on the scenario page). JSON data is returned parsed, anything else as a string. **Throws** if no credential has that alias.
```js
const { username, password } = context.getCredential("admin");
await context.page.getByLabel("Username").fill(username);
await context.page.getByLabel("Password").fill(password);
```
## Step outputs
### `context.getStepOutput(order)` → `Promise<unknown>`
Returns the output (the return value) of another step in the same run. **It is async, so `await` it.**
- A positive `order` is absolute: `1` is the first step.
- A negative `order` is relative: `-1` is the previous step.
Returns `null` if that step does not exist, has not run, or returned nothing. A step's output is stored as JSON, so only JSON-serializable values come back as they were returned (no functions, `Date` becomes a string, Playwright objects are not usable).
```js
// step 2
return { orderId: await context.page.locator("#order-id").textContent() };
// step 3
const { orderId } = await context.getStepOutput(-1);
```
## Logging
### `context.log(...args)`, `context.warn(...args)`, `context.error(...args)`, `context.debug(...args)`
Write an entry with that level to the run log. `console.*` does the same. Strings, numbers and booleans are joined with spaces into the message. Objects and arrays are kept separate as structured **data**, shown pretty-printed under the message: one object is stored as is, several are stored as an array.
```js
context.log("Created user", user.id, user); // message "Created user 42", data = user
```
Every entry is tagged with where it came from: `step:<N>`, or `snippet:<alias>` while a snippet runs.
## Assertions
### `context.assert(fn, description, details?)` → `Promise<void>`
Runs `fn` (sync or async) and requires it to return exactly `true`.
- On success, it logs `description` (tagged `assertion:pass`).
- If `fn` returns anything else or throws, it logs `Assertion failed: <description>` as an error (tagged `assertion:fail`) and throws, which fails the step. The entry's data holds `details` (if given) and the exception message as `error` (if `fn` threw).
Await it, or a failure will not stop the step.
```js
await context.assert(
async () => (await context.page.locator(".cart-item").count()) === 3,
"Cart contains 3 items",
);
```
Pass large diagnostics (e.g. a response body) as `details` rather than in `description`, so the message stays short:
```js
await context.assert(() => response.ok(), "Validation must pass", {
status: response.status(),
body: await response.json(),
});
```
Use `expect` instead for assertions that wait and retry until the page matches, e.g. `await expect(locator).toBeVisible()`.
## Snippets
### `context.runSnippet(name, ...args)` → `Promise<unknown>`
Runs a stored snippet with the same `context` (same page, environment and credentials) and returns the snippet's return value. Extra arguments are available inside the snippet as `args`.
- `name` is the snippet alias, which runs its latest version.
- `alias@<version>` (e.g. `login@3`) pins a specific version.
Start, finish and failure are logged. If the snippet throws, the error propagates and fails the calling step unless you catch it. Snippets can call other snippets.
```js
// step code
const user = await context.runSnippet("login", "admin", { remember: true });
// snippet "login"
const [alias, opts] = args;
const { username, password } = context.getCredential(alias);
// ...
return { username };
```
## Files
### `context.getScenarioFiles(opts?)` → `Promise<{ items, total }>`
Lists the files uploaded to this scenario (see **Files** on the scenario page). `opts` takes `limit` and `offset` for paging. Each item has `id`, `name`, `mimeType`, `size`, `sha256` and `path`, the absolute path on the server's disk, which you can pass to Playwright:
```js
const { items } = await context.getScenarioFiles();
const doc = items.find((f) => f.name === "contract.pdf");
await context.page.locator("input[type=file]").setInputFiles(doc.path);
```
### `context.downloadFile(url, opts?)` → `Promise<FileInfo>`
Fetches `url` on the server and saves the response as a file of the current run. It returns the same fields as a scenario file (`id`, `name`, `mimeType`, `size`, `sha256`, `path`).
- `opts`: `method`, `headers`, `body` (a string) and `filename`. Without `filename`, the last segment of the URL path is used, and the MIME type is derived from the file extension.
- Redirects are followed. Any final status other than 200 throws. The limits are 100 MB and a 30 s timeout.
- `url` may be a `data:` URI (`data:<type>;base64,<data>` or `data:<type>,<percent-encoded>`), which saves content generated in the script without a network request. Pass `filename` in this case.
- Only works during a real scenario run. It throws in ad-hoc execution such as the `exec_code` MCP tool.
- This is a server-side request: it does **not** send the browser's cookies. Pass them in `headers` if the URL needs a session.
```js
await context.downloadFile(
"data:application/json;base64," + btoa(JSON.stringify(report)),
{ filename: "report.json" },
);
```
## Timeouts
Each step has its own timeout, or else the scenario's timeout, or else 60 s. The whole run fails once it exceeds the scenario's timeout (600 s by default). That check happens between steps.
+2
View File
@@ -6,3 +6,5 @@ export { RunScenarioModal } from './RunScenarioModal';
export type { RunScenarioModalProps } from './RunScenarioModal';
export { ExportLogsModal } from './ExportLogsModal';
export type { ExportLogsModalProps, LogExportFormat } from './ExportLogsModal';
export { CodeHelpModal, CodeHelpButton } from './CodeHelpModal';
export type { CodeHelpModalProps } from './CodeHelpModal';
+5
View File
@@ -12,6 +12,11 @@
"confirm_remove_credential": "Are you sure you want to remove this credential from scenario?",
"save_session": "Save session"
},
"code_help": {
"open": "Code reference",
"title": "Code reference",
"close": "Close"
},
"errors": {
"not_found": "Not found",
"not_found_session": "Session {{id}} does not exist or has been deleted.",
+6
View File
@@ -255,6 +255,12 @@
color: var(--color-text);
}
.labelRow {
display: flex;
align-items: center;
gap: var(--space-1);
}
.textarea {
width: 100%;
padding: var(--space-2) var(--space-3);
+17 -1
View File
@@ -554,7 +554,23 @@ function RunDetail() {
rowKey={(f) => f.id}
columns={
[
{ key: 'name', header: t('files.name'), render: (f) => f.name },
{
key: 'name',
header: t('files.name'),
// Served inline, so the browser shows what it can (images, text, PDF).
render: (f) =>
isFileExpired(f) ? (
f.name
) : (
<a
href={runFiles.contentUrl(id!, runId!, f.id)}
target="_blank"
rel="noopener noreferrer"
>
{f.name}
</a>
),
},
{
key: 'mimeType',
header: t('files.mimeType'),
+5 -1
View File
@@ -6,6 +6,7 @@ import { X, Save, ClipboardList } from 'lucide-react';
import { scenarios, steps } from '../../api';
import type { Scenario } from '../../api';
import { Breadcrumbs, Button, Card, Input, CodeEditor, useToast } from '../../ui';
import { CodeHelpButton } from '../../components/modals';
import styles from '../Page.module.css';
export function CreateStepPage() {
@@ -77,7 +78,10 @@ export function CreateStepPage() {
onChange={(e) => setTitle(e.target.value)}
/>
<div className={styles.formField}>
<label className={styles.fieldLabel}>{t('steps.form_exec_code')}</label>
<div className={styles.labelRow}>
<label className={styles.fieldLabel}>{t('steps.form_exec_code')}</label>
<CodeHelpButton />
</div>
<CodeEditor value={execCode} onChange={setExecCode} rows={6} />
</div>
<Input
+5 -1
View File
@@ -6,6 +6,7 @@ import { X, Save, ClipboardList } from 'lucide-react';
import { scenarios, steps } from '../../api';
import type { Scenario, ScenarioStep } from '../../api';
import { Breadcrumbs, Button, Card, Input, CodeEditor, useToast } from '../../ui';
import { CodeHelpButton } from '../../components/modals';
import styles from '../Page.module.css';
export function EditStepPage() {
@@ -89,7 +90,10 @@ export function EditStepPage() {
onChange={(e) => setTitle(e.target.value)}
/>
<div className={styles.formField}>
<label className={styles.fieldLabel}>{t('steps.form_exec_code')}</label>
<div className={styles.labelRow}>
<label className={styles.fieldLabel}>{t('steps.form_exec_code')}</label>
<CodeHelpButton />
</div>
<CodeEditor value={execCode} onChange={setExecCode} rows={6} />
</div>
<Input
@@ -761,7 +761,19 @@ export function ScenarioDetailPage() {
{
key: 'name',
header: t('files.name'),
render: (f) => f.name,
// Served inline, so the browser shows what it can (images, text, PDF).
render: (f) =>
isFileExpired(f) ? (
f.name
) : (
<a
href={scenarioFiles.contentUrl(id!, f.id)}
target="_blank"
rel="noopener noreferrer"
>
{f.name}
</a>
),
},
{
key: 'mimeType',
@@ -5,6 +5,7 @@ import { usePageTitle } from '../../hooks/usePageTitle';
import { X, Save, Braces } from 'lucide-react';
import { snippets } from '../../api';
import { Breadcrumbs, Button, Card, Input, CodeEditor, useToast } from '../../ui';
import { CodeHelpButton } from '../../components/modals';
import styles from '../Page.module.css';
export function CreateSnippetPage() {
@@ -107,10 +108,13 @@ export function CreateSnippetPage() {
/>
</div>
<div className={styles.formField}>
<label className={styles.fieldLabel}>
{t('snippets.form_code')}
<span className={styles.requiredMark}> *</span>
</label>
<div className={styles.labelRow}>
<label className={styles.fieldLabel}>
{t('snippets.form_code')}
<span className={styles.requiredMark}> *</span>
</label>
<CodeHelpButton />
</div>
<CodeEditor
value={code}
onChange={(v) => {
+8 -4
View File
@@ -6,6 +6,7 @@ import { X, Save, Braces } from 'lucide-react';
import { snippets } from '../../api';
import type { Snippet } from '../../api';
import { Breadcrumbs, Button, Card, Input, CodeEditor, useToast } from '../../ui';
import { CodeHelpButton } from '../../components/modals';
import styles from '../Page.module.css';
export function EditSnippetPage() {
@@ -134,10 +135,13 @@ export function EditSnippetPage() {
/>
</div>
<div className={styles.formField}>
<label className={styles.fieldLabel}>
{t('snippets.form_code')}
<span className={styles.requiredMark}> *</span>
</label>
<div className={styles.labelRow}>
<label className={styles.fieldLabel}>
{t('snippets.form_code')}
<span className={styles.requiredMark}> *</span>
</label>
<CodeHelpButton />
</div>
<CodeEditor
value={code}
onChange={(v) => {
+5 -2
View File
@@ -24,10 +24,13 @@ export interface LogOutputProps {
className?: string;
}
/** "+12.345s" (or "-" if before startTime, e.g. clock skew). */
/**
* "+12s" (or "-" if before startTime, e.g. clock skew). Log timestamps are
* stored with second resolution, so fractions would be noise.
*/
function formatElapsed(deltaMs: number): string {
const sign = deltaMs < 0 ? '-' : '+';
return `${sign}${(Math.abs(deltaMs) / 1000).toFixed(3)}s`;
return `${sign}${Math.round(Math.abs(deltaMs) / 1000)}s`;
}
/** Pretty-prints `data` if it's valid JSON, otherwise returns it as-is. */
+8
View File
@@ -16,6 +16,13 @@
border-radius: var(--radius-lg);
box-shadow: 0 20px 48px color-mix(in srgb, #000 28%, transparent);
overflow: hidden;
display: flex;
flex-direction: column;
max-height: 100%;
}
.lg {
width: min(880px, 100%);
}
.header {
@@ -31,6 +38,7 @@
.body {
padding: var(--space-2) var(--space-5) var(--space-4);
color: var(--color-text-muted);
overflow-y: auto;
}
.footer {
+8 -3
View File
@@ -7,9 +7,11 @@ export interface ModalProps {
children?: React.ReactNode;
onClose: () => void;
footer?: React.ReactNode;
/** `lg` widens the dialog for long, document-like content. */
size?: 'md' | 'lg';
}
export function Modal({ open, title, children, onClose, footer }: ModalProps) {
export function Modal({ open, title, children, onClose, footer, size = 'md' }: ModalProps) {
useEffect(() => {
if (!open) return;
const onKeyDown = (event: KeyboardEvent) => {
@@ -24,7 +26,7 @@ export function Modal({ open, title, children, onClose, footer }: ModalProps) {
return (
<div className={styles.overlay} role="presentation" onClick={onClose}>
<div
className={styles.dialog}
className={size === 'lg' ? `${styles.dialog} ${styles.lg}` : styles.dialog}
role="dialog"
aria-modal="true"
aria-label={title}
@@ -33,7 +35,10 @@ export function Modal({ open, title, children, onClose, footer }: ModalProps) {
<div className={styles.header}>
<h3 className={styles.title}>{title}</h3>
</div>
<div className={styles.body}>{children}</div>
{/* Long (lg) content scrolls; make it focusable so the keyboard can scroll it. */}
<div className={styles.body} tabIndex={size === 'lg' ? 0 : undefined}>
{children}
</div>
{footer && <div className={styles.footer}>{footer}</div>}
</div>
</div>
+5 -3
View File
@@ -32,10 +32,10 @@ Available on `context`:
- `context.env`: shallow copy of the environment's key/value map
- `context.getEnv(key)`: environment value lookup (throws if missing)
- `context.getCredential(alias)`: payload of the credential assigned to the scenario under `alias` (parsed JSON when the data is JSON; throws if missing)
- `context.getStepOutput(order)`: output of a previous step by absolute order (`1`, `2`, ...) or relative (`-1` = previous step); `null` if there is none
- `context.getStepOutput(order)`: async (`await` it); output of a previous step by absolute order (`1`, `2`, ...) or relative (`-1` = previous step); `null` if there is none
- `context.dumpDom(selector?)`: simplified DOM snapshot of the page (or of one element)
- `context.log(...args)`, `context.warn(...args)`, `context.error(...args)`, `context.debug(...args)`: run log entries
- `context.assert(() => boolean, description)`: runs the arrow function and requires it to return `true`. Logs `description` (as a `log` entry on success, as an `error` entry on failure or exception), then throws to stop the step if the function returned anything other than `true` or threw
- `context.assert(() => boolean, description, details?)`: runs the arrow function and requires it to return `true`. Logs `description` (as a `log` entry on success, as an `error` entry on failure or exception), then throws to stop the step if the function returned anything other than `true` or threw. On failure, the log entry's data holds `details` (if given) and the exception message (as `error`, if the function threw)
- `context.runSnippet(name, ...args)`: runs stored snippet code with the same context; the snippet receives the extra arguments as `...args`. `name` is the snippet alias for its latest version, or `alias@<version>` (e.g. `login@3`) to pin a specific version. Snippet start/finish/failure is logged
- `context.getScenarioFiles(opts?)`: list files uploaded to the scenario (`limit`, `offset`), including their path on disk
- `context.downloadFile(url, opts?)`: fetch `url` and save the result as a run file (requires a real scenario run — throws when invoked ad hoc, e.g. via the `exec_code` MCP tool). `opts` may include `method`, `headers`, `body`, `filename`. `url` also accepts a `data:` URI (`data:<mediaType>;base64,<data>` or `data:<mediaType>,<percent-encoded data>`) to save content generated in-script (e.g. a credential JWT) directly, without an actual network fetch.
@@ -47,11 +47,13 @@ Also in scope:
Code is syntax-checked before it runs, and executes in the server process: it's trusted code, not sandboxed.
The step and snippet editors have a (?) button next to the Code label that opens a reference of this runtime in the UI. Its content is [client/src/components/modals/code-help.md](../client/src/components/modals/code-help.md): keep it in sync with this section when the `context` API changes.
## Run logs
Each run log entry (`level`, `message`, `createdAt`) also carries a `context`: where the message was generated. Contexts:
- `step:<N>`: the step a log entry belongs to, including the automatic "Step N started / passed / failed" entries
- `step:<N>`: the step a log entry belongs to, including the automatic "Step N started / passed / failed" entries ("Step N passed" carries the step's return value as `data`; "Step N failed" for an assertion failure names only the assertion, whose details are in the `assertion:fail` entry)
- `snippet:<alias>`: entries logged while `context.runSnippet(alias, ...)` is running, including its own automatic "started"/"finished"/"failed" entries. A snippet calling another snippet tags entries with the innermost running snippet's alias
- `assertion:pass` / `assertion:fail`: `context.assert()`'s own log entry
- `run`: run-level entries not tied to a step (e.g. a scenario timeout, or the Playwright trace upload notice)
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "liqa",
"version": "1.23.0",
"version": "1.24.1",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "liqa",
"version": "1.23.0",
"version": "1.24.1",
"license": "MIT",
"workspaces": [
"server",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "liqa",
"version": "1.23.0",
"version": "1.24.1",
"private": true,
"license": "MIT",
"author": "Andrii Arsenin",
@@ -18,6 +18,31 @@ export interface ExecResult {
result: unknown;
}
/**
* Thrown by `context.assert`. Its `message` includes the failure reason;
* `summary` doesn't, so failure reports that follow the `assertion:fail` log
* entry (which already holds the details) can stay short.
*/
export class AssertionFailedError extends Error {
constructor(
message: string,
readonly summary: string,
) {
super(message);
this.name = "AssertionFailedError";
}
}
/** The `AssertionFailedError` behind `err` (directly or via `cause`), if any. */
export function findAssertionFailure(
err: unknown,
): AssertionFailedError | null {
for (let e = err; e instanceof Error; e = (e as { cause?: unknown }).cause) {
if (e instanceof AssertionFailedError) return e;
}
return null;
}
export type ScriptLogger = (
level: "debug" | "log" | "warn" | "error",
message: string,
@@ -50,11 +75,13 @@ export interface ScriptContext {
/**
* Runs `fn` and requires it to return `true`. Logs the outcome under
* `description`, then throws if `fn` returned anything other than `true`
* or threw an exception.
* or threw an exception. On failure, `details` (and the exception message,
* if any) are stored as the log entry's data.
*/
assert: (
fn: () => boolean | Promise<boolean>,
description: string,
details?: unknown,
) => Promise<void>;
/** Runs a named snippet with the same context, plus any extra positional args. */
runSnippet: (alias: string, ...args: unknown[]) => Promise<unknown>;
@@ -194,6 +221,7 @@ export class CodeExecutorService {
assert: async (
fn: () => boolean | Promise<boolean>,
description: string,
details?: unknown,
): Promise<void> => {
let passed: boolean;
let failureReason: string | undefined;
@@ -210,8 +238,18 @@ export class CodeExecutorService {
const message = failureReason
? `Assertion failed: ${description} (${failureReason})`
: `Assertion failed: ${description}`;
emit("error", message, "assertion:fail");
throw new Error(message);
const failureData =
failureReason !== undefined || details !== undefined
? {
...(failureReason !== undefined && { error: failureReason }),
...(details !== undefined && { details }),
}
: undefined;
emit("error", message, "assertion:fail", failureData);
throw new AssertionFailedError(
message,
`Assertion failed: ${description}`,
);
},
getStepOutput: getStepOutput ?? (() => Promise.resolve(null)),
getCredential: (alias: string): unknown => {
@@ -262,7 +300,7 @@ export class CodeExecutorService {
} catch (err) {
emit(
"error",
`Snippet "${alias}" failed: ${(err as Error).message}`,
`Snippet "${alias}" failed: ${findAssertionFailure(err)?.summary ?? (err as Error).message}`,
snippetContext,
);
throw err;
+10 -2
View File
@@ -370,7 +370,11 @@ function buildContext(page: Page, browser: BrowserContext, stepOutputs: StepOutp
log: (...args: unknown[]) => console.log(...args),
warn: (...args: unknown[]) => console.warn(...args),
error: (...args: unknown[]) => console.error(...args),
assert: async (fn: () => boolean | Promise<boolean>, description: string): Promise<void> => {
assert: async (
fn: () => boolean | Promise<boolean>,
description: string,
details?: unknown,
): Promise<void> => {
let passed: boolean;
let failureReason: string | undefined;
try {
@@ -386,7 +390,11 @@ function buildContext(page: Page, browser: BrowserContext, stepOutputs: StepOutp
const message = failureReason
? \`Assertion failed: \${description} (\${failureReason})\`
: \`Assertion failed: \${description}\`;
console.error(message);
if (details !== undefined) {
console.error(message, details);
} else {
console.error(message);
}
throw new Error(message);
},
getStepOutput: async (order: number): Promise<unknown> => {
@@ -13,7 +13,10 @@ import { Repository } from "typeorm";
import { type AppConfig, TraceMode } from "../config/app.config";
import { FileStorageService } from "../file/file-storage.service";
import type { ScriptLogger } from "../code-executor/code-executor.service";
import { CodeExecutorService } from "../code-executor/code-executor.service";
import {
CodeExecutorService,
findAssertionFailure,
} from "../code-executor/code-executor.service";
import { ExecContextBuilder } from "../code-executor/exec-context.builder";
import { traceStorage } from "../common/trace-context";
import { TraceLogger } from "../common/trace-logger";
@@ -446,7 +449,11 @@ export class ScenarioSchedulerService implements OnModuleInit {
} catch (err) {
const msg = (err as Error).message ?? String(err);
this.logger.error(`StepRun #${stepRun.id} threw: ${msg}`);
await this.failStepRun(stepRun, msg);
await this.failStepRun(
stepRun,
msg,
findAssertionFailure(err)?.summary,
);
} finally {
clearTimeout(timeoutId);
}
@@ -571,6 +578,7 @@ export class ScenarioSchedulerService implements OnModuleInit {
"log",
`Step ${stepRun.order} passed`,
`step:${stepRun.order}`,
output,
);
// Find the next waiting step in this run (next by order)
@@ -600,9 +608,14 @@ export class ScenarioSchedulerService implements OnModuleInit {
}
}
/**
* `logSummary` replaces `description` in the "Step N failed" log entry, e.g.
* for assertion failures whose details are already logged.
*/
private async failStepRun(
stepRun: ScenarioRunStepEntity,
description: string,
logSummary?: string,
): Promise<void> {
stepRun.status = "fail";
stepRun.description = description;
@@ -612,7 +625,7 @@ export class ScenarioSchedulerService implements OnModuleInit {
stepRun.runId,
stepRun.id,
"error",
`Step ${stepRun.order} failed: ${description}`,
`Step ${stepRun.order} failed: ${logSummary ?? description}`,
`step:${stepRun.order}`,
);
+79
View File
@@ -208,4 +208,83 @@ describe("ScenarioRunStepEntity.output + context.getStepOutput", () => {
expect(JSON.parse(rows[2].output as string)).toEqual([1, 2, 3, 4]);
});
});
// ── run log data ───────────────────────────────────────────────────────────
describe("run log data", () => {
async function logRows(
runId: string,
context: string,
): Promise<Array<{ message: string; data: string | null }>> {
return dataSource.query(
`SELECT message, data FROM scenario_run_logs WHERE runId = '${runId}' AND context = '${context}' ORDER BY seq`,
);
}
it("attaches the step output to the 'Step N passed' entry", async () => {
await seedSession();
const scId = await createScenario("log-step-output");
await createStep(scId, 0, 'return { token: "abc" };');
const runId = await runScenario(scId);
const rows = await logRows(runId, "step:1");
const passed = rows.find((r) => r.message === "Step 1 passed");
expect(JSON.parse(passed!.data as string)).toEqual({ token: "abc" });
});
it("stores assertion failure details as log data", async () => {
await seedSession();
const scId = await createScenario("log-assert-details");
await createStep(
scId,
0,
'await context.assert(() => false, "Validation must pass", { status: 500 });',
);
const runId = await runScenario(scId);
const [row] = await logRows(runId, "assertion:fail");
expect(row.message).toBe("Assertion failed: Validation must pass");
expect(JSON.parse(row.data as string)).toEqual({
details: { status: 500 },
});
});
it("keeps the 'Step N failed' entry of an assertion failure short", async () => {
await seedSession();
const scId = await createScenario("log-assert-step-fail");
await createStep(
scId,
0,
'await context.assert(() => { throw new Error("boom"); }, "Must not throw");',
);
const runId = await runScenario(scId);
const rows = await logRows(runId, "step:1");
const failed = rows.find((r) => r.message.startsWith("Step 1 failed:"));
expect(failed!.message).toBe(
"Step 1 failed: Assertion failed: Must not throw",
);
const stepRow = await dataSource.query(
`SELECT description FROM scenario_run_steps WHERE runId = '${runId}'`,
);
expect(stepRow[0].description).toContain("(boom)");
});
it("stores the exception message of a throwing assertion as log data", async () => {
await seedSession();
const scId = await createScenario("log-assert-throw");
await createStep(
scId,
0,
'await context.assert(() => { throw new Error("boom"); }, "Must not throw");',
);
const runId = await runScenario(scId);
const [row] = await logRows(runId, "assertion:fail");
expect(JSON.parse(row.data as string)).toEqual({ error: "boom" });
});
});
});