diff --git a/docs/architecture.md b/docs/architecture.md index 278a98b..d14ddf0 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,45 +1,43 @@ # Architecture ``` -┌─────────────────────────────────────────┐ -│ Browser (React SPA — client/) │ -│ │ -│ EnvironmentsPage KeysPage │ -│ SessionsPage ScenariosPage │ -└───────────────────┬─────────────────────┘ - │ HTTP REST (:13000) - │ +┌──────────────────────────────────────────────────────┐ +│ Browser (React SPA - client/) │ +│ │ +│ Environments Credentials Snippets │ +│ Sessions Scenarios Runs │ +└─────────────────────────────┬────────────────────────┘ + │ HTTP REST (:13000) + │ MCP client (AI agent / VS Code Copilot) - │ HTTP /mcp (Streamable MCP) + │ HTTP /mcp (Streamable MCP) ▼ -┌──────────────────────────────────────┐ -│ NestJS app (server/src/) │ -│ │ -│ McpService ← all MCP tools │ -│ │ -│ AuthService login + sessions │ -│ SessionService session CRUD │ -│ EnvironmentService environment CRUD │ -│ BrowserService ad-hoc browsing │ -│ CodeExecutorService sandboxed JS │ -│ │ -│ ScenarioService scenario CRUD │ -│ ScenarioSchedulerService │ -│ └─ @Interval(1 s) tick loop │ -└──────────────────┬───────────────────┘ - │ better-sqlite3 (TypeORM) - ▼ - data/sessions.db +┌──────────────────────────────────────────────────────┐ +│ NestJS app (server/src/) │ +│ │ +│ McpService all MCP tools │ +│ SessionService session persistence │ +│ SessionContextService live browser contexts │ +│ EnvironmentService environment CRUD + import │ +│ CredentialService credential CRUD + import │ +│ SnippetService snippet CRUD + import │ +│ BrowserService ad-hoc open/exec │ +│ CodeExecutorService sandboxed JS runtime │ +│ ScenarioService scenarios/steps/runs CRUD │ +│ ScenarioSchedulerService @Interval(1 s) run worker │ +└───────────────────────┬──────────────────────────────┘ + │ better-sqlite3 (TypeORM) + ▼ + data/sessions.db ``` -- **McpService** — registers every MCP tool and delegates to domain services. -- **AuthService** — drives a Playwright browser to log in via a key descriptor; saves cookies + localStorage as a `Session`. -- **SessionService** — stores and retrieves saved browser sessions. -- **EnvironmentService** — stores named environments (URL maps) used by auth and scenarios. -- **BrowserService** — ad-hoc, stateless `open_url` / `exec_code` operations outside the scenario scheduler. -- **CodeExecutorService** — compiles and runs user-supplied Playwright JS inside a sandboxed `async` function with `page`, `context`, and `helpers` in scope. -- **ScenarioService** — CRUD for scenarios, steps, runs, and run logs; import/export. -- **ScenarioSchedulerService** — polls for pending step runs every second and executes them. All steps in the same run share **one browser instance** (see [scenario.md](scenario.md)). +- **McpService** registers MCP tools and delegates to session, environment, browser, and scenario services. +- **SessionService** persists session records in SQLite. **SessionContextService** manages live Playwright contexts for open sessions. +- **EnvironmentService**, **CredentialService**, and **SnippetService** provide CRUD plus import/export upserts. +- **BrowserService** handles stateless ad-hoc automation (`/open`, `/exec`) and session-aware browsing. +- **CodeExecutorService** validates and executes user JS with `page`, `context`, and helper APIs. +- **ScenarioService** manages scenarios, steps, credential aliases, runs, logs, and scenario import/export. +- **ScenarioSchedulerService** polls pending runs every second and executes step runs to completion. ## Client (`client/`) @@ -47,4 +45,6 @@ A React 19 + TypeScript SPA served separately by Vite during development. In Doc The client calls the server REST endpoints directly — there is no separate BFF layer. Routing uses HashRouter so the Vite proxy and server-side routes are never ambiguous. +The sidebar footer contains the theme switcher and the app version (injected from the workspace root `package.json` at build time). + See [development.md](development.md) for the full client tech stack and directory layout. diff --git a/docs/development.md b/docs/development.md index cf3dba0..1ac0d09 100644 --- a/docs/development.md +++ b/docs/development.md @@ -26,8 +26,10 @@ Copy `.env.example` to `.env` and fill in the required values before running. | Variable | Default | Description | |---|---|---| | `PORT` | `3000` | HTTP port | -| `KEYS_DIR` | `keys` | Directory containing `*.json` key descriptors | +| `KEYS_DIR` | `keys` | Directory for key material used by credential/snippet workflows | | `DB_PATH` | `data/sessions.db` | SQLite database file | +| `SESSION_IDLE_TIMEOUT_MINUTES` | `30` | Idle timeout before open sessions are closed | +| `SESSION_DELETE_CLOSED_DAYS` | `7` | Retention period for closed sessions | | `PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH` | _(Playwright default)_ | Path to Chromium binary. Set automatically in Docker (`/usr/bin/chromium`). | --- @@ -59,7 +61,13 @@ The server listens on port **3000** by default. npm -w client run dev ``` -The Vite dev server runs on **http://localhost:5173** and proxies all API paths (`/environments`, `/sessions`, `/scenarios`, `/keys`, `/login`, `/mcp`) to the server at `localhost:3000`. +The Vite dev server runs on **http://localhost:5173** and proxies these API prefixes to `localhost:13000` (or `localhost:3000` when running server locally): + +- `/environments` +- `/credentials` +- `/snippets` +- `/sessions` +- `/scenarios` **Production build**: @@ -201,7 +209,7 @@ docker compose logs -f ## Client overview -The client (`client/`) is a React 19 + TypeScript SPA built with Vite. It provides a browser UI for managing environments, sessions, keys, and scenarios. +The client (`client/`) is a React 19 + TypeScript SPA built with Vite. It provides a browser UI for managing environments, credentials, snippets, sessions, scenarios, and runs. ### Tech stack @@ -221,11 +229,11 @@ The client (`client/`) is a React 19 + TypeScript SPA built with Vite. It provid ``` client/ src/ - api/ API client modules (environments, keys, sessions, scenarios) + api/ API client modules (environments, credentials, snippets, sessions, scenarios) hooks/ useTheme (light/dark persistence) i18n/ i18next bootstrap + locales/en.json - pages/ EnvironmentsPage, KeysPage, SessionsPage, ScenariosPage - ui/ Reusable component library + pages/ Environment/Credential/Snippet/Session/Scenario/Run pages + ui/ Reusable component library Badge, Breadcrumbs, Button, Card, Input, Select, SidePanel, Table, ThemeSwitcher, Timestamp .storybook/ @@ -239,10 +247,14 @@ Hash-based routing (`/#/path`) avoids conflicts with the Vite dev-server proxy. | Route | Page | |---|---| | `/#/environments` | Environment list | -| `/#/keys` | Key list | +| `/#/credentials` | Credential list | +| `/#/snippets` | Snippet list | | `/#/sessions` | Session list | | `/#/scenarios` | Scenario list | +| `/#/runs` | Cross-scenario runs list | ### Theming -Light/dark theme is toggled by the `ThemeSwitcher` button in the sidebar header. The selection is persisted to `localStorage` and applied as `data-theme` on `` before React mounts (anti-FOUC inline script in `index.html`). +Light/dark theme is toggled by the `ThemeSwitcher` button in the sidebar footer. The selection is persisted to `localStorage` and applied as `data-theme` on `` before React mounts (anti-FOUC inline script in `index.html`). + +The sidebar footer also shows the app version read from the workspace root `package.json` and injected via Vite `define`. diff --git a/docs/key-descriptors.md b/docs/key-descriptors.md deleted file mode 100644 index e629d61..0000000 --- a/docs/key-descriptors.md +++ /dev/null @@ -1,20 +0,0 @@ -# Key descriptors - -Each file in `KEYS_DIR` is named `.json` and takes one of two forms: - -**File key (EDS):** -```json -{ - "keyFile": "path/to/key.dat", - "password": "secret" -} -``` -`keyFile` is resolved relative to `KEYS_DIR`. - -**Login + password:** -```json -{ - "login": "user@example.com", - "password": "secret" -} -``` diff --git a/docs/mcp.md b/docs/mcp.md index 5fe3bb3..3ad46e4 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -1,49 +1,56 @@ # MCP tools -## Auth / session +The MCP endpoint is exposed at `/mcp` (Streamable HTTP transport). + +The server currently registers the following tools. + +## Sessions | Tool | Description | |---|---| -| `list_keys` | List available key IDs from `KEYS_DIR` | -| `login` | Log in with a key against an environment; stores resulting session | -| `list_sessions` | Paginated list of saved sessions | -| `delete_session` | Delete a session by ID | +| `list_sessions` | List stored sessions (paginated) | +| `delete_session` | Delete a session by UUID | ## Environments | Tool | Description | |---|---| -| `list_environments` | Paginated list of environments | -| `get_environment` | Get one environment by ID | -| `create_environment` | Create an environment (`name` + `urls` map) | -| `update_environment` | Update name / urls | -| `delete_environment` | Delete by ID | +| `list_environments` | List environments (paginated) | +| `get_environment` | Get one environment by UUID | +| `create_environment` | Create environment (`name`, `urls`) | +| `update_environment` | Update environment (`id`, optional `name`/`urls`) | +| `delete_environment` | Delete environment by UUID | -`urls` is a free-form map. Recognised keys: `id_url` (auth server), `cabinet_url` (cabinet redirect target), `admin_url`. +`urls` is a free-form key/value map. Common keys: `id_url`, `cabinet_url`, `admin_url`. -## Browser (ad-hoc, stateless) +## Browser (ad-hoc) | Tool | Description | |---|---| -| `open_url` | Navigate to a URL and return page content / reader-mode text | -| `exec_code` | Run Playwright JS in a session's browser context | +| `open_url` | Open URL with optional `sessionName`, return page/title/content | +| `exec_code` | Execute Playwright JS with `page` and `context` in scope | ## Scenarios | Tool | Description | |---|---| -| `list_scenarios` | Paginated list | -| `get_scenario` | Fetch scenario with all steps | -| `create_scenario` | Create a new scenario by name | -| `update_scenario` | Rename | -| `delete_scenario` | Delete (cascades to steps and runs) | -| `create_scenario_step` | Add a step | -| `get_scenario_step` | Fetch one step | +| `list_scenarios` | List scenarios (paginated) | +| `get_scenario` | Get scenario with steps | +| `create_scenario` | Create scenario by `name` | +| `update_scenario` | Rename scenario | +| `delete_scenario` | Delete scenario | +| `create_scenario_step` | Add step to scenario | +| `get_scenario_step` | Get one step | | `update_scenario_step` | Update step fields | -| `delete_scenario_step` | Remove a step | -| `list_scenario_runs` | Paginated run history (with optional status filter) | -| `run_scenario` | Trigger an immediate run | -| `get_scenario_run` | Get a run with all step runs and their outputs | -| `wait_for_scenario_run` | Block until run reaches `pass`/`fail`, return full run | -| `export_scenario` | Export scenario as a portable JSON payload | -| `import_scenario` | Import a scenario from an export payload | +| `delete_scenario_step` | Delete step | +| `list_scenario_runs` | List runs for a scenario | +| `run_scenario` | Trigger run for scenario | +| `get_scenario_run` | Get run with step runs and logs | +| `wait_for_scenario_run` | Wait until run is `pass` or `fail` | +| `export_scenario` | Export scenario payload | +| `import_scenario` | Import scenario payload | + +## Notes + +- Tool IDs and entity IDs are UUIDs. +- `create_scenario_step` schema still accepts legacy `type`/`sessionName` fields for compatibility. Current scheduler executes `execCode` plus optional `validateCode` and does not branch by step type. diff --git a/docs/scenario.md b/docs/scenario.md index 3739ed6..3663e25 100644 --- a/docs/scenario.md +++ b/docs/scenario.md @@ -1,80 +1,82 @@ # Scenario execution -When a scenario is run, the scheduler creates a `ScenarioRun` with one `ScenarioRunStep` per step. Steps are executed **in order** (by `step.order`), one per scheduler tick. +When a scenario run is created, the server creates one `ScenarioRunStep` per scenario step. -**All steps in the same run share a single Playwright browser instance.** The browser is created on the first `exec` or `sign` step (using session cookies + localStorage), reused for every subsequent step, and closed when the run finishes (pass or fail). +- Step runs are ordered by `step.order`. +- First step run starts as `pending`; the rest start as `waiting`. +- Scheduler tick (`@Interval(1000)`) picks pending runs and processes steps in order. +- Steps in the same run share one Playwright browser/context/page. -## Step types +## Execution model -### `login` +Current scheduler behavior is exec-centric: -Authenticates in a **separate, ephemeral browser** and saves the resulting session to the database. Subsequent exec/sign steps load that session into the shared browser. +- `execCode` is required for a step to pass. +- `validateCode` is optional and runs after `execCode`. +- Legacy step type metadata may still appear in MCP payloads, but run execution is not branched by step type. -`execCode` must be a JSON object: -```json -{ "keyId": "3137411915", "environmentName": "liquio-diia-stg" } -``` +## Script runtime -### `exec` +`execCode` and `validateCode` execute as async JavaScript with: -Runs arbitrary Playwright JS on the **shared page**. The page carries over state from the previous step (URL, form inputs, cookies, etc.). +- `page`: Playwright `Page` +- `context`: Playwright `BrowserContext` +- `helpers`: utility object +- `console`: proxied to run logs (`log`, `warn`, `error`, etc.) -`execCode` is executed as the body of an `async` function with these variables in scope: +`validateCode` additionally receives `result`, which is the value returned by `execCode`. -- `page` — current Playwright `Page` -- `context` — current Playwright `BrowserContext` -- `helpers.dumpDom(selector?)` — returns a simplified DOM string -- `helpers.getStepOutput(order)` — returns the JSON-parsed `output` of a previous step. Positive values are absolute step orders; negative values are relative to the current step (`-1` = previous step). +## Available helpers -Any value returned from `execCode` is JSON-serialised and stored as the `output` field on the `ScenarioRunStep` record. Subsequent steps can read it via `helpers.getStepOutput()`. +- `helpers.dumpDom(selector?)`: simplified DOM snapshot +- `helpers.log(...args)`, `helpers.warn(...args)`, `helpers.error(...args)`: structured step logs +- `helpers.getStepOutput(order)`: prior step output by absolute order (`0`, `1`, ...) or relative (`-1` previous step) +- `helpers.getCredential(alias)`: credential payload assigned to the scenario alias +- `helpers.env`: shallow copy of environment URL map +- `helpers.getEnvUrl(key)`: required environment URL lookup (throws if missing) +- `helpers.runSnippet(name, ...args)`: execute stored snippet code in the same page/context/helpers scope + +## Validation contract + +`validateCode` may return: ```js -// example execCode -await page.goto('https://cabinet.example.com/tasks/create/987825'); -await page.locator('input[name="institution"]').fill('caltech'); -await page.locator('button:has-text("Далі")').click(); -await page.waitForLoadState('networkidle'); -return page.url(); // stored in ScenarioRunStep.output -``` - -### `sign` - -Performs EDS signing on the **current shared page** using a key from `KEYS_DIR`. The step clicks the sign widget, selects the file key method, uploads the key file, enters the password, and submits — all without requiring the step author to handle key material in their code. - -`execCode` must be a JSON object: -```json -{ "keyId": "3137411915" } -``` - -## Validation code - -Any step type optionally accepts `validateCode` — Playwright JS that runs on the shared page **after** the main action and must return a result evaluated as: - -```js -return { success: true, description: 'Application created' }; +return { success: true, description: 'ok' }; // or -return { success: false, description: 'Expected success banner not found' }; -// or simply -return someBoolean; +return false; ``` -If `success` is false the step (and the whole run) fails with the provided `description`. +Interpretation: -## Run statuses +- Boolean: `true` = pass, `false` = fail +- Object: `success` controls pass/fail; `description` is persisted as step description -`pending` → `in_progress` → `pass` | `fail` +On validation failure, the step fails and the run is marked `fail`. -Individual step runs also expose `waiting` (not yet reached) and `cancelled` (run failed before this step). +## Statuses -## Adding a new scenario (example workflow) +Run status: -``` -1. create_environment (if not already present) -2. create_scenario → get scenarioId -3. create_scenario_step type=login, order=0 -4. create_scenario_step type=exec, order=1 (navigate + fill form) -5. create_scenario_step type=sign, order=2 -6. create_scenario_step type=exec, order=3 (validateCode checks success banner) -7. run_scenario → get runId -8. wait_for_scenario_run (blocks until pass/fail, returns full run with outputs) -``` +- `pending` +- `in_progress` +- `pass` +- `fail` + +Run-step status: + +- `waiting` +- `pending` +- `in_progress` +- `pass` +- `fail` +- `cancelled` + +## Example workflow + +1. Create environment +2. Create credentials and snippets (optional but common) +3. Create scenario +4. Add ordered steps (`execCode`, optional `validateCode`) +5. Assign credentials to scenario aliases when needed +6. Run scenario +7. Wait for run completion and inspect run logs/outputs