- Add validated LOG_MODE configuration and route trace-aware diagnostics through debug. - Select debug mode in shared test setup while preserving DEBUG namespace control. - Add logger/config regression coverage and document runtime configuration.
405 lines
22 KiB
Markdown
405 lines
22 KiB
Markdown
# Development
|
|
|
|
## Prerequisites
|
|
|
|
- Node.js 24+ (required by `engines` in `package.json`; the Docker image uses Node.js 24)
|
|
- Docker + Docker Compose
|
|
|
|
The repository is a **npm workspace monorepo** with two packages:
|
|
|
|
| Workspace | Directory | Description |
|
|
|---|---|---|
|
|
| `server` | `server/` | NestJS API + MCP server |
|
|
| `client` | `client/` | React 19 + Vite admin UI |
|
|
|
|
Install all dependencies from the repo root:
|
|
|
|
```bash
|
|
npm install
|
|
npx playwright install --only-shell chromium # first time only: headless Chromium for the server
|
|
```
|
|
|
|
No `.env` is needed: every setting below has a default. To override settings, copy `.env.example` to `.env` and uncomment the ones you need. Docker Compose loads `.env` when it exists.
|
|
|
|
### Environment variables (server)
|
|
|
|
All settings are validated at startup by `server/src/config/app.config.ts`.
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `NODE_ENV` | `development` | `production` in the Docker image. Shown in the startup log, and selects the default CORS origins |
|
|
| `CORS_ORIGINS` | *(empty)* | Comma-separated origins allowed to call the API cross-origin. When empty: the Vite dev server (`http://localhost:5173`, `http://127.0.0.1:5173`) unless `NODE_ENV=production`, then the compose port (`http://localhost:13000`, `http://127.0.0.1:13000`) |
|
|
| `PORT` | `3000` | HTTP port |
|
|
| `LOG_MODE` | `console` | `console` preserves the default server output; `debug` routes application diagnostics through the `debug` package and only emits namespaces enabled by `DEBUG` (for example, `DEBUG=liqa:*`). |
|
|
| `DB_PATH` | `data/liqa.db` | SQLite database file |
|
|
| `FILES_DIR` | `data/files` | Where scenario files and run artifacts are stored |
|
|
| `FILE_RUN_EXPIRATION_DAYS` | `7` | How long run artifacts (downloads, failure dumps, traces) are kept |
|
|
| `FILE_CLEANUP_CRON` | `0 3 * * *` | When expired files are deleted |
|
|
| `SESSION_IDLE_TIMEOUT_MINUTES` | `30` | Idle timeout before open sessions are closed |
|
|
| `SESSION_DELETE_CLOSED_DAYS` | `7` | Retention period for closed sessions |
|
|
| `PLAYWRIGHT_TRACE_MODE` | `off` | Default Playwright trace mode for runs: `on`, `retain-on-failure` or `off` (see [scenario.md](scenario.md)) |
|
|
| `RUN_LOG_DATA_COLLAPSE_BYTES` | `256` | Run log `data` blocks whose pretty-printed JSON is over this many bytes start collapsed in the UI (exposed via `/manifest.json`) |
|
|
| `RUN_LOG_DATA_COLLAPSE_LINES` | `5` | Same, for the number of lines |
|
|
| `CLIENT_DIR` | *(empty)* | Directory with the built client to serve at `/` (the Docker image sets it to `client`); empty means the server doesn't serve the UI |
|
|
| `PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH` | _(Playwright default)_ | Path to a Chromium binary to use instead of Playwright's own. Not set in Docker: the image ships Playwright's `chromium-headless-shell` (matching the Playwright version) under `PLAYWRIGHT_BROWSERS_PATH=/ms-playwright`. |
|
|
| `AUTH_ENABLED` | `false` | Opt-in account authentication. With the default `false`, the existing UI, REST API and MCP remain available without accounts. |
|
|
| `AUTH_JWT_SECRET` | *(empty)* | Required and non-empty when `AUTH_ENABLED=true`. Keep it private and stable across restarts; changing it invalidates existing tokens. |
|
|
| `AUTH_COOKIE_NAME` | `liqa_auth` | Name of the browser's HttpOnly session cookie. |
|
|
| `AUTH_JWT_EXPIRES_IN` | `24h` | Default lifetime for browser sessions and CLI-issued tokens. Positive durations use `s`, `m`, `h` or `d`. |
|
|
| `AUTH_REVOKED_TOKEN_CLEANUP_CRON` | `0 3 * * *` | When expired logout revocations are cleaned. Cleanup also runs once at startup and is disabled when auth is off. |
|
|
|
|
---
|
|
|
|
## Starting the application
|
|
|
|
### Server
|
|
|
|
**Development** (watch mode, restarts on file changes):
|
|
|
|
```bash
|
|
npm -w server run start:dev
|
|
```
|
|
|
|
**Production build, then start**:
|
|
|
|
```bash
|
|
npm -w server run build
|
|
npm -w server run start:prod
|
|
```
|
|
|
|
The server listens on port **3000** by default.
|
|
|
|
### Client
|
|
|
|
**Development** (Vite dev server with HMR):
|
|
|
|
```bash
|
|
npm -w client run dev
|
|
```
|
|
|
|
The Vite dev server runs on **http://localhost:5173**. It has no API proxy: the client calls `VITE_API_URL` (default `/api/v1` on its own origin), and the server allows CORS from `http://localhost:5173`. So point it at a running server:
|
|
|
|
```bash
|
|
npm -w client run dev:compose-api # API of the compose container (localhost:13000)
|
|
VITE_API_URL=http://localhost:3000/api/v1 npm -w client run dev # API of a locally started server
|
|
```
|
|
|
|
**Production build**:
|
|
|
|
```bash
|
|
npm -w client run build
|
|
```
|
|
|
|
The built files are emitted to `client/dist/`.
|
|
|
|
---
|
|
|
|
## Database migrations
|
|
|
|
The SQLite schema is managed by TypeORM migrations in `server/src/database/migrations/`; `synchronize` is off.
|
|
|
|
- **Docker:** the container applies pending migrations before starting the server.
|
|
- **Locally:** build, then apply them before starting the server (it refuses to start while migrations are pending):
|
|
|
|
```bash
|
|
npm -w server run build
|
|
npm -w server run migration:run # apply pending migrations to DB_PATH
|
|
npm -w server run migration:show # list migrations and whether they're applied
|
|
```
|
|
|
|
The scripts run from `server/`, so `DB_PATH` (default `data/liqa.db`) is relative to it, and they read `server/.env` if present, like the server does.
|
|
|
|
### Enabling authentication
|
|
|
|
Authentication is disabled by default, so a fresh install needs no auth settings. For Docker Compose, set these in the project `.env` (keep it out of version control):
|
|
|
|
```dotenv
|
|
AUTH_ENABLED=true
|
|
AUTH_JWT_SECRET=<private-random-secret>
|
|
```
|
|
|
|
Use a long, randomly generated secret from your secret manager; do not use the placeholder literally or commit a real value. `AUTH_ENABLED` accepts only `true` or `false`. When enabled, startup fails if the signing secret is blank or the configured token lifetime/cleanup schedule is invalid. Rebuild/restart the container after changing its configuration. Compose applies pending migrations before starting the server. For local development, export the same variables in the server process environment (or put them in `server/.env`); run `npm -w server run build` and `npm -w server run migration:run` before account commands or server startup. Migrations add the user and revoked-token tables and are required even when authentication is disabled.
|
|
|
|
When auth is enabled, REST and MCP controller routes require a valid bearer token or browser cookie except `GET /health`, `GET /healthz`, `GET /manifest.json`, and `POST /api/v1/auth/login`. Login is public so a browser can establish its session. The manifest exposes whether auth is enabled; health endpoints return service status. The UI assets and Swagger UI at `/docs` remain reachable, while API operations invoked from Swagger require authentication. MCP at `/mcp` and the run-events stream are protected. When auth is disabled, the guard allows existing requests through.
|
|
|
|
## Operator account commands
|
|
|
|
Authentication is opt-in. To manage accounts or issue tokens, set `AUTH_ENABLED=true` and `AUTH_JWT_SECRET` in the same environment used by the server. Apply database migrations before using the commands. Passwords are entered interactively without terminal echo; they are never accepted as command-line arguments. The CLI uses the same `DB_PATH` as the server; locally it resolves the default `data/liqa.db` relative to `server/`, while inside the container it uses the configured path (normally `/app/data/liqa.db`, persisted by the `./data:/app/data` volume).
|
|
|
|
Build the server and apply migrations locally, then run commands from the server workspace:
|
|
|
|
```bash
|
|
npm -w server run build
|
|
npm -w server run migration:run
|
|
npm -w server run cli -- user:create
|
|
npm -w server run cli -- user:get alice
|
|
npm -w server run cli -- user:token alice # uses AUTH_JWT_EXPIRES_IN (default 24h)
|
|
npm -w server run cli -- user:token alice 30m # optional lifetime
|
|
npm -w server run cli -- user:delete alice
|
|
```
|
|
|
|
The container image does not include npm or npx. Run the built CLI with Node inside the running service; it uses that container's configured `DB_PATH` and exits without starting the web server or schedulers:
|
|
|
|
```bash
|
|
docker compose exec liqa node dist/cli.js user:create
|
|
docker compose exec liqa node dist/cli.js user:get alice
|
|
docker compose exec liqa node dist/cli.js user:token alice 30m
|
|
docker compose exec liqa node dist/cli.js user:delete alice
|
|
```
|
|
|
|
`user:create` asks for a username and two matching passwords; password entry is hidden and requires an interactive terminal. A duplicate username, blank username/password, mismatch or invalid input fails with a nonzero exit. The other commands require an existing username; lookup output contains only the account id, username and timestamps. Unknown usernames fail with `No account found for that username.` and a nonzero exit. A token duration must be a positive number followed by `s`, `m`, `h` or `d` (for example `30m`, `1h`, `7d`); omitted duration uses `AUTH_JWT_EXPIRES_IN` (`24h` by default). Token output is a bearer credential: handle it as a secret and do not paste it into logs or shared history. User deletion removes the account, so the server rejects that user's previously issued tokens on their next authentication. General CLI failures print a concise hint to check auth settings, `DB_PATH` and migrations.
|
|
|
|
### Browser sessions, logout and development origins
|
|
|
|
With auth enabled, the UI asks for credentials at `/login`. `POST /api/v1/auth/login` sets a signed session in an HttpOnly, `SameSite=Strict` cookie (default name `liqa_auth`) whose expiry matches the JWT lifetime. The cookie gets the `Secure` attribute when the request is HTTPS, including when the server sees `X-Forwarded-Proto: https`; serve production traffic over HTTPS and configure trusted proxy headers accordingly. JavaScript cannot read the token. `POST /api/v1/auth/logout` revokes the presented token and clears the cookie. Revoked tokens remain rejected until their expiration; deleted accounts also cause their tokens to be rejected on the next authentication.
|
|
|
|
Expired revocation rows are removed at startup and by `AUTH_REVOKED_TOKEN_CLEANUP_CRON` (default `0 3 * * *`, daily at 03:00 server time). The cleanup is not scheduled when authentication is disabled. Browser run notifications use the authenticated `GET /api/v1/scenarios/runs/events` event stream.
|
|
|
|
For local Vite development, the server's default CORS origins are `http://localhost:5173` and `http://127.0.0.1:5173` outside production. Vite has no API proxy: set `VITE_API_URL=http://localhost:3000/api/v1` when using a local server, or use `npm -w client run dev:compose-api` to target `http://localhost:13000`. The API's default CORS list also permits the compose origins `http://localhost:13000` and `http://127.0.0.1:13000` in production. Keep the UI and API on the same host name (`localhost` with `localhost`, or `127.0.0.1` with `127.0.0.1`) so the strict cookie remains same-site. For a different origin, set `CORS_ORIGINS` to the exact comma-separated UI origins; use HTTPS in deployed environments.
|
|
|
|
**Changing the schema** (adding or changing an entity):
|
|
|
|
1. Change the entity and build (`npm -w server run build`).
|
|
2. Make sure your local database is on the latest migration (`migration:run`).
|
|
3. Generate a migration from the difference between the entities and that database:
|
|
|
|
```bash
|
|
npm -w server run migration:generate -- src/database/migrations/AddSomething
|
|
```
|
|
|
|
4. Review the generated file, then add its class to `MIGRATIONS` in `src/database/migrations/index.ts` (they're listed explicitly, oldest first).
|
|
5. Run the tests: `test/migrations.spec.ts` checks that the migrations produce exactly the schema the entities describe, and fails if an entity change has no migration.
|
|
|
|
The initial migration (`Initial…`) adopts databases created before migrations existed (by `synchronize`): if the tables already exist, it's only recorded as applied.
|
|
|
|
## Running tests
|
|
|
|
### Server (Jest)
|
|
|
|
```bash
|
|
npm -w server run test
|
|
```
|
|
|
|
Run a single spec file:
|
|
|
|
```bash
|
|
npm -w server run test -- --no-coverage test/scenario.controller.spec.ts
|
|
```
|
|
|
|
Watch mode:
|
|
|
|
```bash
|
|
npm -w server run test:watch
|
|
```
|
|
|
|
Server application logs use debug output during tests. They remain quiet unless the `test` namespace is enabled. To see NestJS and application logs:
|
|
|
|
```bash
|
|
npm -w server run test:debug
|
|
```
|
|
|
|
### Client (Vitest + Storybook)
|
|
|
|
Storybook interaction tests (runs all `*.stories.tsx` files):
|
|
|
|
```bash
|
|
npm -w client run test:storybook
|
|
```
|
|
|
|
Launch Storybook dev server (with live preview):
|
|
|
|
```bash
|
|
npm -w client run storybook
|
|
```
|
|
|
|
---
|
|
|
|
## Formatting code
|
|
|
|
Both workspaces use [Prettier](https://prettier.io/).
|
|
|
|
**Server** — rewrites `src/**/*.ts` and `test/**/*.ts`:
|
|
|
|
```bash
|
|
npm -w server run format
|
|
```
|
|
|
|
**Client** — rewrites `src/**/*.{ts,tsx}` and `.storybook/**/*.{ts,tsx}`:
|
|
|
|
```bash
|
|
npm -w client run format
|
|
```
|
|
|
|
---
|
|
|
|
## Linting
|
|
|
|
Both workspaces use [ESLint](https://eslint.org/) with `typescript-eslint`. The client also includes `eslint-plugin-react-hooks` and `eslint-plugin-react-refresh`.
|
|
|
|
**Server:**
|
|
|
|
```bash
|
|
npm -w server run lint # report issues
|
|
npm -w server run lint:fix # report and auto-fix
|
|
```
|
|
|
|
**Client:**
|
|
|
|
```bash
|
|
npm -w client run lint # report issues
|
|
npm -w client run lint:fix # report and auto-fix
|
|
```
|
|
|
|
Both workspaces target zero errors. Run lint before committing.
|
|
|
|
---
|
|
|
|
## Building the container
|
|
|
|
The root `Dockerfile` builds a single image with the whole application: it builds the client and the server in separate stages, and the runtime image runs the server, which also serves the built client (`CLIENT_DIR`).
|
|
|
|
The runtime image uses Playwright's headless-only Chromium build (`chromium-headless-shell`) with just the shared libraries it links against, plus DejaVu and Liberation fonts (Liberation is metric-compatible with Arial/Times, so pages lay out as they would with those fonts). All browser launches are headless, so the full desktop Chromium isn't installed. If a new Playwright version needs another library, `ldd` on the `chrome-headless-shell` binary in the image lists what's missing.
|
|
|
|
To keep the image small, production dependencies and the browser are installed in a separate `runtime-deps` stage, which drops what nothing loads at runtime (source maps, TypeScript sources and declarations, docs, TypeORM's browser build, better-sqlite3's C sources, and Playwright's ffmpeg, which is only used for video recording). The runtime stage is plain `debian:bookworm-slim` with only the `node` binary copied in, so the image has no `npm`, `npx`, `yarn` or `corepack`. If a package starts failing in the image only, check whether it reads one of the deleted file types at runtime.
|
|
|
|
```bash
|
|
docker compose build
|
|
```
|
|
|
|
To rebuild without the layer cache:
|
|
|
|
```bash
|
|
docker compose build --no-cache
|
|
```
|
|
|
|
---
|
|
|
|
## Licenses
|
|
|
|
The project is MIT-licensed ([LICENSE](../LICENSE)); the `license` field in all three `package.json` files says the same.
|
|
|
|
The image ships third-party code: the server's production dependencies, the client's production dependencies (bundled into its build), the Node.js binary, Chromium and Debian packages. Their notices are in the image as follows:
|
|
|
|
- **npm packages and Node.js**: the `licenses` stage of the Dockerfile runs `scripts/third-party-licenses.mjs`, which writes the project's LICENSE followed by every production dependency of both workspaces with its license files (or, for the few packages that ship none, the declared license, author and source). Node.js's LICENSE for the exact version in the image is fetched from GitHub and appended. The result is `/app/client/licenses.txt`, which the app serves at `/licenses.txt`. The build fails if a package has no license information at all.
|
|
- **Server packages' own license files** stay in `node_modules`: the slimming step deletes Markdown files except `LICENSE*`, `NOTICE*` and `COPYING*`.
|
|
- **Chromium**: `LICENSE.headless_shell` next to the binary under `/ms-playwright`.
|
|
- **Debian packages**: `/usr/share/doc/*/copyright`.
|
|
|
|
The image also has the OCI labels `org.opencontainers.image.licenses=MIT`, `source`, `title` and `version` (CI passes the version as the `VERSION` build arg).
|
|
|
|
To check the notices locally, or to review a new dependency's license, run from the repository root:
|
|
|
|
```bash
|
|
node scripts/third-party-licenses.mjs > /tmp/licenses.txt
|
|
```
|
|
|
|
Before adding a dependency, check its license is compatible with MIT distribution (permissive licenses such as MIT, ISC, BSD, Apache-2.0 are; copyleft ones such as GPL or AGPL need a closer look).
|
|
|
|
---
|
|
|
|
## CI (Gitea Actions)
|
|
|
|
`.gitea/workflows/ci.yml` runs on pushes to `master`, on pull requests and on release tags (`vX.Y.Z`):
|
|
|
|
- **changes** (not on tags): detects which of `server/` and `client/` changed (via `dorny/paths-filter`), also treating root `package.json`/`package-lock.json` and the workflow file itself as affecting both.
|
|
- **server** (skipped if nothing under `server/` changed; always runs on tags): lint (no warnings allowed), type-check, tests, build.
|
|
- **client** (skipped if nothing under `client/` changed; always runs on tags): lint (no warnings allowed), build, Storybook tests in headless Chromium.
|
|
- **image** (release tags only): after the server and client jobs pass, checks that the tag matches the root `package.json` version (`v<version>`), builds the Docker image and pushes it to the Gitea container registry set by `REGISTRY` at the top of the workflow (`git.hq.ars9.space`) as `<registry>/<owner>/<repo>:<version>` and `:latest`. Merges to `master` don't build images.
|
|
|
|
The image job logs in as the repository owner with the `REGISTRY_TOKEN` secret: a Gitea access token with package write access (Settings → Applications → token with `write:package`), added under the repository's Settings → Actions → Secrets. It runs `docker` on the runner, so the runner must give jobs access to a Docker daemon (act_runner mounts the host's Docker socket into job containers by default).
|
|
|
|
Jobs clone the repository from the address the runner was registered with. If that's an internal hostname (such as `http://gitea:3000` from Gitea's compose network), job containers can't resolve it: either register the runner with the public URL, or set `container.network` in the act_runner config to Gitea's Docker network so jobs join it.
|
|
|
|
---
|
|
|
|
## Running with Docker Compose
|
|
|
|
Start (detached):
|
|
|
|
```bash
|
|
docker compose up -d
|
|
```
|
|
|
|
Build and start in one step:
|
|
|
|
```bash
|
|
docker compose up -d --build
|
|
```
|
|
|
|
Everything is served by the single `liqa` container at **http://localhost:13000**: the UI at `/`, the REST API at `/api/v1`, MCP at `/mcp`, Swagger at `/docs`, health at `/health` and the client's public configuration at `/manifest.json`. Unknown paths outside those routes get the client's `index.html`, so client-side routes survive a page refresh.
|
|
|
|
When switching from the old two-container setup (`server` + `client`), remove the old containers with `docker compose up -d --build --remove-orphans`.
|
|
|
|
The SQLite database and stored files are persisted in `./data/`, which is volume-mounted into the container.
|
|
|
|
Stop and remove containers:
|
|
|
|
```bash
|
|
docker compose down
|
|
```
|
|
|
|
View logs:
|
|
|
|
```bash
|
|
docker compose logs -f
|
|
```
|
|
|
|
---
|
|
|
|
## Client overview
|
|
|
|
The client (`client/`) is a React 19 + TypeScript SPA built with Vite. It provides a browser UI for managing scenarios, runs, snippets, credentials, environments and sessions, and for importing exported YAML.
|
|
|
|
### Tech stack
|
|
|
|
| Library | Purpose |
|
|
|---|---|
|
|
| React 19 + TypeScript | UI framework |
|
|
| Vite 6 | Dev server and bundler |
|
|
| react-router-dom 7 (`BrowserRouter`) | Client-side routing |
|
|
| @tanstack/react-query | Data fetching (cross-scenario runs list) |
|
|
| i18next + react-i18next | Internationalisation |
|
|
| lucide-react | Icon library |
|
|
| @monaco-editor/react | Code editor for steps and snippets |
|
|
| react-markdown + highlight.js | Markdown descriptions and code highlighting |
|
|
| yaml | YAML export of entities |
|
|
| moment | Relative timestamp formatting |
|
|
| ESLint 10 + Prettier | Linting and formatting |
|
|
| Storybook 10 + Vitest | Component development and testing |
|
|
|
|
### Directory layout
|
|
|
|
```
|
|
client/
|
|
src/
|
|
api/ API client (client.ts) and types (types.ts)
|
|
components/ modals/ - run, export, export logs and delete confirmation modals
|
|
hooks/ useTheme, usePageTitle, useRunDefaults, useImportFile, useRunNotifications
|
|
i18n/ i18next bootstrap + locales/en.json
|
|
lib/ highlight.js setup, API error toast events
|
|
pages/ scenario/, run/, snippet/, credential/, environment/, session/, import/
|
|
ui/ Reusable component library (Badge, Button, Card, CodeEditor, Modal,
|
|
Select, SidePanel, Table, Toast, ... - see ui/index.ts)
|
|
.storybook/
|
|
stories/ *.stories.tsx for ui/ components
|
|
```
|
|
|
|
### Routing
|
|
|
|
`BrowserRouter` with plain paths. The root and unknown paths redirect to `/scenarios`. The server answers these paths with the client's `index.html` (see [architecture.md](architecture.md)), so they work on refresh.
|
|
|
|
| Route | Page |
|
|
|---|---|
|
|
| `/scenarios` | Scenario list |
|
|
| `/scenarios/:id` | Scenario details (steps, versions, credentials, used snippets, files) |
|
|
| `/scenarios/:id/runs`, `/scenarios/:id/runs/:runId` | Runs of a scenario, run details |
|
|
| `/runs` | Cross-scenario runs list |
|
|
| `/snippets`, `/credentials`, `/environments`, `/sessions` | Entity lists (plus `/:id`, `/new`, `/:id/edit` where applicable) |
|
|
| `/import` | Central import |
|
|
|
|
### Theming
|
|
|
|
Light/dark theme is toggled by the `ThemeSwitcher` button in the sidebar footer. The selection is persisted to `localStorage` and applied as `data-theme` on `<html>` before React mounts (anti-FOUC inline script in `index.html`).
|
|
|
|
The sidebar footer also shows the server's app version from `/manifest.json` (see [architecture.md](architecture.md#client-client)), falling back to the client build's version (read from the workspace root `package.json` and injected via Vite `define`). If the two differ, the version's tooltip suggests reloading the page.
|