- trigger the image job on vX.Y.Z tag pushes instead of master pushes - run server and client checks unconditionally on tags before the build - fail the build when the tag doesn't match the package.json version - document the release routine: version bump, changelog and tag are done on master after work branches are merged
351 lines
15 KiB
Markdown
351 lines
15 KiB
Markdown
# Development
|
|
|
|
## Prerequisites
|
|
|
|
- Node.js 24+ (required by `engines` in `package.json`; the Docker image uses Node.js 24)
|
|
- Docker + Docker Compose
|
|
|
|
The repository is a **npm workspace monorepo** with two packages:
|
|
|
|
| Workspace | Directory | Description |
|
|
|---|---|---|
|
|
| `server` | `server/` | NestJS API + MCP server |
|
|
| `client` | `client/` | React 19 + Vite admin UI |
|
|
|
|
Install all dependencies from the repo root:
|
|
|
|
```bash
|
|
npm install
|
|
npx playwright install --only-shell chromium # first time only: headless Chromium for the server
|
|
```
|
|
|
|
No `.env` is needed: every setting below has a default. To override settings, copy `.env.example` to `.env` and uncomment the ones you need. Docker Compose loads `.env` when it exists.
|
|
|
|
### Environment variables (server)
|
|
|
|
All settings are validated at startup by `server/src/config/app.config.ts`.
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `NODE_ENV` | `development` | `production` in the Docker image. Shown in the startup log, and selects the default CORS origins |
|
|
| `CORS_ORIGINS` | *(empty)* | Comma-separated origins allowed to call the API cross-origin. When empty: the Vite dev server (`http://localhost:5173`, `http://127.0.0.1:5173`) unless `NODE_ENV=production`, then the compose port (`http://localhost:13000`, `http://127.0.0.1:13000`) |
|
|
| `PORT` | `3000` | HTTP port |
|
|
| `DB_PATH` | `data/liqa.db` | SQLite database file |
|
|
| `FILES_DIR` | `data/files` | Where scenario files and run artifacts are stored |
|
|
| `FILE_RUN_EXPIRATION_DAYS` | `7` | How long run artifacts (downloads, failure dumps, traces) are kept |
|
|
| `FILE_CLEANUP_CRON` | `0 3 * * *` | When expired files are deleted |
|
|
| `SESSION_IDLE_TIMEOUT_MINUTES` | `30` | Idle timeout before open sessions are closed |
|
|
| `SESSION_DELETE_CLOSED_DAYS` | `7` | Retention period for closed sessions |
|
|
| `PLAYWRIGHT_TRACE_MODE` | `off` | Default Playwright trace mode for runs: `on`, `retain-on-failure` or `off` (see [scenario.md](scenario.md)) |
|
|
| `RUN_LOG_DATA_COLLAPSE_BYTES` | `256` | Run log `data` blocks whose pretty-printed JSON is over this many bytes start collapsed in the UI (exposed via `/manifest.json`) |
|
|
| `RUN_LOG_DATA_COLLAPSE_LINES` | `5` | Same, for the number of lines |
|
|
| `CLIENT_DIR` | *(empty)* | Directory with the built client to serve at `/` (the Docker image sets it to `client`); empty means the server doesn't serve the UI |
|
|
| `PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH` | _(Playwright default)_ | Path to a Chromium binary to use instead of Playwright's own. Not set in Docker: the image ships Playwright's `chromium-headless-shell` (matching the Playwright version) under `PLAYWRIGHT_BROWSERS_PATH=/ms-playwright`. |
|
|
|
|
---
|
|
|
|
## Starting the application
|
|
|
|
### Server
|
|
|
|
**Development** (watch mode, restarts on file changes):
|
|
|
|
```bash
|
|
npm -w server run start:dev
|
|
```
|
|
|
|
**Production build, then start**:
|
|
|
|
```bash
|
|
npm -w server run build
|
|
npm -w server run start:prod
|
|
```
|
|
|
|
The server listens on port **3000** by default.
|
|
|
|
### Client
|
|
|
|
**Development** (Vite dev server with HMR):
|
|
|
|
```bash
|
|
npm -w client run dev
|
|
```
|
|
|
|
The Vite dev server runs on **http://localhost:5173**. It has no API proxy: the client calls `VITE_API_URL` (default `/api/v1` on its own origin), and the server allows CORS from `http://localhost:5173`. So point it at a running server:
|
|
|
|
```bash
|
|
npm -w client run dev:compose-api # API of the compose container (localhost:13000)
|
|
VITE_API_URL=http://localhost:3000/api/v1 npm -w client run dev # API of a locally started server
|
|
```
|
|
|
|
**Production build**:
|
|
|
|
```bash
|
|
npm -w client run build
|
|
```
|
|
|
|
The built files are emitted to `client/dist/`.
|
|
|
|
---
|
|
|
|
## Database migrations
|
|
|
|
The SQLite schema is managed by TypeORM migrations in `server/src/database/migrations/`; `synchronize` is off.
|
|
|
|
- **Docker:** the container applies pending migrations before starting the server.
|
|
- **Locally:** build, then apply them before starting the server (it refuses to start while migrations are pending):
|
|
|
|
```bash
|
|
npm -w server run build
|
|
npm -w server run migration:run # apply pending migrations to DB_PATH
|
|
npm -w server run migration:show # list migrations and whether they're applied
|
|
```
|
|
|
|
The scripts run from `server/`, so `DB_PATH` (default `data/liqa.db`) is relative to it, and they read `server/.env` if present, like the server does.
|
|
|
|
**Changing the schema** (adding or changing an entity):
|
|
|
|
1. Change the entity and build (`npm -w server run build`).
|
|
2. Make sure your local database is on the latest migration (`migration:run`).
|
|
3. Generate a migration from the difference between the entities and that database:
|
|
|
|
```bash
|
|
npm -w server run migration:generate -- src/database/migrations/AddSomething
|
|
```
|
|
|
|
4. Review the generated file, then add its class to `MIGRATIONS` in `src/database/migrations/index.ts` (they're listed explicitly, oldest first).
|
|
5. Run the tests: `test/migrations.spec.ts` checks that the migrations produce exactly the schema the entities describe, and fails if an entity change has no migration.
|
|
|
|
The initial migration (`Initial…`) adopts databases created before migrations existed (by `synchronize`): if the tables already exist, it's only recorded as applied.
|
|
|
|
## Running tests
|
|
|
|
### Server (Jest)
|
|
|
|
```bash
|
|
npm -w server run test
|
|
```
|
|
|
|
Run a single spec file:
|
|
|
|
```bash
|
|
npm -w server run test -- --no-coverage test/scenario.controller.spec.ts
|
|
```
|
|
|
|
Watch mode:
|
|
|
|
```bash
|
|
npm -w server run test:watch
|
|
```
|
|
|
|
Print the NestJS logs of the test app (they're silenced by default; `DEBUG=test`):
|
|
|
|
```bash
|
|
npm -w server run test:debug
|
|
```
|
|
|
|
### Client (Vitest + Storybook)
|
|
|
|
Storybook interaction tests (runs all `*.stories.tsx` files):
|
|
|
|
```bash
|
|
npm -w client run test:storybook
|
|
```
|
|
|
|
Launch Storybook dev server (with live preview):
|
|
|
|
```bash
|
|
npm -w client run storybook
|
|
```
|
|
|
|
---
|
|
|
|
## Formatting code
|
|
|
|
Both workspaces use [Prettier](https://prettier.io/).
|
|
|
|
**Server** — rewrites `src/**/*.ts` and `test/**/*.ts`:
|
|
|
|
```bash
|
|
npm -w server run format
|
|
```
|
|
|
|
**Client** — rewrites `src/**/*.{ts,tsx}` and `.storybook/**/*.{ts,tsx}`:
|
|
|
|
```bash
|
|
npm -w client run format
|
|
```
|
|
|
|
---
|
|
|
|
## Linting
|
|
|
|
Both workspaces use [ESLint](https://eslint.org/) with `typescript-eslint`. The client also includes `eslint-plugin-react-hooks` and `eslint-plugin-react-refresh`.
|
|
|
|
**Server:**
|
|
|
|
```bash
|
|
npm -w server run lint # report issues
|
|
npm -w server run lint:fix # report and auto-fix
|
|
```
|
|
|
|
**Client:**
|
|
|
|
```bash
|
|
npm -w client run lint # report issues
|
|
npm -w client run lint:fix # report and auto-fix
|
|
```
|
|
|
|
Both workspaces target zero errors. Run lint before committing.
|
|
|
|
---
|
|
|
|
## Building the container
|
|
|
|
The root `Dockerfile` builds a single image with the whole application: it builds the client and the server in separate stages, and the runtime image runs the server, which also serves the built client (`CLIENT_DIR`).
|
|
|
|
The runtime image uses Playwright's headless-only Chromium build (`chromium-headless-shell`) with just the shared libraries it links against, plus DejaVu and Liberation fonts (Liberation is metric-compatible with Arial/Times, so pages lay out as they would with those fonts). All browser launches are headless, so the full desktop Chromium isn't installed. If a new Playwright version needs another library, `ldd` on the `chrome-headless-shell` binary in the image lists what's missing.
|
|
|
|
To keep the image small, production dependencies and the browser are installed in a separate `runtime-deps` stage, which drops what nothing loads at runtime (source maps, TypeScript sources and declarations, docs, TypeORM's browser build, better-sqlite3's C sources, and Playwright's ffmpeg, which is only used for video recording). The runtime stage is plain `debian:bookworm-slim` with only the `node` binary copied in, so the image has no `npm`, `npx`, `yarn` or `corepack`. If a package starts failing in the image only, check whether it reads one of the deleted file types at runtime.
|
|
|
|
```bash
|
|
docker compose build
|
|
```
|
|
|
|
To rebuild without the layer cache:
|
|
|
|
```bash
|
|
docker compose build --no-cache
|
|
```
|
|
|
|
---
|
|
|
|
## Licenses
|
|
|
|
The project is MIT-licensed ([LICENSE](../LICENSE)); the `license` field in all three `package.json` files says the same.
|
|
|
|
The image ships third-party code: the server's production dependencies, the client's production dependencies (bundled into its build), the Node.js binary, Chromium and Debian packages. Their notices are in the image as follows:
|
|
|
|
- **npm packages and Node.js**: the `licenses` stage of the Dockerfile runs `scripts/third-party-licenses.mjs`, which writes the project's LICENSE followed by every production dependency of both workspaces with its license files (or, for the few packages that ship none, the declared license, author and source). Node.js's LICENSE for the exact version in the image is fetched from GitHub and appended. The result is `/app/client/licenses.txt`, which the app serves at `/licenses.txt`. The build fails if a package has no license information at all.
|
|
- **Server packages' own license files** stay in `node_modules`: the slimming step deletes Markdown files except `LICENSE*`, `NOTICE*` and `COPYING*`.
|
|
- **Chromium**: `LICENSE.headless_shell` next to the binary under `/ms-playwright`.
|
|
- **Debian packages**: `/usr/share/doc/*/copyright`.
|
|
|
|
The image also has the OCI labels `org.opencontainers.image.licenses=MIT`, `source`, `title` and `version` (CI passes the version as the `VERSION` build arg).
|
|
|
|
To check the notices locally, or to review a new dependency's license, run from the repository root:
|
|
|
|
```bash
|
|
node scripts/third-party-licenses.mjs > /tmp/licenses.txt
|
|
```
|
|
|
|
Before adding a dependency, check its license is compatible with MIT distribution (permissive licenses such as MIT, ISC, BSD, Apache-2.0 are; copyleft ones such as GPL or AGPL need a closer look).
|
|
|
|
---
|
|
|
|
## CI (Gitea Actions)
|
|
|
|
`.gitea/workflows/ci.yml` runs on pushes to `master`, on pull requests and on release tags (`vX.Y.Z`):
|
|
|
|
- **changes** (not on tags): detects which of `server/` and `client/` changed (via `dorny/paths-filter`), also treating root `package.json`/`package-lock.json` and the workflow file itself as affecting both.
|
|
- **server** (skipped if nothing under `server/` changed; always runs on tags): lint (no warnings allowed), type-check, tests, build.
|
|
- **client** (skipped if nothing under `client/` changed; always runs on tags): lint (no warnings allowed), build, Storybook tests in headless Chromium.
|
|
- **image** (release tags only): after the server and client jobs pass, checks that the tag matches the root `package.json` version (`v<version>`), builds the Docker image and pushes it to the Gitea container registry set by `REGISTRY` at the top of the workflow (`git.hq.ars9.space`) as `<registry>/<owner>/<repo>:<version>` and `:latest`. Merges to `master` don't build images.
|
|
|
|
The image job logs in as the repository owner with the `REGISTRY_TOKEN` secret: a Gitea access token with package write access (Settings → Applications → token with `write:package`), added under the repository's Settings → Actions → Secrets. It runs `docker` on the runner, so the runner must give jobs access to a Docker daemon (act_runner mounts the host's Docker socket into job containers by default).
|
|
|
|
Jobs clone the repository from the address the runner was registered with. If that's an internal hostname (such as `http://gitea:3000` from Gitea's compose network), job containers can't resolve it: either register the runner with the public URL, or set `container.network` in the act_runner config to Gitea's Docker network so jobs join it.
|
|
|
|
---
|
|
|
|
## Running with Docker Compose
|
|
|
|
Start (detached):
|
|
|
|
```bash
|
|
docker compose up -d
|
|
```
|
|
|
|
Build and start in one step:
|
|
|
|
```bash
|
|
docker compose up -d --build
|
|
```
|
|
|
|
Everything is served by the single `liqa` container at **http://localhost:13000**: the UI at `/`, the REST API at `/api/v1`, MCP at `/mcp`, Swagger at `/docs`, health at `/health` and the client's public configuration at `/manifest.json`. Unknown paths outside those routes get the client's `index.html`, so client-side routes survive a page refresh.
|
|
|
|
When switching from the old two-container setup (`server` + `client`), remove the old containers with `docker compose up -d --build --remove-orphans`.
|
|
|
|
The SQLite database and stored files are persisted in `./data/`, which is volume-mounted into the container.
|
|
|
|
Stop and remove containers:
|
|
|
|
```bash
|
|
docker compose down
|
|
```
|
|
|
|
View logs:
|
|
|
|
```bash
|
|
docker compose logs -f
|
|
```
|
|
|
|
---
|
|
|
|
## Client overview
|
|
|
|
The client (`client/`) is a React 19 + TypeScript SPA built with Vite. It provides a browser UI for managing scenarios, runs, snippets, credentials, environments and sessions, and for importing exported YAML.
|
|
|
|
### Tech stack
|
|
|
|
| Library | Purpose |
|
|
|---|---|
|
|
| React 19 + TypeScript | UI framework |
|
|
| Vite 6 | Dev server and bundler |
|
|
| react-router-dom 7 (`BrowserRouter`) | Client-side routing |
|
|
| @tanstack/react-query | Data fetching (cross-scenario runs list) |
|
|
| i18next + react-i18next | Internationalisation |
|
|
| lucide-react | Icon library |
|
|
| @monaco-editor/react | Code editor for steps and snippets |
|
|
| react-markdown + highlight.js | Markdown descriptions and code highlighting |
|
|
| yaml | YAML export of entities |
|
|
| moment | Relative timestamp formatting |
|
|
| ESLint 10 + Prettier | Linting and formatting |
|
|
| Storybook 10 + Vitest | Component development and testing |
|
|
|
|
### Directory layout
|
|
|
|
```
|
|
client/
|
|
src/
|
|
api/ API client (client.ts) and types (types.ts)
|
|
components/ modals/ - run, export, export logs and delete confirmation modals
|
|
hooks/ useTheme, usePageTitle, useRunDefaults, useImportFile
|
|
i18n/ i18next bootstrap + locales/en.json
|
|
lib/ highlight.js setup, API error toast events
|
|
pages/ scenario/, run/, snippet/, credential/, environment/, session/, import/
|
|
ui/ Reusable component library (Badge, Button, Card, CodeEditor, Modal,
|
|
Select, SidePanel, Table, Toast, ... - see ui/index.ts)
|
|
.storybook/
|
|
stories/ *.stories.tsx for ui/ components
|
|
```
|
|
|
|
### Routing
|
|
|
|
`BrowserRouter` with plain paths. The root and unknown paths redirect to `/scenarios`. The server answers these paths with the client's `index.html` (see [architecture.md](architecture.md)), so they work on refresh.
|
|
|
|
| Route | Page |
|
|
|---|---|
|
|
| `/scenarios` | Scenario list |
|
|
| `/scenarios/:id` | Scenario details (steps, versions, credentials, used snippets, files) |
|
|
| `/scenarios/:id/runs`, `/scenarios/:id/runs/:runId` | Runs of a scenario, run details |
|
|
| `/runs` | Cross-scenario runs list |
|
|
| `/snippets`, `/credentials`, `/environments`, `/sessions` | Entity lists (plus `/:id`, `/new`, `/:id/edit` where applicable) |
|
|
| `/import` | Central import |
|
|
|
|
### Theming
|
|
|
|
Light/dark theme is toggled by the `ThemeSwitcher` button in the sidebar footer. The selection is persisted to `localStorage` and applied as `data-theme` on `<html>` before React mounts (anti-FOUC inline script in `index.html`).
|
|
|
|
The sidebar footer also shows the server's app version from `/manifest.json` (see [architecture.md](architecture.md#client-client)), falling back to the client build's version (read from the workspace root `package.json` and injected via Vite `define`). If the two differ, the version's tooltip suggests reloading the page.
|