rules/user-app-test-isolation.md

Rama main · réplica de lectura del repositorio original.

# User-app test isolation (Tests panel)

Applies to `src/ipc/services/isolated_test_db.ts`, `test_case_lifecycle_server.ts`, `src/ipc/utils/supabase_test_user.ts`, `neon_test_data.ts`, and the generated Playwright fixture shim in `playwright_bootstrap.ts`. The concurrent-run bullets at the end also apply to the run scheduler (`src/ipc/services/test_run_queue_service.ts`, `neon_test_branch_lease.ts`), E2E port allocation in `e2e_test_runtime.ts`, and the Tests panel's queue labels.

- **Discover cleanup targets from `pg_class`, not `information_schema.columns`.** The latter also lists views, and `DELETE` on a non-updatable view fails with `55000 cannot delete from view` (seen in user logs as `Best-effort cleanup of public.<view>... failed`). Filter `c.relkind IN ('r', 'p') AND NOT c.relispartition`, as `neon_test_data.ts` and `supabase_test_user.ts` do.
- **Per-test lifecycle work runs before every test case, so each Management API round trip multiplies.** Batch per-table SQL into one request, giving each statement its own `BEGIN … EXCEPTION WHEN others … END` so one failing table does not roll back the others. A `DO` block cannot return rows, so to report per-table errors use a `CREATE OR REPLACE FUNCTION pg_temp.…` that returns them, followed by `SELECT` (this assumes the Management API returns the last statement's rows, which unit tests cannot confirm — parse the result defensively). `WHEN others` does **not** catch `QUERY_CANCELED`, so a statement timeout still rolls back the whole block; keep a per-statement fallback, since separate requests commit independently. SQL is mocked in the unit tests, so they cannot catch transaction semantics like this.
- **The per-test timeouts are nested and must move together:** server hook (`TEST_CASE_HOOK_TIMEOUT_MS`) < fixture `fetch` (`TEST_CASE_REQUEST_TIMEOUT_MS`) < Playwright fixture timeout (`TEST_CASE_FIXTURE_TIMEOUT_MS`), all derived in `test_case_lifecycle_server.ts`. Raising only the server value just turns a clear timeout error into a bare client abort. The fixture shim is regenerated on every run (while it carries the Dyad sentinel), so shim-side changes need no config migration.
- **Supabase fixture admin credentials belong only in `PreparedIsolation.testRunnerEnv`.** Inject `SUPABASE_SECRET_KEY` and the matching `SUPABASE_URL` into the Node Playwright runner in both sandbox and normal-preview paths; keep them out of dotenv files, install/dev-server environments, recorder auth, IPC results, and telemetry.
- Redact runner-only environment values before streaming output or returning results. Buffer partial credentials across pipe chunks and strip ANSI highlighting before matching; Playwright assertion diffs can insert escape codes inside a secret. Verify with a real spec that prints the key and fails an assertion on it.
- Sanitize owned Playwright reports and error-context files after every runner exit, including discovery, cancellation, timeout, and spawn failure, before retaining artifacts. Remove opaque attachments when credentials were injected; stream/IPC redaction alone leaves secrets readable on disk. After deletion, remove missing attachment references from reports and explain removed screenshots in the failure details, so the panel and agent never receive stale screenshot paths. Set `bootstrapPreviewRouted: true` when testing the preview runner with `skipBootstrap`, or the test silently exercises the ordinary runner.
- **Dyad's automatic user provisioning precedes spec `beforeEach` hooks.** Per-case Supabase tests may continue without default credentials only after a recognized HTTP 500 database-create rejection and a read-only check that no auth user exists for the attempted email. Recognize both Auth's generic error and structured PostgreSQL constraint (SQLSTATE class 23) or raised-exception (`P0001`) errors, never arbitrary message text. Other errors, cancellation, and cleanup failures remain fatal; recordings still require successful provisioning. Custom fixtures own cleanup of their users and prerequisite rows.

- Tests that mock or skip Playwright bootstrap but launch a real runner must generate its DNS preload with `ensurePreviewDnsPreload`; otherwise Node fails before the spec loads. Assert preload paths against `await fs.promises.realpath(appPath)`, matching the runner's native path resolution; `fs.realpathSync` can retain Windows 8.3 aliases such as `RUNNER~1` while the runner expands them to `runneradmin`.

- Playwright clears its configured output directory at startup. Keep generated timing reporters and their JSONL output outside the directory passed to `--output`; real-runner tests must use the same layout or a passing run can silently produce no case timing events.
- Playwright's `TestResult.duration` excludes fixtures with their own timeout, including database provisioning. Use the top-level `Before Hooks` and `After Hooks` spans for setup/cleanup timing and the interval between them for execution; do not subtract those hook durations from `TestResult.duration`.
- When diagnosing parallel startup failures on macOS, check wildcard listeners: probes on `127.0.0.1` and `::1` can both bind while an existing `::` listener occupies the port. Port reservations must cover the running server's lifetime, not just startup.
- A run timing status of `completed` does not prove every case passed; inspect the retained Playwright report and case outcomes before attributing inconclusive results to concurrency. Unique test accounts still share provider auth rate limits, independently of local CPU/RAM usage.
- Neon capacity waits must be published per run, including followers behind the project admission lock. The app-level `neonBranchLimit` notice identifies only one run; use each active run's `waitingForNeonBranch` flag for queue and Tests-panel labels, and preserve individual case selections.
- Serial admission clears a run record's `context`, and `compatibleWithRun` treats a missing context as incompatible. Checks against other active runs (such as `validateTestRunInputs`) must skip context-less runs; otherwise turning parallel runs off mid-setup fails in-flight parallel runs with a misleading "no longer compatible" error. Likewise, never collapse a `loadTestRunContext` failure into that compatibility error: rethrow its `DyadError` (missing app/folder) or wrap and log other errors, since no sibling run may exist.
- Once a finished run leaves `activeRuns` while a sibling keeps running, `applyTestRunFinishedAtom` falls back to the sibling's projected identity and rejects a second full finish. Delayed spec-list re-keying must pass `reconcileOnly: true`, which touches only results and skips anything a newer run owns. Ownership must be checked per case too: `buildSingleTestFileResult` (partial and grep merges) returns a file aggregate with no `runId`, so a file-level check alone lets an older run overwrite a newer case result.
- When processes fail to settle, the handlers deliberately skip `prepared.teardown()`. Anything released only by teardown then never settles — a Neon lease left in `active` makes branch-capacity waiters wait forever. Call `prepared.retainForRecovery()` on that path, and give any new shared-capacity registry a retire path that drops the live owner without deleting the provider state kept for startup recovery.
- In the generated fixture shim, `context.route()` disables Playwright's HTTP cache for the whole browser context, so an opt-out that merely calls `route.fallback()` still changes app behavior. Skip installing the route instead. Only annotations declared in test or describe details (`test("…", { annotation: … }, …)`) are visible at fixture setup; `test.info().annotations.push` runs after the `context` fixture.
- A run with surviving owned processes still occupies a concurrency slot after its queue entry finishes. Scope cleanup conflicts to its app/workspace/database so independent apps can proceed, but count active runs plus unsettled owners against the global limit until `hasUnsettledE2eTestProcesses` clears. An empty child registry is not settlement: a daemonized server's wrapper `close` drops the tracked child while the server still listens, so every failed verdict (`stop()` returning false, failed `settleE2eTestProcesses`) must reach `retainUnsettledE2eTestOwner(signal)` via `onProcessSettlementFailed`. That record lasts until quit, so `tests_handlers*.test.ts` (one shared module-level queue service) must mock it as a spy, or every later run in the file times out at 5s.
- Admission context failures: fail a `NotFound`/`Precondition` `DyadError` from `loadTestRunContext` immediately, and retry other failures on the 2s admission interval rather than the 30s context refresh, so a persistent failure does not hold its app's FIFO for a minute.

- Keep application-authored Supabase remediation in `SupabaseTestUserActionError` or an explicitly recognized error type such as `SupabaseTokenRefreshError`; an `External` classification alone does not make arbitrary provider text safe to display. Preserve existing error kinds; classify unknown local failures as `Internal`, reserving `External` for recognized HTTP/transport failures. For unexpected local failures, retain bounded error types, SQLite codes, and Dyad source locations without copying exception messages or SQL parameters.
- Token refresh guidance must distinguish 400/401/403 auth rejections (`Auth`, reconnect) from 429 (`RateLimited`, wait/retry) and server failures (`External`, retry). Cover both legacy and organization-scoped refresh paths without copying provider response text.
- Classify rejected Auth response-body reads with the same operation wrapper as the request; swallowing `response.text()` failures loses timeout and cancellation identity. Keep successful create responses draining until the returned user ID is persisted, even during cancellation.
- `SupabaseManagementAPIError` from `@dyad-sh/supabase-management-js` stores HTTP status in `response.status`, not `status`. Extract it from the recognized SDK error and validate an integer in 400–599; use the real SDK error in regression tests without exposing its private message or response body.
- Preserve `SupabaseTestUserCreationRejectedError` identity for the per-case fixture fallback, but project its database/trigger guidance when recordings stop on the same rejection.
- Supabase setup failures must say that tests or recording were stopped before execution while preserving the classified remediation. Do not claim existing data was untouched: setup can already create or clean up temporary users in the real project before it fails.