docs/development/testing-and-data.md

Rama main · réplica de lectura del repositorio original.

# Testing & Test Data

How we test the parts of YourPHR that meet real-world FHIR — the import engine, the Layer-1 classifiers, and the patient-legible display — and which test data to use for what. Living document; append as the strategy evolves.

See also: [`data-quality-framework.md`](./data-quality-framework.md) (the DQF this testing asserts against — our doctrine + per-source quality profiles), [`test-sandboxes.md`](./test-sandboxes.md) (sandbox accounts + status), and the project `AGENTS.md` (only *synthetic* fixtures may be committed).

## The concern

Our automated coverage of __data we actually encounter__ is thin: classifiers are tested mostly with small hand-written fixtures, plus a couple of Synthea bundles. The bugs we hit in practice are not caught by those.

Two parts to the gap:

1. __Breadth__ — not enough realistic, varied data exercised.
2. __Assertions__ — *having* more data (e.g. loading patients into a dev instance) only helps *manual* QA. Regression protection needs __golden tests__: a fixed input plus an expected output. More bundles without assertions don't move the needle.

## Two failure classes — cover both

- __US-Core conformance / happy path__ — "do we display every compliant profile correctly + completely." Caught by clean, compliant data (Synthea, US-Core IG examples, Inferno).
- __Non-US-Core vendor quirks__ — the *reason this fork exists*. Every real bug this project hit (Epic `class` = `HOV` rendered raw instead of the readable `type[].text` "Outpatient"; duplicate conditions; a Cerner export that is documents-only with no `Patient`/`Encounter`) is a __vendor non-conformance__, not a spec violation. Compliant data would pass all of them.

Implication: __clean sources give breadth; vendor exports give the messy reality.__ For *this* app, the most "real life" data is the __vendor sandbox exports__, because that is the shape patients actually upload (Epic MyChart, Cerner, Veradigm, Blue Button) — not idealized US-Core.

## Test-data sources

| Source | Best for | Caveat | License |
|---|---|---|---|
| Hand-written per-classifier fixtures | targeted logic branches | tiny; not realistic | n/a (ours) |
| __Synthea / SyntheticMass__ | volume + a complete happy-path patient | "too clean" — never reproduces vendor non-conformance | Apache-2.0 |
| __Vendor sandbox exports__ (Epic Camila, Cerner nancysmart, …) | the non-US-Core quirks that bite us | sandbox ≠ production; narrow; provenance must be the *raw* vendor response | public synthetic |
| __VA Clinical Health__ ([#370](https://github.com/jwilleke/yourphr/issues/370)) | another real-world US-Core-aligned vendor shape (VistA/Cerner); veterans mission | live OAuth API (not a ZIP) — fixtures require a SMART connect + export; onboarding cost | public synthetic |
| __US-Core IG examples__ | canonical per-profile conformance ("every profile we display") | happy-path; a better fit than base-R4 fhir-test-cases for a US-Core PHR | HL7 (permissive — verify) |
| __Inferno US-Core test kit__ | edge cases across US-Core 3.1.1 → 8.0+; what EHRs are certified against | data embedded in a server test kit → extraction effort | Apache-2.0 |
| __fhir-test-cases__ ([#354](https://github.com/jwilleke/yourphr/issues/354)) | spec validity + intentionally-*invalid* R4 (import resilience) | base R4, not US-Core; modest ROI | Apache-2.0 |

All listed licenses are permissive — commit only __curated subsets__ with a NOTICE/attribution, and never a real (non-synthetic) export.

## Can we trust the vendor sandbox slices?

Yes *for what they are*, no *if we overclaim them*.

__Trustworthy as:__

- __PHI-safe__ — Epic/Cerner sandbox patients are public + synthetic (see [`test-sandboxes.md`](./test-sandboxes.md)).
- __Genuinely vendor-shaped__ — e.g. the Camila Encounter carries Epic's own OID arc (`1.2.840.114350.*`) and an `open.epic.com/FHIR/StructureDefinition/...` extension. That is real Epic output; the `HOV` quirk is an authentic Epic convention, which is exactly why it is worth testing.

__Limits (do not overclaim):__

- __Sandbox ≠ production__ — curated + limited; real prod data is messier and more varied.
- __Narrow__ — one patient per vendor is a thin slice even of the sandbox.
- __Provenance is the real risk__ — if a slice is something *we re-exported through our own app* rather than the __raw vendor FHIR response__, a golden built on it can be self-confirming (asserting our output against our own prior transformation).
- __Snapshot drift__ — vendors can change sandbox data. Fine for a *frozen fixture*, but only if we record what it is.

__The reframe that resolves it:__ a golden test's trust comes from __(1) a frozen, genuinely-real-shaped input + (2) a human-authored expected output__ — not from the input being exhaustive. Camila does not have to represent all of Epic; she has to be *real Epic* (she is), and our expected "HOV → Outpatient" has to be *what we decided correct looks like* (reviewed by a human). That is a legitimate regression guard — as long as we never call it "Epic coverage."

## The golden-test harness

One __table-driven runner__: `bundle/resource → import → classify → display → assert`. Seed it in tiers:

- __Vendor depth (highest value):__ Epic/Cerner slices — assert the legible output (HOV → "Outpatient", conditions dedup, docs-only empty state). This is the regression net for the patient-legible work ([#262](https://github.com/jwilleke/yourphr/issues/262)).
- __Conformance breadth:__ curated US-Core IG examples — one per profile we display.
- __Full patient:__ one Synthea patient, end-to-end.
- __Resilience:__ Inferno validation-triggering + fhir-test-cases malformed → assert we skip-and-survive (never crash on bad input).

Adding a newly-found quirk later = drop a fixture + its expected output. That is the durable win.

## Guardrails

- __Commit only synthetic/public__ slices into `backend/pkg/database/testdata/` (and `frontend/src/lib/fixtures/`). Real exports stay in gitignored `sample-data/`.
- __Verify raw-ness__ — each committed vendor slice must be the *raw vendor FHIR response*, not an app re-export.
- __Provenance header per fixture__ — source, patient id, sandbox base URL, export date, FHIR/US-Core version. Answers "what is this + can I re-derive it" in the file.
- __Human-authored expected outputs__ — the assertions are our spec, not derived from the input.
- __Pair vendor with the US-Core-IG baseline__ — so we can tell a *vendor quirk* from *spec*.

## Credentials in tests

- __Test credentials are generated at run time, never written as literals__ (for example `` `pw-${randomUUID()}` `` from `node:crypto`). A password, token, key or client secret in a test is a throwaway made by the test itself.
- __No real secret is ever committed, tests included.__ Real values live in the instance's `.env` or the operator's secret store, never in the repo.
- __Why it matters:__ GitGuardian doesn't scan test code (`.gitguardian.yaml`: `**/__tests__/**`, `e2e/**`, `**/*.spec.ts`, `scripts/*-tests.ts`), so it no longer raises alerts for test credentials. It also wouldn't catch a real secret pasted into a test by mistake. This rule is the only guard there.

## Issue map

- [#354](https://github.com/jwilleke/yourphr/issues/354) — integrate __fhir-test-cases__ (one source: spec validity + malformed corpus).
- [#369](https://github.com/jwilleke/yourphr/issues/369) — medical-history grouping endpoint / perf (the document-heavy fixture lives here).
- [#262](https://github.com/jwilleke/yourphr/issues/262) — patient-legible display (the vendor golden assertions guard this).
- [#131](https://github.com/jwilleke/yourphr/issues/131) — E2E; a non-US-Core (Epic-shaped) seed would exercise the legible path end-to-end.
- [#370](https://github.com/jwilleke/yourphr/issues/370) — add __VA Clinical Health__ as a SMART provider (self-serve sandbox; yields a VA-shaped fixture as a bonus).
- *Proposed:* an umbrella issue — "realistic test-data corpus + golden harness" — relating the above, pulling US-Core-IG + Synthea + vendor slices + Inferno.

## Recommended first steps

1. __Verify provenance__ of `sample-data/epic/…Camila….json` (raw vendor response vs our re-export); add a provenance header.
2. __Prototype the harness with one vendor fixture__ — the Camila `HOV → "Outpatient"` assertion — to set the shape before scaling.
3. __Then__ widen: US-Core-IG breadth, more sandbox patients, the resilience tier.
4. File the umbrella issue once the shape is proven.