docs/data-apps.md
Rama main · réplica de lectura del repositorio original.
# Data apps architecture
How data apps work in Lightdash: AI-generated interactive React applications built from a prompt on top of a
project's semantic layer. This is the architecture-level view — shapes, boundaries and invariants that rarely change.
Vocabulary is defined in [`docs/data-apps/CONTEXT.md`](./data-apps/CONTEXT.md); code is the reference for anything
finer-grained. User-facing behaviour is documented at
[docs.lightdash.com/data-apps](https://docs.lightdash.com/data-apps).
---
## What a data app is
- **Source code, not a block model.** A data app is a React project. Lightdash stores the source and the built
assets per version; there is no intermediate content schema.
- **Every prompt appends a version.** Versions are append-only. Restore, duplicate, promotion and as-code upload all
create versions too; nothing rewrites history.
- **Owned by a project**, living either in a space or as a **personal app** of its creator. Identified by a uuid
internally and by a project-scoped slug in URLs and as-code files.
- **Runs in a sandboxed iframe with no API access.** Every query the app makes is proxied by the Lightdash page
hosting it, so permissions, user attributes and row-level filters are enforced by Lightdash, never by the app.
- **Enterprise-only**, gated by a runtime config flag, a feature flag and the data app permission scopes. Custom
chart types are the exception to the feature flag (see below).
## Actors
| Actor | Role |
| ------------------ | ------------------------------------------------------------------------------------------ |
| **User / builder** | Writes prompts, attaches context, previews, iterates, shares. |
| **Coding agent** | The model inside the sandbox that writes and rebuilds the source (Claude Code or Codex). |
| **AI agent** | Lightdash's conversational agent (Ask AI). Finds, reads, and starts builds of data apps; the build itself runs through the same pipeline as the builder. |
| **External agent** | MCP clients read apps; the CLI edits source through data apps as code. Neither generates. |
Only the coding agent writes a data app's source. The AI agent's `generateDataApp` tool creates a personal app
(creation experience `ai_agent`) and enqueues the build; the tool result starts pending and the worker patches it to
success (builder link) or error when the version reaches a terminal status, with a thread-read self-heal for anything
the worker missed.
---
## The build pipeline
Creating or iterating an app appends a version and enqueues a background job. The version moves through ordered
stages — pending, sandbox, catalog, generating, building, packaging, ready — with `error` as the only other terminal
state. Cancellation is an error carrying a "cancelled by user" marker, not a separate state. The frontend polls for
the transition out of the in-progress stages.
1. **Sandbox** — an isolated environment is created (or resumed) from a versioned starter image through a pluggable
sandbox provider. The image ships a React + Vite project, the Lightdash App SDK and the coding agent's skill: a
slim core plus reference files the agent reads on demand.
2. **Catalog** — the project's compiled explores (tables, dimensions, metrics, joins, parameters, AI hints) are
written into the sandbox as YAML, alongside the prompt and any attached context.
3. **Generate** — the coding agent edits the project with scoped file access limited to the app source and the
context directories. Its stream is parsed live into a status narration; output is redacted before it is stored.
4. **Build** — a production Vite build. Build failures trigger bounded auto-fix rounds by the agent.
5. **Package** — built assets are uploaded to object storage and served from there; the source is stored as a
tarball for future iterations and for download as code. Static **data references** (explores, fields, filters,
parameters, linked charts, external hosts) are extracted from the source and persisted with the version.
6. **Metadata** — on the first version the agent names and describes the app.
### Templates and clarifications
A new app starts from a **template** (Dashboard, Slide show, PDF report, From scratch) that seeds the agent's
instructions for the first version only; later prompts work from the existing source. Before the first build a
stateless, best-effort call may ask up to a few **clarifying questions**; answers travel with the version separately
from the verbatim prompt and are folded into the agent's instructions at build time. A clarifier outage never blocks
a build.
### Context
Everything the user attaches to a prompt is persisted with the version and surfaced to the agent as files plus a
one-line pointer per item in the prompt:
- **Charts** — the saved chart's metric query and visualization config. A chart is either **linked** (the app runs
the saved chart live, so later edits flow through) or a **frozen copy** of its query.
- **Dashboards** — expanded into their charts plus a **blueprint** of the dashboard's structure (tabs, tiles,
filters, parameters) so the agent can recreate its design.
- **Sample data** — opt-in per chart or dashboard: the first rows of the query, for content-level decisions. Sample
rows live only inside the sandbox during the build and are never persisted.
- **Images, screenshots of the running preview, and files** — staged in object storage under an opaque id, written
into the sandbox with a kind-aware name so the agent knows a design reference from a screenshot of its own work
from reference material.
- **External connections** — linked APIs, with their shape and any saved samples.
- **Theme** — an organization-level bundle of CSS, fonts, images and instructions, copied into the sandbox; changing
the theme is a style-only iteration.
The agent generates code, not data: it sees the catalog and the attached context, never a warehouse dump.
### Iteration
A follow-up prompt resumes the paused sandbox (keeping the agent's working state) or, if the sandbox is gone,
creates a fresh one and restores the latest ready source. The user can pick the coding agent's model per prompt;
reasoning effort follows the version (lower for first builds, higher for targeted edits).
---
## Versions
- **Ready versions** are the only ones that can be previewed, restored, duplicated or promoted.
- **Preview an older version** is read-only pinning in the builder; the public view always serves the latest ready
version, and the prompt input is locked while pinned because iteration always branches from the head.
- **Restore** copies an earlier ready version's artifacts to a new head version, instantly and without a rebuild.
- **Duplicate** forks any app the caller can view into a new personal app at version 1, seeded from the latest ready
version. History, images, pending clarifications and the sandbox are not carried over.
- **Promote** snapshots the latest ready version from a preview project into its upstream project: the first
promotion creates a linked upstream app, later ones append versions to it. Spaces are mirrored by path; personal
apps stay personal. Promoting a dashboard promotes the apps on its tiles and remaps the tiles. Creating a preview
project duplicates every production app into it with the link already set, so iterating there and promoting
updates the original.
- **Delete** follows the instance-wide soft-delete setting; soft-deleted apps keep their slug and can be restored by
an admin. Hard delete removes rows, the sandbox and every stored artifact.
---
## Runtime
### Serving and isolation
Built assets are served from object storage behind a short-lived, version-scoped token embedded in the preview URL
path — not session cookies — so the iframe can be hosted cross-origin. The iframe runs without same-origin access and
with a strict CSP: scripts, styles and fetches only from the serving origin; images additionally from origins that
linked no-auth connections explicitly opt in. Because the sandbox origin is opaque, the serving origin is listed
alongside `'self'` on every directive. Frame ancestors are defense-in-depth only; the sandbox plus CSP are the real
boundary.
### The bridge
The hosting Lightdash page is the app's only route to data. The SDK inside the iframe posts requests over
`postMessage`; the host validates each against a small allowlist (run a metric query, run a saved chart live, poll
results, underlying data, scheduled exports, current user, external fetch through a linked connection, saved app
state) and executes it with the viewer's own session. The same channel carries capability announces (an SDK feature manifest, screenshot
and lineage availability), URL state sync, the host colour scheme, cache invalidation and the screenshot round trip.
Unknown routes are rejected.
### What the host provides
- **Network inspector** — Queries and Requests tabs capturing everything the app talks to, held in memory unless
persisted; Open in Explore and Save to Lightdash on inline queries; **Inspect data** highlights, via lineage
metadata the agent emits at build time, which rendered elements came from which query.
- **Refresh** — reloads the iframe and re-runs queries without a build, invalidating the results cache for the rest
of the session, in the builder, the standalone view and dashboard tiles alike.
- **Shareable URL state** — the app can publish bounded state that the host mirrors into the URL and, for scheduled
deliveries, into the schedule, so a link or a delivery reopens the same view.
- **Saved app state** — behind the `enable-data-app-state` flag, the host stores values the app saves through the
SDK: per person (`useAppState`) or shared by every viewer with a version and a history of who changed what
(`useSharedAppState`). Writes carry the version last read and conflict instead of overwriting. Embedded viewers
read shared values and never write. Caps: 64 KB per value, 200 keys per app per scope, 1,000 history versions
per key.
- **Host colour scheme** — the app follows the host's light/dark mode through a handshake; capture surfaces force
light.
- **Thumbnails** — one per ready version, captured automatically when a build becomes ready or by hand from the running app. An app's card shows its latest ready version's thumbnail.
- **Exports** — CSV/XLSX and PDF exports run through the normal Lightdash export pipeline; the app never serializes
data itself.
- **Embedding and dashboards** — apps render as dashboard tiles (dashboard filters propagate to linked charts) and in
embeds, where the JWT grants app viewing.
### Trust boundaries
| Boundary | Guarantee |
| ---------------------------- | -------------------------------------------------------------------------------------------------- |
| Sandbox → Lightdash | The agent's egress is allowlisted to its model provider; it never holds Lightdash credentials. |
| Iframe → Lightdash | No API access; only allowlisted bridge routes, executed as the viewer. |
| Uploads | Clients hold opaque ids only; storage paths are derived server-side, so no direct object reference. |
| External connections | Credentials are injected server-side by the proxy; the app and browser never see them. |
| Screenshots / lineage | Rasterised inside the iframe from its own DOM; the host never reads the app's DOM. |
---
## Data model
Entities, not columns:
- **App** — identity (uuid, project-scoped slug), name and description, space or personal, template, theme, sandbox
handle, link to its upstream app when it lives in a preview project, soft-delete markers.
- **Version** — sequence number, verbatim prompt, status and status narration, error, attached context
(clarifications, chart/dashboard/image/file/connection references, model and theme snapshot), viz schema for
custom chart types, declared dependencies, generation usage, data references.
- **Access grants** — per-app user and group access, in addition to space access.
- **Links** — external connections linked to an app; dashboard tiles that reference an app.
- **App state** — per-user values (app + user + key), shared values (app + key, versioned) and shared history
(every version with its author, `user` today and `agent` reserved for server-side writers). JSON values.
Deleting a shared key removes its current value but keeps its history, and a recreated key continues the
version sequence, so a writer holding a version from before the delete still conflicts. Key caps are
enforced inside the write transaction under an advisory lock.
Artifacts (built assets, source tarballs, staged uploads, thumbnails) live in object storage under the app's prefix.
---
## Permissions
Checks live in the service layer against a subject that carries the app's space context and creator:
- **View** requires the project role that can run queries (plain viewers cannot open apps) plus space access, or
ownership for personal apps. Embedded viewers get it from the JWT.
- **Create** is an editor-level project scope; creating directly inside a space additionally requires manage rights
on that space. Developers can create and manage apps in their own preview projects.
- **Manage** (iterate, cancel, edit, move, pin, delete, thumbnails, upgrade, restore a version, promote, upload as
code) is scoped to the app's space or to the owner's personal apps; project admins manage everything.
- **Duplicate is deliberately looser**: it needs only view on the source plus create on the project, because it
never touches the source app.
- **Restore from soft delete and permanent delete** are admin-only and bypass the contextual check.
- **Custom dependencies** are a separate admin-level scope on top of an organization feature flag.
- **App state** reads need view; writes need view and a registered session (embeds are read-only); deleting a
shared key needs manage. All of it sits behind the `enable-data-app-state` flag, resolved per user before
organization. Because view starts at interactive viewer, plain viewers cannot use it either. Every viewer who
can write may write any shared key; per-app comment-only or edit roles are not modelled.
Once an app moves into a space, its creator has no special rights; access follows the space.
---
## External connections
A project admin registers a third-party HTTP API — base URL, allowed methods and path prefixes, auth, size and time
limits, optional instructions — and chooses whether builders may link it. Apps link a connection under an alias and
fetch through a Lightdash-mediated proxy that pins the host, enforces the method and path rules, injects credentials
server-side and audits byte counts, never bodies. Admins can test a connection through the exact proxy path and save
sanitised samples that are handed to the coding agent so generated code matches the API's real shape. Connections
are also a content-as-code resource; secrets never travel in files.
### Forwarding viewer identity
Connection managers can enable **Forward viewer identity** for an individual connection. The API and
content-as-code field is `forwardUserIdentity`; it defaults to `false` for new and existing connections.
Omitting it from a PATCH preserves the current value. Enabling it shares the available request identity
with that connection's destination using backend-set headers:
| Header | Value |
| ----------------------------- | -------------------------------------------------------------- |
| `X-Lightdash-User-Email` | The requesting principal's email, when available |
| `X-Lightdash-User-Id` | The requesting principal's Lightdash user UUID, when available |
| `X-Lightdash-App-Id` | The app UUID authorized for the request |
| `X-Lightdash-Organization-Id` | The connection's organization UUID |
| `X-Lightdash-Project-Id` | The connection's project UUID |
Lightdash forwards known attributes without requiring email verification or a human viewer. Missing
attributes are omitted. JWT embeds can forward their token's email claim, but have no Lightdash user UUID;
service accounts use their own principal's user UUID. The app creator and connection owner are never used
as substitutes for the requester.
App code cannot supply or override these headers. Their names are reserved case-insensitively and cannot
be configured as custom headers or API-key headers, even when forwarding is disabled. Existing configurations
using reserved names must remove those collisions. Connection credentials are still sent as usual.
These identity headers are unsigned: the external service must authenticate Lightdash through its trusted
connection credential or network boundary before using them for authorization. It decides how to handle
missing attributes and whether the supplied email is sufficient for its policy.
Connection tests respect unsaved settings and send the testing administrator's available identity,
organization, and project, without an app ID. Identity is attached only to proxied resource requests,
including OAuth retries; it is not sent to OAuth token endpoints or on direct browser image requests.
### Trusted private destinations on self-hosted instances
Operators can set `APP_RUNTIME_EXTERNAL_CONNECTION_ALLOWED_PRIVATE_HOST_CIDRS` to comma-separated
`hostname@CIDR` entries, for example `api.internal.example@10.20.0.0/16,auth.internal.example@10.30.1.5/32`.
Repeat a hostname to allow multiple ranges. Hostnames are case-insensitive and a trailing DNS dot is ignored.
This instance-wide setting allows external connections, their test requests, and OAuth client-credentials
requests to reach explicitly approved private/internal addresses. A private destination must match both
its hostname and one of that host's CIDRs; every DNS answer is checked. Public addresses keep their existing
behavior. Token hosts must be listed separately when they differ from the API host.
Operators are trusted to choose safe destinations: explicitly allowed CIDRs can include loopback and
metadata addresses, and hostname entries cover all HTTPS ports. HTTPS certificate verification, DNS pinning,
redirect blocking, and connection permissions and method/path limits remain unchanged. The setting does not
apply to other outbound fetches. It defaults to empty, takes effect at backend startup, and rejects malformed
entries with a configuration error. Backend routing, DNS, and trusted certificates must already be available.
---
## Data apps as code
An app's source can be downloaded, edited locally, previewed against real data and uploaded as a new version with the
CLI. Identity travels by slug: uploading to a project where the slug exists appends a version, otherwise creates the
app. Only the source is sent; the server ignores local scaffolding and rebuilds in its own trusted sandbox, so the
trust model is unchanged. Downloads carry a point-in-time context snapshot (semantic layer, theme instructions) for
local reference. Local creation scaffolds the same starter template. Custom npm dependencies are an experimental,
flag-gated extension: registry-only, lockfile required, screened for malicious packages, install scripts never run.
---
## Custom chart types
A custom chart type is a data app built from a dedicated template that declares a viz schema instead of running
queries: the explorer hands it rows and a field mapping, and it renders. They share the pipeline, storage and
permissions of data apps but are excluded from app listings, have their own gallery, and are created and edited in
**Chart Studio**, available as a standalone page and within the explorer. They are downloaded as code separately.
Chart Studio does not follow the `enable-data-apps` flag: it is available with a valid licence and a configured
sandbox provider, reported as `dataApps.runtimeAvailable` on the health response. The coding agent key is resolved
per organization when a build starts. It keeps the data app permission scopes and AI credits.
A type can carry one icon from a curated Tabler set; the as-code manifest's `icon` field round-trips
it (omitted for non-chart-type apps), null clears it, and an off-list value is rejected on upload.
Official chart types can also be installed prebuilt from a chart registry, are read-only once installed, and are
customized by forking — the registry, library, install and fork model is documented in
[`docs/chart-types.md`](./chart-types.md).
A saved chart pins the version of the type it was saved with, so iterating on a type never changes existing charts.
Charts saved before pins exist follow the latest version until they are next edited and saved. When a newer version
exists, the explorer's configure panel offers an upgrade that lists the field, option and palette changes; upgrading
re-pins the chart being edited and reconciles its field mapping and option values against the new contract, and
nothing is persisted until the chart is saved.
Chart Studio supplies the current schema and field mapping to the coding agent, plus a saved-chart reference
when opened from a saved chart. Authors can attach a screenshot of the preview, pick an element, and opt in to
**Include sample data** for the next build. Samples use the current Explorer or Test-panel results, are bounded
to 10 rows and 20 columns, and respect the instance's sample-data setting. Context travels separately from the
user's prompt; sample data is not stored in version history. Queued builds refresh their schema, mapping and
opted-in sample data together when they start.
Chart-type builds omit the explores catalog and remove stale catalog files from reused sandboxes. Their SDK
bridge blocks independent queries while allowing underlying-data interactions resolved by the host.
### Ordered field inputs
A viz schema field can set `multiple: true` to accept an ordered selection of metrics,
dimensions, series, or any result columns. For example:
```json
{ "name": "values", "label": "Measures", "type": "metric", "required": true, "multiple": true }
```
`useVizContext().fieldMapping.values` then contains an array such as
`["orders_total_revenue", "orders_total_order_amount", "orders_order_count"]`. The chart
iterates that array in order. A multiple dimension input uses the same declaration with
`"type": "dimension"`. Fields appear in the order they are added. Replacing a field keeps
its position; removing and adding it again appends it to the end. The saved chart and
chart-as-code config preserve that selection order. `[]` records an explicitly
cleared input; a required input with no selections is unbound.
Omitting `multiple` (or setting it to `false`) retains the original string binding in
`fieldMapping`. Multiple inputs use ordered arrays in that same mapping in saved
charts, chart-as-code, and SDK context. Narrow with `Array.isArray(binding)` before
iterating; older single-field bindings remain strings.
Compatible upgrades preserve the order of surviving bindings. Changing an input from
single to multiple moves its current field into an array; changing back retains the
first compatible field as a string and appears in the upgrade summary. Registry
manifests and app-as-code `vizSchema` use the same declaration schema.
---
## Infrastructure
| Concern | Shape |
| ------------------ | -------------------------------------------------------------------------------------------------- |
| Sandbox | Pluggable provider (E2B by default; Docker for local dev; cloud micro-VM options for self-hosting). |
| Coding agent | Claude Code via the Anthropic API or Bedrock, or Codex; corporate gateways supported. |
| Starter image | Versioned with each release; iteration resumes on the version an app was built with. |
| Object storage | S3-compatible bucket for artifacts, uploads and thumbnails, optionally separate from the main one. |
| Jobs | The scheduler worker runs the pipeline; the API only enqueues. |
Configuration is documented for operators in the self-host docs; the parsed config in the backend is the source of
truth for names and defaults.
---
## Where to look
- `packages/common/src/ee/apps/` — shared types, version stages, data references, SDK bridge routes and features.
- `packages/backend/src/ee/services/AppGenerateService/` — the pipeline, authorization, coding agent environments.
- `packages/backend/src/ee/clients/AppThumbnailClient.ts` — thumbnail rules: capture, manual images, copies, which image an app shows. Callers authorize first.
- `packages/backend/src/routers/appPreviewRouter.ts` — serving, tokens, CSP.
- `packages/frontend/src/features/apps/` — builder, preview, bridge, inspector.
- `packages/frontend/src/features/chartTypes/builder/` — standalone Chart Studio.
- `packages/frontend/src/components/Explorer/ChartTypeAuthoring/` — Chart Studio within the explorer.
- `packages/query-sdk/` — the SDK shipped inside apps.
- `sandboxes/data-apps/` — the starter template, skill and references, benchmark harness.
- `packages/cli/src/handlers/apps/` — data apps as code.