README.md
Rama main · réplica de lectura del repositorio original.
<div align="center">
<img src="frontend/public/ice.png" alt="IceGuard" width="96" height="96" />
# IceGuard
**A modern web console to manage, inspect and maintain Apache Iceberg™ tables across multiple catalogs.**
**🌐 [adelfardi.github.io/iceguard-site](https://adelfardi.github.io/iceguard-site/)**
[](https://adelfardi.github.io/iceguard-site/)
[](https://github.com/adelfardi/iceguard/actions/workflows/ci.yml)
[](https://github.com/adelfardi/iceguard/releases/latest)
[](https://github.com/adelfardi/iceguard/pkgs/container/iceguard-backend)
[](LICENSE)
[](backend)
[](frontend)
[](CONTRIBUTING.md)
<br/>
<img src="https://github.com/adelfardi/iceguard/releases/download/v0.3.0/demo.gif" alt="IceGuard demo: dashboard, versioning graph, storage health, Spark maintenance" width="90%" />
<sub>Dashboard → a table's branches & tags graph, storage health down to each partition, timeline and schema evolution → a Spark maintenance action with its result + logs → catalogs, wizards and pipelines. <a href="https://github.com/adelfardi/iceguard/releases/download/v0.3.0/demo.mp4">▶ MP4 version</a> · <a href="https://demo.iceguard.cloud">live demo</a></sub>
</div>
> ⚠️ **Independent community project — not affiliated with or endorsed by the Apache Software Foundation.**
> "Apache", "Apache Iceberg", "Iceberg" and "Apache Polaris" are trademarks of the ASF.
---
## What is IceGuard?
IceGuard is an admin & maintenance console for the **Apache Iceberg** lakehouse. Point it at one or
more catalogs (REST, Nessie, Polaris) and get a single UI to browse namespaces and tables, evolve
schemas and partitions, edit properties, sample data, inspect storage layout, view lineage, and run
(or schedule) maintenance like snapshot expiration and data-file compaction.
It is **not** a catalog — it sits on top of your existing Iceberg catalogs.
### What IceGuard actually is
IceGuard is **two components**:
1. **Frontend** — the React/TypeScript web UI.
2. **Backend** — the Quarkus REST API.
It has **one required dependency**: a **PostgreSQL** database (you provide it), where the backend
stores IceGuard's own state — registered catalogs, pipelines, schedules, alerts and execution
history. Postgres is not part of IceGuard; it's infrastructure IceGuard needs to run, like a JVM.
IceGuard then **connects to your own** Iceberg catalog(s) and object store.
> The `docker-compose.yml` in this repo (MinIO, an Iceberg REST Catalog, Nessie, Apache Polaris,
> optionally Spark) is **only a local test sandbox** so you can try IceGuard end-to-end without
> bringing your own infrastructure. **It is not part of the product** — in production you point
> IceGuard at your existing catalogs and storage.
## Features
- **Multi-catalog** — register and switch between REST, Nessie, Polaris and Unity Catalog (None/Bearer/OAuth2
supported). Catalogs can be **tagged** and filtered, and edited on a dedicated page (credentials, arbitrary
properties, S3 access modes).
- **Dashboard** — configurable, live widgets (storage, maintenance reliability, recently active partitions,
file sizes, catalogs, executions): add them from a gallery, pin them from a table, drag to reorder.
- **Namespaces & tables** — browse the tree, create namespaces, create/drop/rename tables, insert sample rows.
- **Schema editor** — add / rename / retype / re-doc / drop multiple columns, applied in a **single commit**.
- **Properties editor** — add / update / remove table properties in a **single commit**.
- **Partition evolution** — add or drop partition fields (identity, bucket, truncate, year/month/day/hour).
- **Versioning** — a git-style graph of the snapshot history with one lane per branch and tag; branch, tag
(with retention policy) and snapshot lists, snapshot details and rollback. Nessie history is rebuilt from
its commit log.
- **Storage & health** — point-in-time totals, file-size histogram and per-partition aggregates (server-side
paginated, typed filters), with a **health grade** for the table and for every partition (file size vs
target, small files, delete files; configurable thresholds). Drill down to a partition's files and to a
file's rows, data and delete files alike. See **[docs/STORAGE_HEALTH.md](docs/STORAGE_HEALTH.md)**.
- **Evolution** — schema-version history with column diffs, and a visual snapshot-to-snapshot diff.
- **Timeline** — snapshots + executions on one timeline, click any item for its output/logs.
- **Maintenance** — expire snapshots, rewrite data files (strategy, sort order, `where`, options; whole table or
selected partitions), rewrite manifests, rewrite position / equality delete files, remove orphan files,
rollback. Pluggable executors: a **Java API** executor (real compaction and orphan removal on small,
append-only tables) and a **Spark** executor (local `local[*]` with tunable resources, or a remote Spark
cluster).
- **Commit activity** — the quietest window of the day, detected from snapshot timestamps, to schedule
maintenance off-peak.
- **Pipelines** — chain maintenance actions with per-task parameters, retries and delay, and a cron schedule
built graphically; Airflow-style run view with rerun and retry of a failed task.
- **Alerts** — threshold rules on table metrics with optional SMTP email notifications.
- **Public demo mode** — `ICEGUARD_DEMO_MODE=true` on the frontend serves a read-only UI (banner, writes refused
with a clear message), as on [demo.iceguard.cloud](https://demo.iceguard.cloud).
## Supported catalogs
Any Iceberg REST-compatible catalog works. The **Add Catalog** wizard shows a card per engine, and
the type is auto-detected from the catalog name/URI:
| | Catalog | Typical URI | Auth |
|:---:|---|---|---|
| <img src="frontend/public/logos/iceberg.png" alt="Iceberg REST" height="26"/> | **Iceberg REST** | `http://host:8181` | None / Bearer |
| <img src="frontend/public/logos/nessie.svg" alt="Nessie" height="26"/> | **Nessie** | `http://host:19120/iceberg` | None |
| <img src="frontend/public/logos/polaris.png" alt="Apache Polaris" height="26"/> | **Apache Polaris** | `http://host:8182/api/catalog` | OAuth2 |
| <img src="frontend/public/logos/unity-catalog.svg" alt="Unity Catalog" height="26"/> | **Unity Catalog** | `https://<workspace>/api/2.1/unity-catalog/iceberg-rest` | Bearer (PAT) / OAuth2 |
<sub>Unity Catalog (Databricks or the OSS `unitycatalog.io`) is reached through its Iceberg REST
Catalog API — browse works today; writes depend on the endpoint (Databricks managed-Iceberg
read/write is in preview, requires `EXTERNAL USE SCHEMA` + a PAT/OAuth token).</sub>
Nessie has a few specifics (snapshot history rebuilt from its commit log, S3 region requirement,
flaky-remote handling) — see **[docs/NESSIE.md](docs/NESSIE.md)**.
## Stack
| Layer | Tech |
|-------|------|
| Frontend (product) | React, TypeScript, Vite, Tailwind CSS, shadcn/ui, TanStack Query, Zustand, Recharts |
| Backend (product) | Java 21, Quarkus 3.17, RESTEasy Reactive, Hibernate Panache, Flyway, Apache Iceberg 1.11 |
| Required dependency | PostgreSQL (backend state) |
| Test sandbox only | Docker Compose: MinIO (S3), Iceberg REST Catalog, Nessie, Apache Polaris, (optional) Spark |
## Quick start
Requirements: Docker + Docker Compose. (For hot-reload development: JDK 21 & Maven, Node 20+.)
The app is two services — **frontend** + **backend** — plus a required **Postgres**.
`docker-compose.yml` bundles them in three tiers via Compose profiles:
| Level | Command | Services | When |
|------|---------|----------|------|
| 1 — app (official) | `docker compose up -d --build` | frontend + backend | Bring your own Postgres (`ICEGUARD_DB_URL` in `.env`) |
| 2 — + database | `docker compose --profile db up -d --build` | + Postgres | Self-contained app + its required DB |
| 3 — + test sandbox | `docker compose --profile db --profile sandbox up -d --build` | + MinIO + REST catalog | Try everything locally |
> **No local build?** Use the **published images** from GHCR instead — same levels, just point at
> the images file (no `--build`):
> ```bash
> docker compose -f docker-compose.images.yml --profile db --profile sandbox up -d
> ```
> Defaults to `:latest`; pin a release with `TAG=0.1.0 docker compose -f docker-compose.images.yml …`.
For the quickest tour, use **Level 3** and open **http://localhost:8090**.
#### Add your first catalog (sandbox)
In the UI, go to **Catalogs → Add Catalog** and register the bundled Iceberg REST catalog:
| Field | Value |
|---|---|
| Catalog type | **Iceberg REST** |
| URI | `http://rest-catalog:8181` *(Docker service name — not `localhost`; the backend reaches it over the Compose network)* |
| Warehouse | `s3://warehouse/rest/` |
| Authentication | **None** |
| S3 / Storage access | **Static keys** |
| • Endpoint | `http://minio:9000` |
| • Access key / Secret key | `minioadmin` / `minioadmin` |
| • Region | `us-east-1` |
| • Path-style access | enabled ✅ |
The S3 settings let the backend read/write data files on the sandbox MinIO. Once added, create a
namespace and a table to see browsing, schema, storage and maintenance in action.
Tear down with `docker compose --profile db --profile sandbox down` (add `-v` to wipe data)
— use `-f docker-compose.images.yml` in the same command if you started from the images file.
| Service | URL | Notes |
|---------|-----|-------|
| Frontend (UI) | http://localhost:8090 | nginx; proxies `/api` to the backend |
| Backend API | http://localhost:8080 | Swagger at `/q/swagger-ui` |
| MinIO console | http://localhost:9001 | `minioadmin` / `minioadmin` (sandbox profile) |
| Iceberg REST catalog | http://localhost:8181 | sandbox profile; register it as `http://rest-catalog:8181` |
### Develop (hot reload)
Run the dependencies in Docker and the app from source:
```bash
# deps only (Postgres :5433, REST catalog :8181, Nessie :19120, MinIO) — not the backend container
docker compose -f docker-compose.dev.yml up -d postgres rest-catalog nessie minio minio-init
cd backend && mvn quarkus:dev -Dquarkus.profile=docker # :8080 (Swagger: /q/swagger-ui)
cd frontend && npm install && npm run dev # :5173 (proxies /api to :8080)
```
### Advanced: full multi-catalog sandbox
For every catalog type at once (REST **+ Nessie + Polaris**) and real Spark compaction:
```bash
docker compose -f docker-compose.dev.yml up -d
./scripts/seed-catalog.sh # after the backend is up
```
The **REST Catalog** and **Nessie** demo catalogs work out of the box on MinIO.
**Polaris writes require real AWS S3** — copy `.env.example` to `.env`.
### Kubernetes (Helm)
A chart lives in [`charts/iceguard`](charts/iceguard). Like Level 1 above it deploys the two
IceGuard services only — **frontend + backend** — against a **PostgreSQL you provide**
(images default to the published `ghcr.io/adelfardi/iceguard-{frontend,backend}:0.3.0`):
```bash
helm install iceguard ./charts/iceguard -n iceguard --create-namespace \
--set database.host=pg.internal \
--set database.existingSecret=iceguard-db-credentials # key: password
kubectl port-forward -n iceguard svc/iceguard-frontend 8090:80 # then http://localhost:8090
```
The schema must be empty on first install — Flyway owns it and migrates at startup. A managed
database usually wants the full URL instead: `--set database.jdbcUrl='jdbc:postgresql://…?sslmode=require'`.
Prefer plain `kubectl`? Render the manifests and apply them — Helm is only needed to produce them:
```bash
helm template iceguard ./charts/iceguard -n iceguard --skip-tests \
--set database.host=pg.internal --set database.existingSecret=iceguard-db-credentials \
> iceguard.yaml
kubectl create namespace iceguard && kubectl apply -n iceguard -f iceguard.yaml
```
Ingress, network policies, OIDC, replicas/PDBs and the full values reference are documented in
[`charts/iceguard/README.md`](charts/iceguard/README.md).
### Using IceGuard for real
Deploy the **frontend** and **backend**, point the backend at a **PostgreSQL** you provide
(`QUARKUS_DATASOURCE_JDBC_URL` / username / password), then add your own catalog(s) from the UI
(**Catalogs → Add Catalog**): REST / Nessie / Polaris URI, warehouse, auth (None / Bearer / OAuth2)
and, if needed, S3 credentials. No MinIO/Spark/sandbox containers required.
## Architecture
```
IceGuard (the product) required dep your existing systems
┌───────────────────────────────┐
│ React + Vite (UI, :5173) │
│ │ │
│ Quarkus REST API (:8080) ──────┼──> PostgreSQL ─────────┐
└───────────────────────────────┘ (IceGuard state) │
│ │
└──────────────────> Iceberg catalogs (REST / Nessie / Polaris)
└──> S3 / object store (data & metadata)
```
IceGuard's own state (registered catalogs, pipelines, schedules, alerts, execution history) lives in
the PostgreSQL you provide; table data & metadata live in **your** object store via **your** Iceberg
catalogs.
## Project layout
```
backend/ Quarkus REST API (com.iceguard.*)
frontend/ React + TypeScript SPA
charts/iceguard/ Helm chart (Kubernetes deployment)
scripts/ seed + helper scripts
docker-compose.yml app stack (frontend + backend; profiles: db / sandbox)
docker-compose.dev.yml advanced multi-catalog sandbox (Nessie, Polaris, Spark)
```
## Known limitations
- The **Java** executor really runs compaction & orphan-removal, but only on **small, append-only**
tables (hard size/file-count limits; refuses above them) — use the **Spark** executor for large or
merge-on-read tables.
- **Polaris + MinIO**: browsing works, but writes fail (Polaris ignores the S3-compatible endpoint) — use real AWS S3.
- Catalog credentials are stored in PostgreSQL (not encrypted at rest) — treat the DB as sensitive.
- Test coverage is early-stage. Contributions very welcome 🙂
## Contributing
Contributions, issues and ideas are welcome! Please read **[CONTRIBUTING.md](CONTRIBUTING.md)** and our
**[Code of Conduct](CODE_OF_CONDUCT.md)**. Good first issues are labelled `good first issue`.
How the pipelines and automation work (CI, releases, Dependabot, CodeQL, Trivy) is documented in
**[docs/CI.md](docs/CI.md)**.
## License
[Apache License 2.0](LICENSE).