README.md
Rama main · réplica de lectura del repositorio original.
<div align="center">
<img src="docs/media/architecture.svg" alt="Eco-Guard architecture" width="100%">
# Eco-Guard
**A self-hosted inference gateway for teams who want their models behind one
OpenAI-compatible API, with the operational furniture that usually gets left
out: model management, cost accounting, live hardware telemetry, and a control
plane that is actually pleasant to use.**
[](https://github.com/swadhinbiswas/Ecoguard/actions/workflows/ci.yml)
[](https://swadhinbiswas.github.io/Ecoguard/)
[](https://www.python.org/)
[](https://fastapi.tiangolo.com/)
[](https://vuejs.org/)
[](LICENSE)
[Website](https://swadhinbiswas.github.io/Ecoguard/) ·
[Documentation](https://swadhinbiswas.github.io/Ecoguard/docs/) ·
[API reference](https://swadhinbiswas.github.io/Ecoguard/docs/api-reference/inference.html) ·
[Architecture decisions](https://swadhinbiswas.github.io/Ecoguard/docs/adr/) ·
[Benchmarks](#benchmarks)
</div>
---
<img src="docs/media/ui/overview.png" alt="Eco-Guard overview dashboard" width="100%">
## Why this exists
Running a model locally is easy for one person on one laptop. Running it for a
team is where things get messy: who gets access, which model is loaded, what
each request costs, why latency doubled last night, and what happens when the
GPU box falls over.
Eco-Guard is the layer around the model. It loads GGUF files (or talks to
Ollama, vLLM, TGI, or OpenAI), serves them over `/v1/chat/completions`, records
what happened, enforces limits, and puts a UI on top so nobody has to read a
log file to answer "is it up?". It deploys as one container with SQLite by
default and PostgreSQL + Redis when you need them.
I built it because I wanted the operational side of local inference to be
boring, and because most "LLM platform" repos are either a thin wrapper around
a single endpoint or an unmaintainable sprawl of half-finished features. This
one aims to be the middle: small enough to read, complete enough to run.
## What it does
**Model management**
- Scans a directory for GGUF files and reads their metadata (architecture,
parameter count, quantization, context length, chat template) without
loading them.
- Downloads models from Hugging Face with live progress, cancel, and resume-safe
`.part` files. Search and file listing are built into the UI.
- Loads and unloads models on demand, sizing context and GPU offload from the
GGUF header and live hardware telemetry (VRAM, RAM, CPU).
- Supports a dedicated embedding model via `EMBEDDING_MODEL_PATH`; otherwise the
loaded model serves embeddings when it can.
**Serving**
- OpenAI-compatible `/v1/chat/completions` (streaming and non-streaming, tools,
JSON mode), `/v1/completions`, `/v1/embeddings`, `/v1/models`.
- Native API with chat, completions, batch (up to 100 prompts with per-item
error isolation), model comparison, and SSE streaming.
- Real streaming: generation runs on a worker thread and is bridged to async
with backpressure, so the event loop stays free and client disconnects stop
the work.
- Backends: local `llama.cpp`, or any OpenAI-compatible endpoint (Ollama, vLLM,
TGI, OpenAI). Switching is one environment variable.
**Operating it**
- Prometheus metrics plus a WebSocket feed for the UI: latency, tokens/s,
time-to-first-token, queue depth, cache hit rate, GPU utilization, VRAM,
temperature, power.
- Structured JSON logs with request correlation, OpenTelemetry tracing (OTLP).
- Health and readiness probes that know the difference between "process is
alive" and "can serve inference".
- Latency drift detection with a z-score baseline and optional retraining
triggers.
**Controlling it**
- API keys stored as SHA-256 hashes with roles (`admin`, `operator`, `member`,
`viewer`) enforced per route.
- scrypt password hashing, account lockout, HTTP-only JWT cookies, SSO
(Google/GitHub) with CSRF state validation.
- Sliding-window rate limiting that is exact across workers when Redis is
present and correct in-process when it is not.
- Content safety pipeline: prompt-injection detection, PII redaction, content
classification, with per-stage actions and a dry-run endpoint.
- Per-workspace budgets and token quotas, audit logging, and retention cleanup.
- IP allowlisting by CIDR for the whole API surface, and optional trusted-host
enforcement.
**Chat and playground**
- Persistent chat sessions with streaming responses, system prompts, and
per-session sampling parameters.
- Playground that runs one prompt across up to four models and shows output,
latency, and tokens/s side by side.
## Screenshots
| Chat | Models |
|---|---|
|  |  |
| Observability | Playground |
|---|---|
|  |  |
## Quick start
### Docker Compose (recommended)
```bash
git clone https://github.com/swadhinbiswas/Ecoguard.git
cd Ecoguard
export ECOGUARD_DOCKER_POSTGRES_PASSWORD=$(openssl rand -hex 16)
export ECOGUARD_DOCKER_JWT_SECRET=$(openssl rand -hex 32)
export ECOGUARD_DOCKER_ADMIN_PASSWORD=$(openssl rand -hex 16)
docker compose up --build -d
```
Open <http://localhost:8000>. The landing page is at `/`; the console is at
`/app`, and the first visit walks through a three-step setup (admin account →
download a model → load it) before you have a working endpoint. The first build
compiles `llama-cpp-python`, which takes a few minutes. Set
`ECOGUARD_INSTALL_LOCAL=0` if you plan to point Eco-Guard at Ollama or vLLM
instead.
### Local development
```bash
uv sync --group dev --extra local # Python 3.12; the extra installs llama.cpp
uv run uvicorn src.main:app --reload # http://localhost:8000
# Optional: the Vue dev server with hot reload
cd frontend && npm install && npm run dev
```
Without `DATABASE_URL` the app falls back to SQLite in `./data` and applies
migrations on start, so nothing else has to be running. Production refuses to
start unless PostgreSQL, a JWT secret, admin credentials, and explicit CORS
origins are configured.
### Then use it like OpenAI
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="eg_your_key")
response = client.chat.completions.create(
model="default",
messages=[{"role": "user", "content": "Explain GGUF in two sentences."}],
)
print(response.choices[0].message.content)
```
```bash
curl http://localhost:8000/v1/chat/completions \
-H "Authorization: Bearer eg_your_key" \
-H "Content-Type: application/json" \
-d '{"model":"default","messages":[{"role":"user","content":"hi"}],"stream":true}'
```
## Deploy
Eco-Guard is one ASGI process plus a database for its metadata and models on
disk. The [deployment guide](docs/operations/deployment.md) walks through each
path, and every environment variable is listed in
[configuration](docs/operations/configuration.md).
### Docker Compose
`docker compose up --build -d` starts `api`, `redis` (7-alpine) and `postgres`
(16-alpine). The API waits for the PostgreSQL health check, mounts `./models`,
`./data` and `./backups`, health-checks `/api/v1/health` every 30 seconds with
a 60-second start period, and runs with `ENVIRONMENT=production`. Three
variables are mandatory; Compose fails at startup without them:
```bash
export ECOGUARD_DOCKER_JWT_SECRET=$(openssl rand -hex 32)
export ECOGUARD_DOCKER_ADMIN_PASSWORD=$(openssl rand -hex 16)
export ECOGUARD_DOCKER_POSTGRES_PASSWORD=$(openssl rand -hex 16)
```
`ECOGUARD_DOCKER_ADMIN_USERNAME` defaults to `admin`, and
`ECOGUARD_DOCKER_CORS_ORIGINS` defaults to `http://localhost:8000`.
Postgres and Redis keep named volumes, so `docker compose down` leaves the data
and `docker compose down -v` removes it. TLS and request-size limits belong to
the proxy in front; the container listens on plain HTTP. The first build
compiles `llama-cpp-python`, which takes a few minutes. Set
`ECOGUARD_INSTALL_LOCAL=0` when you will point at Ollama or vLLM instead and do
not want to pay for that compile.
### Docker image
The multi-stage `Dockerfile` builds the frontend, copies the virtualenv into a
slim runtime image, and runs as the `ecoguard` user. The default command is
`uvicorn src.main:app --host 0.0.0.0 --port 8000`, and the image health-checks
`/api/v1/health` every 30 seconds with a 45-second start period.
`llama-cpp-python` is an optional extra, so `docker build` leaves it out unless
you ask for it. Either build the slim image and point Eco-Guard at a remote
backend, or compile llama.cpp into it:
```bash
docker build -t eco-guard:latest . # remote backends
docker build --build-arg INSTALL_LOCAL=1 -t eco-guard:local . # llama.cpp inside
docker run -p 8000:8000 -v "$PWD/models:/app/models" eco-guard:local
```
### Kubernetes
`k8s/` holds the ConfigMap, Deployment, Service, HPA, NetworkPolicy and
PodDisruptionBudget. The Deployment runs two replicas, takes `DATABASE_URL`,
`JWT_SECRET`, `ADMIN_USERNAME`, `ADMIN_PASSWORD` and `API_KEYS` from a Secret
named `eco-guard-secrets` that you create, reads everything else from
`eco-guard-config`, and probes `/api/v1/health` for liveness and
`/api/v1/ready` for readiness. Models come from a PVC called
`eco-guard-models-pvc`, which no manifest ships, so create both before
applying:
```bash
kubectl create pvc eco-guard-models-pvc --size 20Gi
kubectl create secret generic eco-guard-secrets \
--from-literal=JWT_SECRET="$(openssl rand -hex 32)" \
--from-literal=ADMIN_USERNAME=admin \
--from-literal=ADMIN_PASSWORD="$(openssl rand -hex 16)" \
--from-literal=DATABASE_URL="postgresql+asyncpg://user:pass@postgres:5432/ecoguard" \
--from-literal=API_KEYS=""
kubectl apply -f k8s/
```
The pod runs with a read-only root filesystem, all capabilities dropped, and
`/app/models` mounted from the PVC so downloads still work from the UI. A
commented nodeSelector and `nvidia.com/gpu` limit in the Deployment cover GPU
scheduling when you run llama.cpp on a GPU node.
`monitoring/servicemonitor.yaml` registers the service with the Prometheus
Operator.
### Helm
`helm/eco-guard/` renders ServiceAccount, ConfigMap, Secret, Deployment,
Service, HPA, PersistentVolumeClaim, Ingress, ServiceMonitor, NetworkPolicy and
PodDisruptionBudget:
```bash
helm upgrade --install eco-guard ./helm/eco-guard \
--values my-values.yaml \
--set image.repository=ghcr.io/you/eco-guard \
--set ingress.host="ecoguard.example.com"
```
The template refuses to render without `secrets.jwtSecret`,
`secrets.adminUsername` and `secrets.adminPassword`, so keep them in a values
file; command lines end up in shell history and process listings.
`replicaCount` starts at 2, `autoscaling`
goes from 2 to 10 replicas at 70% CPU or 80% memory, `modelVolume` requests
20Gi, `redis.enabled` is off, and the pod runs as UID 1000 with a read-only
root filesystem.
One gap to close before you install: the Secret builds `DATABASE_URL` from
`postgresql.auth` for a service named `<release>-postgresql`, which is the
Bitnami subchart's name, and `Chart.yaml` does not declare that dependency.
Either declare it and run `helm dependency update ./helm/eco-guard`:
```yaml
# helm/eco-guard/Chart.yaml
dependencies:
- name: postgresql
version: ">=0"
repository: https://charts.bitnami.com/bitnami
condition: postgresql.enabled
```
or edit the Secret template to point at the PostgreSQL you already run.
### Manual
In `production` the app refuses to start when `auth_enabled` (the default) is
on and `JWT_SECRET` is shorter than 32 characters or `ADMIN_USERNAME` and
`ADMIN_PASSWORD` are not both set, when `DATABASE_URL` is missing or is not a
`postgresql+asyncpg://` URL, when `CORS_ORIGINS` contains `*`, or when
`COOKIE_SECURE` is false. Passwords taken through the API need 8 characters or
more. CORS otherwise allows the local dev origins, so point `CORS_ORIGINS` at
the origin you actually serve the UI from:
```bash
make migrate # when AUTO_MIGRATE=false
uv run uvicorn src.main:app --host 0.0.0.0 --port 8000 --workers 4
```
Set `ROOT_PATH` when the app is not mounted at the domain root. Build the
frontend with `make ui` if you want the landing page and console served by the
same process; without `frontend/dist/` the API answers and the root path does
not. Set `REDIS_URL` before running more than one worker or replica; the next
section explains why.
### Render
`render.yaml` is a working blueprint: Python 3.12, the frontend built at
deploy time, managed PostgreSQL, a generated `JWT_SECRET`, and a health check
on `/api/v1/health`. Render has no GPU, so the blueprint sets `BACKEND=remote`
and expects `BACKEND_URL`, `BACKEND_API_KEY` and `BACKEND_MODEL` to point at
Ollama, vLLM, TGI or OpenAI.
### The website
GitHub Pages serves the landing page at the repository root and the
documentation under `/docs/`, both published by the `Docs` workflow. It builds
the docs with `uv run python -m tools.docs.build`, builds the frontend with
`VITE_BASE=/<repo>/`, and uploads one artifact containing the landing page,
`docs/`, a `.nojekyll` file, and a copy of `index.html` named `404.html`. That
copy is what makes deep links work: GitHub serves `404.html` for
`/app/overview`, the SPA boots with the original URL still in the address bar,
and the router resolves it. Links from before the move, such as
`/architecture/overview.html`, are sent to `/docs/architecture/overview.html`.
A push to `main` that touches `frontend/`, `docs/` or `tools/docs/` republishes
both.
## Scale
### Put Redis in front first
Without `REDIS_URL` the rate limiter, the circuit state and cache coordination
all live inside each process, so two replicas each enforce their own limit and
each believe the circuit is closed. Add Redis, then confirm `GET
/api/v1/system/status` reports `redis.connected: true`. With it, a limit set
once holds across every uvicorn worker.
### Scale with replicas
One model, one process. llama.cpp serialises on a lock because one context
generates one sequence at a time, and `MAX_CONCURRENT_INFERENCE` (default 4)
only decides how many requests are admitted before they queue. Requests beyond
that show up as `ecoguard_inference_queue_depth` and
`ecoguard_requests_in_flight`.
Throughput per replica is therefore fixed by the model and the hardware, so
growth comes from replicas behind a load balancer. The chart's HPA (2 to 10
pods at 70% CPU or 80% memory) and PodDisruptionBudget (`minAvailable: 1`) keep
enough of them alive through deploys.
`uvicorn --workers 4` earns its keep when the API sits in front of a remote
backend, where this process handles rate limiting, budgets, caching and
dashboards while token generation happens elsewhere. With local inference each
worker loads its own copy of the model, so more workers means more memory and
no extra tokens per second.
### Tuning that moves the numbers
| Knob | Default | What it does |
|---|---|---|
| `MODEL_N_GPU_LAYERS` | `-1` | auto: all layers on the GPU when 90% of free VRAM clears the model size |
| `MODEL_N_CTX` | `4096` | context per request; the KV cache is roughly 8K tokens per GB for a 7B-class model |
| `MODEL_N_THREADS` | `0` | auto: physical cores, capped at 16; CPU decode is memory-bandwidth bound |
| quantization | in the file | Q4_K_M is the usual size/quality point; bigger quants need more VRAM and more time per token |
| `MAX_CONCURRENT_INFERENCE` | `4` | requests admitted before queueing; decode itself stays serial |
| `STREAM_QUEUE_SIZE` | `256` | chunks buffered per slow streaming client before backpressure |
`GET /api/v1/system/hardware` reports the CPU, RAM and GPU figures the
sizing rules use, `GET /api/v1/models/recommendation?model_id=...` returns the
GPU layer, context and thread counts it would pick for that model, and
`GET /api/v1/models/loaded` reports what the backend actually received.
Caching keeps repeat requests off the queue: exact-match caching is on by
default for deterministic requests (temperature near 0) with a 300-second TTL
and 1000 entries. Semantic caching stays off until you give it a model whose
embeddings are worth trusting.
### Watching and backing it up
`monitoring/grafana/dashboard.json` imports into Grafana as-is, and
`monitoring/prometheus/alerts.yml` ships seven rules: 5xx error rate, p95
latency, drift above the threshold, model unloaded, rate limiting, low token
throughput and a down instance. Everything else exported at `/metrics` is there
for panels you add yourself: queue depth, time to first token, cache hits,
circuit state, GPU utilisation, VRAM and temperature.
`POST /api/v1/mlops/backups` shells out to `pg_dump` and writes a
timestamped dump, with `pg_restore` on the other side; SQLite uses
`sqlite3 .backup`. The models directory is plain files, so whatever backs up
the disk covers them. [Backups](docs/operations/backups.md) has the commands
and the verification steps.
## Benchmarks
Measured with `make benchmark MODEL=...` on a desktop with an Intel i5-13600K,
32 GB RAM and an RTX 3060 Ti. The PyPI `llama-cpp-python` wheel is CPU-only, so
these numbers are **CPU inference** for `Qwen2.5-0.5B-Instruct` (Q4_K_M,
0.46 GB, 32k context):
| Metric | Value |
|---|---|
| Model load | 0.58 s |
| Time to first token (avg) | 70.6 ms |
| Time to first token (p95) | 71.1 ms |
| Full response latency (avg, 128 tokens) | 1.94 s |
| Generation speed | 57.7 tok/s |
| Generation speed (best) | 59.1 tok/s |
| 1 concurrent request | 1.59 req/s |
| 4 concurrent requests | 1.45 req/s |
| Embedding latency | 61.3 ms |
Reproduce with `uv run python -m benchmarks.benchmark --model <path.gguf>`; the
raw JSON lands in `benchmarks/results.json`. For GPU inference, install a CUDA
build (`CMAKE_ARGS="-DGGML_CUDA=on" uv pip install llama-cpp-python
--no-binary llama-cpp-python`) or build the image with
`--build-arg CMAKE_ARGS="-DGGML_CUDA=on"`. Local inference is serialised per
model: one GPU context generates one sequence at a time, so a second request
waits for the first.
## Architecture
Requests pass through an explicit middleware stack, outermost first:
```
SecurityHeaders → CORS → RequestContext → BodySizeLimit → GZip
→ RateLimit → ShutdownGuard → Auth → DemoReadOnly → routes
```
CORS sits outside auth and rate limiting so error responses still carry CORS
headers; request logging sits outside auth so rejected requests are logged with
a correlation id. `src/main.py` builds the stack by wrapping the ASGI app
explicitly, so the outermost-first order is visible in one place and does not
depend on `add_middleware`'s ordering rules.
```
src/
├── main.py # app factory, lifespan, middleware composition
├── api/ # curated routers (system, models, inference, chat,
│ # openai, mlops, analytics, admin, workspaces, auth, ws)
├── core/
│ ├── backends/ # llama.cpp + OpenAI-compatible backends, thread bridge
│ ├── auth/ # users, API keys, JWT, RBAC, SSO, middleware
│ ├── model_manager.py # GGUF discovery, downloads, load/unload
│ ├── gguf.py # dependency-free GGUF parser
│ ├── hardware.py # CPU/RAM/GPU detection and model sizing
│ ├── guardrails.py # injection, PII, content safety
│ └── ... # config, errors, cache, rate limit, tracing, gitops
├── services/ # inference, chat store, cost, drift, notifications
├── mlops/ # registry, experiments, datasets, jobs, drift,
│ # evaluation, benchmarks, prompts, RAG, backups
├── db/ # engine, session, model registry (30 tables)
├── schemas/ # pydantic request/response models
└── monitoring/ # Prometheus metrics, GPU polling
frontend/ # Vue 3 SPA (14 views, no chart/UI dependencies)
tools/docs/ # Markdown → static docs site generator
alembic/ # one clean linear migration history
benchmarks/ # inference benchmark harness
```
The API surface is curated: 159 routes including the OpenAI surface. Three
prompt-template systems, two audit systems and two backup systems were
consolidated, and stubbed features were removed. See
[ADR-0006](docs/adr/0006-curated-api-surface.md).
## Observability
Prometheus metrics are exposed at `/metrics`:
```
ecoguard_http_requests_total{method,route,status}
ecoguard_inference_duration_seconds{backend}
ecoguard_time_to_first_token_seconds{backend}
ecoguard_inference_tokens_total{direction,backend}
ecoguard_inference_tokens_per_second{backend}
ecoguard_model_loaded
ecoguard_requests_in_flight
ecoguard_cache_events_total{result}
ecoguard_guardrail_events_total{guardrail,action,severity}
ecoguard_backend_errors_total{backend,kind}
ecoguard_circuit_state{name}
ecoguard_gpu_utilization_percent{gpu,name}
ecoguard_gpu_memory_used_mb{gpu,name}
ecoguard_drift_score
```
The UI subscribes to `/ws/metrics` for a live feed, and the Grafana dashboard in
`monitoring/grafana/` ships ready to import.
## Security
- Passwords are hashed with scrypt (per-password salt, constant-time compare);
failed logins lock the account for a configurable window.
- API keys are 256-bit random tokens stored as SHA-256 hashes, which are fast
to verify and impossible to brute-force. The token is shown exactly once, at
creation.
- Sessions are HTTP-only, SameSite cookies; SSO uses state validation and
refuses unverified email addresses.
- Security headers (CSP, HSTS in production, X-Frame-Options, nosniff,
Referrer-Policy, Permissions-Policy) are applied to every response.
- Model ids and file paths are resolved and confined to the models directory to
prevent traversal; the API never accepts arbitrary filesystem paths for model
operations.
- Production startup fails loudly on weak configuration (missing JWT secret,
wildcard CORS, non-PostgreSQL database, insecure cookies).
- Data export and deletion endpoints exist for GDPR-style requests; audit logs
record administrative actions.
Report vulnerabilities per [SECURITY.md](SECURITY.md).
## What is not real (yet)
These are the current limits:
- Fine-tuning ships without a trainer. The endpoint validates your request
and runs one only if you provide `ECOGUARD_TRAINER_COMMAND`; otherwise it
returns a clear configuration error. There is no built-in LoRA trainer.
- Retraining triggers only create a trigger and a dataset export. Wiring
either of them to a trainer is your job.
- Semantic caching is off by default because it depends on the quality of
the loaded model's embeddings. A dedicated embedding model makes it useful;
a 0.5B chat model makes it approximate.
- The registry does not verify artifacts cryptographically beyond an
optional SHA-256 checksum computed at registration.
- No multi-GPU tensor parallelism. One model, one process, so scale with
replicas behind a load balancer.
- Concurrency is a queue. A single llama.cpp context generates one sequence
at a time, so extra requests wait; parallel throughput needs replicas or a
remote backend.
## Development
```bash
make install-local # dependencies + llama.cpp
make run # API with reload
make ui # build the production UI
make docs # build the documentation site
make check # ruff + mypy + pytest (what CI runs)
make benchmark MODEL=models/your-model.gguf
```
The test suite is hermetic: 155 tests run in under three seconds with a fake
backend, no network, and a temporary SQLite database. Tests cover auth and
RBAC, inference and streaming, the OpenAI wire format, model management and
GGUF parsing, migrations (upgrade, parity, downgrade), security utilities, and
the MLOps flows. CI additionally builds the frontend, builds and smoke-tests the
Docker image, and builds the documentation site.
## Documentation
Full documentation is generated from `docs/` into a static site:
- [Getting started](docs/getting-started/installation.md): install, first run, model setup
- [Architecture](docs/architecture/overview.md): request lifecycle, backends, database, security
- [API reference](docs/api-reference/inference.md): every endpoint with examples
- [Operations](docs/operations/deployment.md): deployment, observability, backups, troubleshooting
- [ADRs](docs/adr/): why the system is built this way
The same content is published at
<https://swadhinbiswas.github.io/Ecoguard/docs/>, rebuilt by the `Docs`
workflow whenever `frontend/`, `docs/` or `tools/docs/` change.
```bash
make docs-serve # build and serve at http://localhost:8000
```
## License
MIT, see [LICENSE](LICENSE).