ROADMAP.md
Rama main · réplica de lectura del repositorio original.
# marbor Roadmap
marbor is a **single-instance, self-hosted inference control plane**. This roadmap is
the evolution of one thing - the **router's intelligence** - not a march toward distributed
infrastructure. It runs as one static Go binary, SQLite only, in front of multiple GPU
backends (Ollama, vLLM, TGI, llama.cpp, MLX) behind a single OpenAI-compatible endpoint.
## The moat: router intelligence progression
```
Warm-aware router
↓
State-aware router
↓
KV-preserving router
↓
Predictive router
↓
Context-aware router
```
Everything below is in service of moving down that line. The open-source core is
production-hardened and free for any use under Apache-2.0.
---
## Shipped (open-source core, Apache-2.0)
**Routing & scheduling**
- Warm-aware routing - route to the node already holding the model warm in VRAM
- VRAM-aware cold placement - cold requests go to the node with the most free VRAM (v0.8.0)
- Weighted placement scoring - warm / free-VRAM / queue-depth / health / success factors, with model pinning and post-failure node cooldown (v0.11-v0.14)
- Request queue with backpressure - configurable depth + timeout (v0.8.2)
- Persistent warm state - the warm map survives restarts and is reconciled against live `/api/ps` on boot, so routing intelligence is not lost on restart (v0.11-v0.14)
**KV / session preservation**
- Session affinity - `X-Session-ID` sticky routing with TTL and graceful fallback, keeping backend KV cache warm across multi-turn conversations (v0.6.0)
**Prediction**
- Predictive prewarming - an in-memory transition ring buffer plus time-of-day patterns warm the next likely model *before* the request arrives (v0.11-v0.14)
**Multi-runtime**
- `RuntimeProbe` abstraction - Ollama, vLLM, TGI, and llama.cpp behind one OpenAI-compatible endpoint; path-aware routing (`/api/*` → Ollama, `/v1/*` → any) (v0.10.0)
**Reliability & operations**
- Pre-stream failover - dead node → retry alternate → cloud → 502 (v0.3.x)
- Active/active HA - run two independent instances behind any TCP/L4 load balancer; peer `/health` awareness eliminates the single-proxy SPOF (v0.7.0)
- Cost-aware cloud overflow - OpenAI/Anthropic fallback only when local capacity is saturated, with real per-token cost tracking and savings math (v0.2.x-v0.3.x)
- Auth, per-key model allow-lists, rate limits, and daily/monthly quotas persisted across restarts (v0.3.x)
- Observability - embedded admin dashboard, 14 Prometheus metrics, best-effort request audit log, analytics (v0.2.x-v0.9.x)
- Day-2 ops - node drain, runtime key/node mutation, SIGHUP + HTTP config reload, structured JSON logging (v0.9.0)
- `marbor bench` - reproducible cold-vs-warm TTFT measured through the marbor proxy (v0.11-v0.14)
- Router decomposition - `placement` / `health` / `queue` split behind interfaces, so the next stages extend safely without touching the hot path (v0.11-v0.14)
Full, dated release history lives in [CHANGELOG.md](CHANGELOG.md).
---
## Now - Validation on real hardware
The next milestone is proof, not features. The current README benchmark was measured on a
node where the model did not fully fit in VRAM (documented honestly there). Before the
router intelligence deepens further, it gets validated on dedicated hardware:
- Cold-vs-warm TTFT on NVIDIA GPUs where the model fully fits in VRAM (n ≥ 10 per scenario,
direct-to-node control run, `--json` output, hardware documented)
- Warm-aware routing vs round-robin across 2+ nodes - the number no plain load balancer can produce
- Heterogeneous fleet validation: NVIDIA + Apple Silicon backends behind one endpoint
- Published, reproducible results via the [`bench/`](bench/) harness
Community feedback from early deployments shapes the priority of everything below.
---
## Next
### Step 6 - Prefix locality hints
Route requests that share a large stable prefix (system prompt + retrieved documents) to
the node that last served that prefix, so backend KV cache is reused **across users**, not
just within a single session.
- Hash the stable prefix (system prompt / RAG context); ignore the final user message
- Store `prefix_hash → last_node` with a TTL
- Soft-prefer the last node; never hard-lock - reroute if it is unhealthy, cold, or full
**Payoff:** RAG pipelines, shared corporate knowledge bases, contracts/legal - anywhere many
requests share a large prefix. This is the deepest KV-preservation stage of the moat.
---
## Later - under consideration (all within the single-instance model)
Ordered by pull, not promise - items graduate to "Next" when real deployments ask for them:
- **Scheduler depth** - extend placement scoring with queue-time and model-load-cost
awareness (estimated total latency per candidate node, not just static weights)
- **Predictive accuracy hardening** - measure prewarming hit-rate and tune against real
traffic patterns rather than adding new prediction signals
- **Deeper GPU observability** - per-model tokens/sec, cold-start counters, warm-hit
ratios surfaced as first-class dashboard and Prometheus signals
- **Concurrency & Queueing** - Enhancing queue scheduling (e.g., prioritizing low-latency requests, implementing fair-share queues per API key, or handling backpressure during high concurrency)
- **SQLite write-path hardening at 20+ nodes** - the single-gateway design lives or dies
on write throughput; benchmark and tune before it becomes a ceiling
- **Additional runtime adapters** (TensorRT-LLM, SGLang, …) - strictly demand-driven;
added when a real deployment needs one, not speculatively
---
## Explicitly NOT on the roadmap (by design)
marbor is single-instance by design. These are deliberately out of scope - they trade
the product's core simplicity for infrastructure it doesn't need:
- ❌ etcd or any external datastore (SQLite only)
- ❌ distributed / shared state between instances
- ❌ multi-router consensus HA, leader election, gossip protocols, CRDTs - HA is achieved by running independent instances behind an L4 load balancer
- ❌ federated / multi-region marbor
- ❌ direct KV transport or shared KV between nodes
- ❌ tenant isolation and compliance packs (HIPAA/SOC 2) - later-stage, not now
- ❌ prompt-based model selection / policy engine ("small prompts → 7B") - gateway territory
already served well by LiteLLM; marbor routes *placement*, not *model choice*
---
## Design constraints
- **Single marbor instance; SQLite is the only datastore.**
- One static Go binary, zero runtime dependencies - no CGO, no Python, no Docker required.
- Multiple heterogeneous GPU backends behind one OpenAI-compatible endpoint.
- Every number on the dashboard is a real measurement - no fabricated data, ever.
- Built for infra/MLOps operators self-hosting AI on their own hardware.
---
## Commercial
The open-source core above - **including the routing intelligence that is the moat** - stays
free under Apache-2.0. A commercial offering (enterprise governance and a managed control
plane) is planned once the core moat is complete; its details are intentionally not part of
this public roadmap.