docs/MODEL_LAB.md

Rama main · réplica de lectura del repositorio original.

# Model Lab Roadmap

Model Lab is the planned open-model workbench for Codewhale. The north star is
simple: Codewhale should make open-source and open-weight models practical in
terminal coding workflows across every provider that offers them. Model Lab is how
those models become discoverable, evaluable, routable, servable, and exportable
without weakening the current terminal-agent contract: local workspace control,
explicit provider auth, approval gates, and clear privacy boundaries.

This document is roadmap language. Some worksets below are roadmap-only.

## Implemented Today

- DeepSeek is the first-class default provider today, with `deepseek-v4-pro`,
  `deepseek-v4-flash`, streaming thinking blocks, Fin routing, `DEEPSEEK_*`
  environment variables, and `~/.deepseek` config compatibility.
- OpenRouter, Novita, Fireworks, NVIDIA NIM, AtlasCloud, Wanjie Ark, Hugging
  Face Inference Providers, generic OpenAI-compatible endpoints, SGLang, vLLM,
  and Ollama are supported provider paths where their IDs appear in
  `/provider`, `codewhale --provider`, or `codewhale models`.
- Hugging Face Inference Providers are available through the
  OpenAI-compatible router at `https://router.huggingface.co/v1`. Select the
  route with `huggingface`, `hugging-face`, `hugging_face`, or `hf`; configure
  `HUGGINGFACE_API_KEY` or `HF_TOKEN` for auth.
- Model auto-routing chooses a concrete DeepSeek model and thinking level per
  turn. It is not a TUI mode.
- Fin is the fast `deepseek-v4-flash` thinking-off path for routing,
  summaries, cheap checks, RLM child calls, wakeup verification, and
  binary-completion checks.
- Self-hosted OpenAI-compatible endpoints can be used through SGLang, vLLM,
  Ollama, or the generic `openai` provider configuration.

## Still Planned

- A native Hugging Face Hub browser, model passport picker, or direct Hub search
  workflow. The OpenAI-compatible Hugging Face Inference Providers route is
  implemented separately as a chat provider.
- Built-in Hugging Face model card, dataset, adapter, safetensors, Spaces, or
  Jobs workflows.
- Native Unsloth, NeMo, or Arcee integrations.
- A dedicated Model Lab UI tab.
- Built-in eval leaderboards, hosted observability, or training-infrastructure
  orchestration.

Until those land, use the provider paths above, MCP servers, or external
workflows explicitly configured by the user.

## Model Lab Principle

Model Lab should help users answer practical questions:

- Which model should handle this turn?
- Which open or open-weight model can I run locally or through a trusted
  provider?
- Which provider offers this model with the latency, price, context window,
  license, and privacy posture I need?
- What did this model cost, how did it perform, and what data left my machine?
- Can I reproduce, export, or self-host the route?

It should never hide provider boundaries, silently upload local artifacts, or
describe a model as available before Codewhale can actually route to it.

## Hugging Face Workset

Implemented today:

- Hugging Face Inference Providers as an explicit OpenAI-compatible router
  provider, selected with `huggingface`, `hugging-face`, `hugging_face`, or
  `hf`.
- Model IDs are sent to the router exactly as selected, including
  org-prefixed Hugging Face model IDs.

Planned scope:

- Hub API auth and model discovery.
- Model cards, licenses, tags, safetensors metadata, adapters, and dataset
  links surfaced in a terminal-friendly way.
- Native Hub browser and model-passport metadata on top of the already separate
  Hugging Face Inference Providers chat route.
- Hugging Face Jobs as an optional remote execution path for user-approved
  experiments.

Non-goal for now: claiming native Hub search, model passports, Spaces/Jobs, or
Model Lab UI exists before those surfaces are implemented in code.
The inference-provider API key does not imply Hub browsing/export, upload, or
Jobs authorization.

## Unsloth Workset

Planned scope:

- Fine-tuning recipes and adapter workflows for users who already own the data
  and compute path.
- Export guidance that keeps dataset, adapter, and checkpoint locations explicit.
- Compatibility notes for models that can return to local serving or a hosted
  OpenAI-compatible endpoint.

## NeMo Workset

Planned scope:

- Training and alignment workflow notes for users operating NVIDIA-centric
  infrastructure.
- Clear boundaries between NVIDIA NIM inference support that exists today and
  future NeMo training or customization workflows.

## Arcee Workset

Planned scope:

- Small-model routing and specialization experiments.
- Exportable routes that make it clear when a task is handled by a smaller
  model, Fin, or full DeepSeek reasoning.

## Serving Workset

Planned scope:

- Better local and private serving ergonomics for SGLang, vLLM, Ollama, and
  OpenAI-compatible gateways.
- Health checks, model listing, context-window metadata, and route validation.
- No silent network exposure: public endpoints must be configured explicitly.

## Eval Workset

Implemented authoring foundation:

- The best-of-N Workflow starter can generate 2–16 structured, independent
  worktree candidates with cache-stable shared instructions and a read-only
  review. This is generation/review evidence, not runtime-owned hard-gate proof.

Planned scope:

- A frozen experimental-search spec that binds the baseline, requested and
  resolved model names, public evidence, and evaluator identity before
  candidate admission (see [Workflow Experimental Search](WORKFLOW_EXPERIMENTAL_SEARCH.md)).
- Reproducible task suites for coding, review, docs, release checks, and
  long-context workflows.
- Side-by-side route comparisons where the exact model, provider, thinking
  level, prompt, and tool policy are captured.
- Runtime-owned hard gates and command scoring after worker write authority is
  revoked; clean-baseline replay; duplicate-patch detection; multi-round
  Pareto/diversity promotion; aggregate receipts over Fleet receipts; and a
  Workflow-panel leaderboard. No winner is applied or merged automatically.

## Observability Workset

Planned scope:

- Local-first traces for turn routing, tool calls, approvals, cost, cache
  behavior, and context pressure.
- Export rules that redact secrets and require explicit user action before data
  leaves the machine.

## Training Infra Workset

Planned scope:

- Recipes for dataset preparation, adapter training, artifact naming, and
  promotion into serving.
- Separation between local/private artifacts and anything published to a hub or
  registry.

## Privacy And Export Rules

- Local files, prompts, transcripts, traces, model outputs, eval results,
  adapters, datasets, and checkpoints should remain local unless the user
  explicitly chooses a provider or export destination.
- Provider auth must remain explicit. `DEEPSEEK_*`, OpenRouter,
  `HUGGINGFACE_API_KEY` / `HF_TOKEN`, and self-hosted credentials should not be
  inferred from unrelated config.
- Exportable artifacts should include provenance: source model, provider,
  route, tool policy, eval inputs, and redaction status.
- Public sharing, hosted telemetry, sponsorship badges, and external branding
  require maintainer approval.