internal/autogen/CLAUDE.md
Rama main · réplica de lectura del repositorio original.
# internal/autogen
## Purpose
`autogen` generates a complete quartermaster config YAML by discovering local GGUF models,
reading their headers, and computing a per-model load plan (`-ngl` / `--n-cpu-moe` / context
window / KV quant) from a VRAM budget. It is a fork-specific Go port of the `domina-llm-eval`
PowerShell planner (`Read-GgufMetadata` / `Get-LlamaLoadPlan` / `Get-DenseCtx` /
`Get-KvCostModel`) plus the `Generate-Config.ps1` orchestration, letting the harness stop
pre-generating config variants by hand. Kept deliberately separable for clean upstreaming.
## Which doc
| Doc | Read it when you're touching |
|---|---|
| this file | discovery, gguf parsing, the control files (generate + sidecar), the profile loop, regen gating |
| [`sizing.md`](sizing.md) | VRAM math — compute buffer, `-ub`/`-b`, KV quant, checkpoints, drafters, RoPE, `estVramGB` |
| [`liveoffload.md`](liveoffload.md) | the spawn-time placement guard, `minGpuFraction`, live-VRAM budget capping |
| [`backends.md`](backends.md) | which exe serves a model, the registry, the vllm emitter |
| [`classes.md`](classes.md) | non-LLM emitters — SAM, image/diffusion, embedding, TTS (2 engines), ASR |
## Key files
| File | Role |
|---|---|
| `gguf.go` | GGUF header parser (`ReadGgufMetadata`): metadata KV section + tensor section → `Metadata`, plus `scanChatTemplate`. Package doc lives here. |
| `discover.go` | Walks the models root(s) for `.gguf`, derives model IDs/quants/publishers, collapses split shards, skips mmproj projectors and diffusion encoders/VAEs. `DiscoverGgufModels` (single root) + `DiscoverGgufModelsMulti` (main root + per-UI-category `categoryRoots`). |
| `family.go` | Sidecar inheritance across a model's family: `ModelBaseKey`/`FamilyKey` (Go twins of `baseKey`/`familyOf` in `ui-svelte/src/lib/modelTable.ts`), the header compatibility gate, `inheritSidecars` (fills blank `DraftPath`/`MmprojPath` from a compatible sibling) and `DraftSidecarFor` (the roots-aware `DraftSidecarForDir`). |
| `identity.go` | `IdentityOf`/`identityFrom` — the gguf's OWN identity (header `general.basename`/`size_label`/`finetune`) folded into the Models table's two grouping keys, with junk (commit hashes, "source") filtered out. Empty keys = this file identifies nothing; the caller falls back to `ModelBaseKey`/`FamilyKey`. |
| `quantlabel.go` | `quantLabelFrom` — how a file is ACTUALLY quantized, from the byte-weighted tensor-type histogram (`"Q4_K"`, `"IQ4_XS mix"`), plus `ggmlTypeName` and the `tensorScan` struct the tensor walk returns. |
| `metacache.go` | In-memory metadata cache keyed by file size+mtime (`ReadGgufMetadataCached`). |
| `kvcost.go` | KV-cache cost model (`GetKvCostModel`) + context-budget math (`MaxCtxForBudget`, `KvReserveGB`, `RoundedCtx`, `GetDenseCtx`, `defaultKvQuant`). → `sizing.md` |
| `plan.go` | VRAM budget → placement: `-ngl`/`--n-cpu-moe` for dense and MoE (`GetLoadPlan`, `densePlacement`); MoE expert-share table + `effectiveShare`. → `sizing.md` |
| `generate.go` | Top-level orchestration (`Generate`): builds per-model profiles (solo, ctx tiers, named variants), sizes each, emits the YAML. Before the emit loop it builds the name-detected prompt-enhancer table (`s.autoEnhancers = detectEnhancers(rows, s, gf.Overrides)`). `emitModel`/`RenderSoloCmd` **dispatch by model class** (SAM → image → decision → embedding → TTS → ASR → LLM). This file is the profile loop; the three phases live in the siblings below. |
| `generate_sizing.go` | Phase 1 — sizing math: `sizeProfile`, `--ctx-checkpoints` count + `checkpointReserveGB`, `forceLowActiveMoE`/`applyForcedOffload`/`estForOffload`, `computeBufferGB`, `MmprojVramGB`, `draftOverheadGB`. Pure arithmetic over `Metadata` + `Settings`. → `sizing.md` |
| `generate_cmd.go` | Phase 2 — command rendering: `buildCmdLines` (the per-class argv builder) and `RenderSoloCmd` (same with a `${PORT}` placeholder, for the UI preview + ad-hoc commands), plus `effectiveUb`, `effectiveSpec`/`specHas`, `cmdPath`, `needsQwenFixedChatTemplate`, `defaultSamplerFor`/`samplerLines`. |
| `generate_emit.go` | Phase 3: YAML emission: non-model sections (`emitSlotCache`, `emitAPIKeys`, `emitPromptEnhancers`, `emitGroupsAndListeners`, `writeGroup`) and per-model bits (`emitProfile`, `writeDisplayName`, `writeEstVram`/`writeEstRam`, `writeEnhancerBlock`/`writeEnhancerPrompt`, `effortLevels`, `formatCtxTag`, `slugify`). |
| `llmlora.go` | llama-server LoRA adapters: `resolveLlmLoraDir` (the `loraDirs["llm"]` -> `loraDir` -> model's-own-dir ladder), `resolveLoraPath` (bare name joined onto it; `isAbsLoraPath` is hand-rolled, NOT `filepath.IsAbs`, so a config written on Windows still reads absolute on Linux) and `llmLoraLines` (`--lora` / `--lora-scaled`). **Deliberately not the sd-server shape**: sd-server takes a DIRECTORY and each request picks from it, llama.cpp binds named adapter FILES at spawn, so the folder here emits nothing on its own and only Override.Loras reaches the backend. `--lora` is the flag table's only `Additive` entry, which is what stops a user's own `--lora` in custom args from silently replacing the model's list. |
| `estimate.go` | One-shot preview (`EstimatePlan`) of a candidate tuning for the web editor; reuses the solo-profile sizing path without writing config. |
| `flagtable.go`, `flagtable_sd.go`, `customargs.go`, `pins.go`, `validate.go`, `backendhelp.go` | Custom launch arguments: a per-backend flag table (`LlamaFlags`, `SdFlags`) drives knob suppression (`ComposeCmd`), the editor's ownership locks, and validation; `PinsFromArgs` folds a pinned `-c`/`-ctk`/`-ub`/`-ngl`/... into the sizer so emitted flags and estimates describe the launch that runs; `BackendFlags` probes a backend's `--help` for the union. → [`../ui-svelte/launch-args.md`](../ui-svelte/launch-args.md) |
| `overrides.go` | Control-file types (`GenerateFile`, `Settings`, `Override`, `VariantSpec`, `GroupSpec`), defaults, loading/merging, `globLike` (PowerShell `-like`). `Settings.RootList()`/`CategoryRoots`/`CategoryOrder`/`IsCategory` = multi-root scan folders, one optional extra root per UI tab (**`CategoryOrder` must mirror `MODEL_CATEGORIES` in `ui-svelte/src/lib/modelUtils.ts`** - the picker renders on every tab, so a missing category is a root that is set, shown, and never scanned; a test enforces it and `/api/settings/root/pick` 400s on an unknown one). `Settings.LoraDirs` = the per-category LoRA folder; `"image"`/`"video"` are sd-server's `--lora-model-dir`, `"llm"` is only a base path (see `llmlora.go`). `Override` carries granular DRY, granular spec, the image fields (including the per-model `PromptEnhancerPrompt`/`PromptEnhancerEditPrompt` twins), and `Loras` (`LoraRef`, whose `Scale` is a POINTER because scale 0 means loaded-but-inert, not unset). |
| `appsettings.go` | **Process-level** settings (`AppSettings`): listen + playground addresses, the dashboard access policy, models-folder watching, update polling, HF token. Not part of `Settings` and never emitted into the config — `main()` reads them before the config exists, via `LoadAppSettings` (generate file's `settings.app`, then the sidecar's `app:` block). `UpsertSidecarApp`/`LoadSidecarApp` are the dashboard's side. |
| `sidecar.go` | UI-owned overrides file (`quartermaster-overrides.yaml`): per-model overrides, the global settings patch, managed API keys, and the `BackendEntry` registry. |
| `hash.go` | Inputs hashing + hash-gated regen (`InputsHash`, `EnsureConfig`, `CurrentInputsHash`). |
| `budgets.go` | Hardware-derived STARTING budgets: `RecommendedVramGB` (free VRAM) / `RecommendedRamGB` (available RAM), and `seedHardwareBudgets`, which `LoadGenerateFile` uses to replace `applyDefaults`' 7 GB/24 GB placeholders when neither the file nor the sidecar pinned them. One cached probe per process. |
| `vram.go` | Live free-VRAM sampling via `internal/perf` (`SampleFreeVramGB`, `resolveAutoVram`) for the `autoVram` setting. POOLED across every eligible adapter since issue #4, via `gpuset.go`. Budgets against the **idle high-water mark** (`noteFreeVramGB`), never the raw sample — autoVram re-resolves on every `EnsureConfig` *and* every estimate preview, both of which run while models are loaded. |
| `gpuset.go` | Multi-GPU device set (`GpuSet`, `GpuDevice`): eligibility (`gpuSetFromStats`, `EligibleGpuStats`, `LiveGpuSet`), the device policy (`GpuPolicy`, `SharedMemoryAuto/On/Off`, `detectedIntegrated`, `effectiveGpuStat`, `DescribeGpuStats`), pooled `FreeGB`/`TotalGB`, `MainIndex`, the `--tensor-split` ratio (`TensorSplit`, `FormatSplit`), the extra-device overhead the sizer must charge (`ExtraDeviceOverheadGB`), and `ResolveGpuSet` (60s-cached resolve into `Settings.Gpus`). The header comment carries the split math. |
| `backenddev.go` | What the BACKEND calls each device (`ListBackendDevices` over `--list-devices`, memoised per binary), and the map from a resolved `GpuSet` onto those ids (`BackendIDs`, `DeviceFlagFor`). Refuses on any unmatched device so the caller degrades to unnamed placement. |
| `liveoffload.go` | Spawn-time placement recompute (`LiveOffloadArgs`), including the live `--tensor-split` retune (`retuneTensorSplit`). → `liveoffload.md` |
| `vllm.go` | Backend selection (`resolveBackend`, `resolveBackendPreferring`, `kindClass`) + the vllm emitter. → `backends.md` |
| `hf.go` | Hugging Face model folders (`config.json` + safetensors): the three-rule detector (`hfModelDir`), the `IsHF` row, `ReadHFMetadata` (config.json → `Metadata` for vllm sizing), `emitHFModel` (vllm-preferred, `# SKIPPED` without one), and `ReadModelMetadata`/`HFRowFor` for server callers holding a path. → `backends.md` |
| `rope.go` | `ropeCeiling`/`ropeFactor` — the only path that lifts the trained-ctx ceiling. → `sizing.md` |
| `encoderpool.go` | Diffusion component auto-discovery: classifies every VAE / CLIP / T5 / audio VAE / text-encoder LLM on disk from its header (safetensors tensor table or gguf metadata), pairs each encoder with the mmproj beside it, and fills the blanks in `settings.encoders`. Matched to a DiT by `Metadata.CondHidden`. VAEs and audio VAEs carry a FAMILY, and `Vae`/`AudioVae` are scoped by it: the shapes alone would let a model load a decoder for a latent it never produced. `LlmHinted` is the path-based escape for a family whose encoder is a republished copy of a common model (LTX). Prompt-enhancer files (`IsPromptEnhancerFile`) are withheld from the text-encoder pool: a same-arch BF16 PE outranks the real encoder on file size in `better()`, and a mis-picked encoder fails as a confidently unrelated image. Recurrent LLMs (`isRecurrentLlm`: `full_attention_interval` or `ssm.*` sizes set) are withheld too: sd.cpp's conditioner cannot load a GatedDeltaNet/SSM model, and a same-width hybrid (MiMo-V2.6-Distill-Qwen-9B vs Qwen3-VL-8B) outranked the real encoder on size and killed sd-server at "model metadata validation failed". -> `classes.md` |
| `companions.go`, `companions.json` | What a split DiT needs beside itself and where to get it, for the model browser (`PlanImageCompanions`). WHICH roles are missing is `resolveComponents` run against the live pool, exactly as a regen runs it; WHERE each comes from is the embedded, hand-curated `companions.json` (family -> role -> `{repo, path, label}`), because nothing machine-readable says: a DiT states an encoder WIDTH, not a repo. First family whose `arch` (prefix) / `name` (substring) / `condHidden` all match wins, so specific entries go first. Every source is checked ungated before it goes in. A role with no recipe comes back in `Unsourced`. -> `internal/server/hubapi.md` |
| `promptenhancer.go` | Name-based prompt-enhancer classifier: `promptEnhancerName`/`PromptEnhancerID` (direction `PEDirText`/`PEDirEdit`, family key), `IsPromptEnhancerFile`, `detectEnhancers`/`autoEnhancers.For` (pairs a rewriter with the image model its name carries), `resolveEnhancerIDs` (+ `enhancerBaseID`, which maps a legacy `<enhancer>-vision` pin to the base id), `enhancerCtx`, and the `PEDisabled` (`"none"`) sentinel. NAME-only because a PE gguf reports `general.architecture=qwen3vl`, identical to every other VL chat model, so the arch gate the asr/tts/embedding classifiers use says nothing here. Narrow rule: a `pe` token must be immediately followed by a `t2i`/`i2i` token, or the name spells `promptenhanc`. |
| `audiocpp.go` | audio.cpp emitter: the `IsAudioCppModel` gate (header-exact, no filename heuristics), the 96-family task table, the containment flags that stop audio.cpp's own model manager being a second scheduler, and the `audiocpp:` block the spawn-time `--config` materializer consumes. Also `withoutAudioCpp`/`onlyAudioCpp`, the mirrored filters that keep each engine's rows out of the other's resolver, and the exported `IsAudioCppKind`/`AudioCppFamilySupport` the model browser's catalog annotates itself with (`internal/server/audiocppcatalog.go`) - one table, so the catalog cannot advertise a family the emitter refuses. → `classes.md` |
| `audiocppdev.go` | Which ADAPTER an audio.cpp model loads onto: the `--list-devices` parser for upstream's own `Vulkan:0 "name" [GPU]` shape (neither `backenddev.go` shape fits), the memoized probe sharing that file's budget, and `pickAudioCppDevice`, which takes the first `[GPU]` row of the emitted `--backend`. Refuses (emits no flag) for cpu/unknown flavours, a single-device backend, or a listing with no discrete GPU. `Override.AudioDevice` overrides it; negative means emit nothing. → `classes.md` |
| `audiocppaligner.go` | Qwen3-ASR's word-timestamp companion: `detectQwen3Aligner` finds a `qwen3_forced_aligner` gguf across every scanned root (`Settings.qwen3Aligner`, derived and unexported like `autoEnhancers`), `qwen3AsrSessionOptions` pins it and the silero VAD as ABSOLUTE session options and adds the aligner to `estVramGB`, `audioCppAssetPath` walks up from the binary for `assets/`. -> `classes.md` |
| `extramodels.go` | `settings.extraModels`: hand-written config.yaml model blocks (name -> block) for engines with no backend kind, emitted as written after the extra image models. Each is strict-decoded into `config.ModelConfig` (`KnownFields`) and skipped with a `# SKIPPED` reason on failure, so a typo cannot fail the config load; `cmd` plus `estVramGB`/`estRamGB` are required. Emitted names join `emitted`, which is what puts them in a swap group. **Generate-file only, deliberately not in the sidecar**: a UI-writable block would be an arbitrary command line reachable from the dashboard. |
| `audio.go`, `asr.go`, `sam.go`, `image.go`, `decision.go`, `embedding.go` | Non-LLM class emitters. → `classes.md` |
## Important types & functions
- `Metadata` (`gguf.go`) — parsed subset of a GGUF header (arch, block count, expert count,
attention dims, RoPE/SWA/SSM fields, `IsMoE`, `IsMTP`, `ExpertWeightShare`, `PoolingType`,
`ChatTemplatePreservesThinking`/`ChatTemplateEffortLevels`). Optional fields are 0 when
absent; consumers guard on `> 0`. `PoolingType` is the authoritative embedder signal, `DecisionType` the decision-model one. `NonCausal` (`attention.causal=false`) or `ClassifierHead` (`classifier.output_labels`) with neither of those is a classifier llama-server cannot serve: `emitModel` drops it as `# SKIPPED` before the LLM path (von-1.2, a ModernBERT converted without `decision.type`). DFlash drafters are also non-causal but never get here: discovery pairs them by filename.
`Metadata` also carries what the file says about ITSELF: `QuantLabel` (tensor-derived, see
`quantlabel.go`) and the raw identity KVs `BaseName`/`SizeLabel`/`FineTune`/`GeneralName`.
Those four are unjudged - frequently absent, occasionally a commit hash - so read them through
`IdentityOf`, never directly.
- `ReadGgufMetadata` — parses the metadata KV section, then `readTensorScan` walks the tensor
section once for everything the weights themselves answer: the expert-weight share, the vocab
element count, the diffusion markers, and the type histogram `QuantLabel` comes from.
- `ReadGgufMetadataFrom(rs io.ReadSeeker, path string, sizeBytes int64)` — the same parser over
an **already-open source**, which is what lets the model browser size a repo file it has not
downloaded (`internal/server/hubapi.md` hands it a `bytes.Reader` over a Range-fetched
prefix). `ReadGgufMetadata` is a thin `os.Open`+`os.Stat` wrapper around it.
**`sizeBytes` is the file's FULL length, not the length of what the reader holds** — it
becomes `FileSizeGB`, the weights figure every sizing path charges, so passing the prefix
length would report a model that fits any budget. A truncated source surfaces as
`io.ErrUnexpectedEOF`, the caller's signal to re-fetch a longer prefix (a header carries no
length of its own).
- `ReadGgufMetadataCached` (`metacache.go`) — size+mtime-keyed wrapper; what `Generate`/
`emitModel` and the API actually call.
- `GgufRow` / `DiscoverGgufModels` (`discover.go`) — one served-model row per gguf (shard-1
only), `ID = baseID-quant`; `DraftPath`/`DraftKind`/`DraftSizeGB` auto-pair a same-dir MTP
sidecar or DFlash drafter, `MmprojPath` a same-dir clip projector. When a dir ships neither,
`inheritSidecars` (`family.go`) borrows one from a compatible family member — see the gotcha
below.
- Projector placement (`generate.go` profile loop) — if a model resolves to a projector, EVERY
profile of it loads one (`prof.Vision`), not just the `-vision` twin. llama-server reads
`--mmproj` at spawn and nowhere else, so a served id launched without it answers every image
with `image input is not supported` for the life of the process: a text id that silently
rejects images is a trap, and the router never inspects the body to route around it. What
differs is placement. Default profile, ctx tiers and named variants get the projector in RAM
(`CpuMmproj` → `--no-mmproj-offload`): zero VRAM, so the window and the layer placement are
exactly what they would have been without it, at the cost of a one-off host-side encode on
the requests that actually carry an image. The `-vision` twin is the id that pays VRAM for it
(`MmprojGB` charged to `Overhead`), which is what it exists for. `LiveOffloadArgs` and the
editor preview (`configapi_estimate.go`) both skip the `MmprojGB` charge when the argv
carries the flag, so all three price the same launch. Two pins, two audiences:
`Override.Mmproj` (the "Image projector" dropdown on the model config modal's Default tab,
shown only when a projector resolves at all) places it on the non-twin profiles — `gpu` buys
VRAM residency there too, `none` opts the whole model out of vision including the twin.
`VariantSpec.Mmproj` on the reserved `vision` variant places the TWIN's, and is the only
thing that does; blank keeps the GPU default, `none` there drops the twin alone and leaves
the text ids taking images. It is deliberately absent from every other variant tab, since
those inherit the model-wide pin. Careful: an image-class model routes through
`emitImageModel` before the twin gate, so a variant literally named `vision` is an ordinary
image variant for those — assert on `--mmproj`, not on the `-vision` id.
- `GetLoadPlan` (`plan.go`) — `-ngl`/`--n-cpu-moe` from a VRAM budget; MoE path uses a 0.5
PCIe-thrash crossover, falling back to naive `-ngl` (`densePlacement`) past it.
- `Generate` (`generate.go`) — discover → per-model `emitModel` → `emitGroupsAndListeners`.
- `EstimatePlan` (`estimate.go`) — preview load plan for the editor, mirroring the solo profile.
- `LoadGenerateFile` (`overrides.go`) — applies defaults and merges the sidecar.
- `EnsureConfig` (`hash.go`) — hash-gated generate; `CurrentInputsHash` lets callers detect a
would-be-different config cheaply.
- Sidecar API (`sidecar.go`) — `LoadSidecarOverrides`, `UpsertSidecarOverride`,
`DeleteSidecarOverride`, `UpsertSidecarSettings`, `ClearSidecarSettings`,
`LoadSidecarCategoryRoots`/`UpsertSidecarRoot`, `LoadSidecarBackends`/`UpsertSidecarBackends`
(backend exe paths — top-level so a VRAM reset can't wipe them; overlaid onto `Settings`
BEFORE `applyDefaults` so a blank sd/tts derives as a sibling of a UI-set llama exe),
`LoadSidecarPromptEnhancers`/`UpsertSidecarPromptEnhancers` (**the rewrite-model table**, also
top level for the same reason as backends. Whole-list replace, last-wins dedupe by lowercased
model id but keeping the FIRST position, so editing a row does not make it jump to the bottom
of the settings table. `SystemPrompt` is deliberately NOT trimmed: the leading indentation and
trailing newline of a published PE prompt are part of the prompt),
`LoadSidecarExtraImageModels`/`UpsertSidecarExtraImageModels` (**the "Add model manually"
table**. Unlike every other top-level list this one EXTENDS the generate file's
`settings.extraImageModels` instead of replacing it (`mergeExtraImageModels`): a UI row
replaces the file row of the same name in place, the rest append. Replace semantics would
either hide a power user's hand-written rows from generation or copy them into the sidecar on
the first save. Deleting a FILE row from the table records its name in
`removedExtraImageModels` (`LoadSidecarRemovedExtraImageModels`), which the merge drops at load;
a kept row of the same name clears the entry),
`LoadSidecarAPIKeys`/`UpsertSidecarAPIKey`/`DeleteSidecarAPIKey`,
`LoadSidecarBackendSources`/`UpsertSidecarBackendSources` (**tracked GitHub repos the in-app
installer downloads builds from** — deliberately separate from `BackendList`: a
`BackendSource` is a *place to get builds*, a `BackendEntry` is an *executable path the
launcher uses*; installing from a source writes the entry. Rows with no id/repo or no variant
carrying a derived pattern are dropped on write).
## Data flow
1. **Load** — `LoadGenerateFile` reads the hand-authored generate control file, applies
PowerShell-default settings, then overlays the UI-owned sidecar: first the `SettingsPatch`
(dashboard VRAM/headroom edits), then the sidecar overrides *ahead* of the file's overrides
(so UI edits win under first-match resolution). `--models-dir` overrides `settings.modelsRoot`.
2. **Gate** — `EnsureConfig` hashes the resolved models root(s), raw generate bytes and sidecar
bytes (`InputsHash`/`InputsHashRoots` when extra category roots exist). If the stored
`.modelhash` matches and the output exists, regeneration is skipped. `autoVram` always forces
a regen (the live VRAM snapshot isn't visible to the hash) and re-samples via `resolveAutoVram`.
The whole walk-and-write is serialized under one package-level mutex shared with
`CurrentInputsHash`: the hub download-completion regen, the watch-models poll and a UI save
can all land in the same instant, and two of them interleaving their scans and `config.yaml`
writes tears the file.
3. **Discover** — `DiscoverGgufModels`/`DiscoverGgufModelsMulti` walk the roots; `Generate`
sorts rows and resolves each row's `Override` by path glob (+ optional quant).
4. **Size** — `emitModel` dispatches by class, reads metadata (cached), resolves KV quant,
derives the KV cost model, then builds a profile set: solo + ctx-tier variants + named custom
variants (`Override.Variants` + `settings.defaultVariants`). Each profile runs
`sizeProfile` → `GetLoadPlan` → `forceLowActiveMoE` (or `applyForcedOffload` when `cpuOffload`
is pinned). Non-LLM classes use their own paths — see `classes.md`.
5. **Emit** — `emitProfile` writes each model's YAML `cmd` block. Per-model engine knobs on
`Override` override the defaults here: `FlashAttn` (`-fa`), `Mmap`, `Mlock`, `Threads` (`-t`),
`Parallel`, `Ub` (`-ub/-b`), granular DRY, the sampler defaults (`--temp`/`--top-k`/`--top-p`/
`--min-p`/`--presence-penalty`, layered over the arch baseline) and granular speculative
sub-knobs. Zero/empty = the generator default (but see the sampler note below: those are
pointers, and 0 is a value). `Override.CustomArgs` is composed in
(`ComposeCmd`), which suppresses every generated flag whose knob the text owns; `ExtraArgs` is
the legacy bucket and is still appended verbatim, but only by the emitters that have no flag
table yet (audio, SAM, vllm). llama-server and sd-server both compose.
`buildCmdLines` is shared by `emitProfile` and `RenderSoloCmd` (the editor's launch-parameters
preview, `PUT /api/models/{model}/preview`, no persistence). `emitGroupsAndListeners` assigns
models to groups by name-glob (first match wins) and binds listen addresses; every glob-matched
group is an exclusive swap group. The synthesised **coexistence** groups are the exception:
SAM models, CPU-only TTS.cpp voices and CPU-only Parakeet ASR models are pulled out of the glob
assignment (`coexistSets` → `sam` / `tts` / `asr`, one group per non-empty class) and emitted by
`writeCoexistGroup` as `exclusive:false`, `persistent:true`, `swap:false`, so they neither evict
nor are evicted, and are appended to every listener since they bind no port of their own.
The fourth coexist set, **`cpu`**, is different: `Override.Device: cpu` (`cpudevice.go`) emits
`-ngl 0 --device none` on every profile (LLM, decision, embedding), prices the whole footprint
(compute buffer included) as `estRamGB` with no `estVramGB`, and `Override.SwapGroup` (blank
follows the device) puts the model's profiles in `coexist.CPU`. That group is emitted
`swap:true`, `persistent:false`, `pool: ram`, and `emitRamBudget` writes `ramBudgetGB`
(= `MaxRamGB`, multiResident only), so the router packs as many CPU models as RAM holds (see
`internal/router/CLAUDE.md`, "Two pools"). `SwapGroupErr` refuses a GPU-device model in the cpu
group (it would hold VRAM the RAM pool never accounts for). `LiveOffloadArgs` leaves a
`--device none` argv alone, and the estimate preview takes `device=cpu` (or reads
`--device none` off a seeded argv) via `EstimateInput.CpuDevice`. Context is still sized by the
VRAM-based ladder for CPU models.
### Prompt enhancers
An image **or video** model can name a **prompt enhancer**: a chat model that rewrites its prompt
into a more precise one before rendering. Five pieces, deliberately split:
- `Settings.PromptEnhancers []PromptEnhancer` (`overrides.go`): the table, declared ONCE. Keyed
by catalog model id, carrying the fixed system prompt, a display name and a `vision` flag.
Top-level and not per-model because the payload is a multi-KB system prompt that several image
models share verbatim.
- `Settings.autoEnhancers` (unexported, `promptenhancer.go`): the pairing table `Generate` builds
from the rows it just discovered (`detectEnhancers(rows, s, overrides)`), keyed by the image model's base key
and holding one id per direction (`autoEnhancers.For`). Unexported because it is DERIVED from
what is on disk this run: it must never round-trip through a settings file, so a `Settings`
built anywhere else (the UI, a test) has none and auto-pairing is off rather than stale.
`resolveEnhancerIDs` reads it at emit time: a configured id wins (including `PEDisabled`), an
empty one takes the detected id for its direction. Detection is NAME-only because a PE gguf
reports `general.architecture=qwen3vl`, identical to every other VL chat model, so the arch
gate the asr/tts/embedding classifiers rely on says nothing here. The rule is deliberately
narrow: a `pe` token must be immediately followed by a `t2i`/`i2i` token, or the name spells
`promptenhanc`, so "Pegasus-7B" is not reclassified out of the encoder pool. The family key is
everything before the marker folded through `ModelBaseKey`, so every quant of an image model,
and a rewriter republished at another quant, still pair. `IsSam`/`IsTrellis` rows are skipped:
the rewriter is a chat model.
- **A prompt enhancer is emitted as ONE profile** (`emitModel`, gated on `promptEnhancerName(row.ID)`),
and it is a latency fix, not tidiness. The default vision placement (projector in system RAM,
`--no-mmproj-offload`, fast twin on the side) is right for a chat model, where most requests are
text, and INVERTS for an i2i rewriter, which is handed an image on every call: the base id paid
a CPU encode each time (measured at five minutes on one 2048px reference) and a user who picked
the obvious id got the slow one. So an enhancer gets no fleet `DefaultVariants` and no `-vision`
twin; its base profile carries the projector in VRAM when it is `i2i` (or undirected) and none
at all when it is `t2i` (never shown an image); and its window is `enhancerCtx` (32k) instead of
the sizer's budget max (262k, ~16GB, which evicted the image model on every press). An explicit
per-model `Ctx`, `Mmproj` pin, `CtxVariants` or `Variants` still wins. Pairing is therefore
always the base id, and `enhancerBaseID` rewrites a pin to the retired `-vision` twin.
- `Override.PromptEnhancer` / `Override.PromptEnhancerEdit`: which entry an image model opts
into, **one per direction**. A **model id, never a path**: an enhancer is a model the ONE router
schedules and evicts like any other, and a path would be a second loader outside the scheduler,
which the architectural invariant forbids. Image variants inherit both for free through
`o := base` in `mergeImageVariant`. Qwen ships the pair (PE-T2I, PE-I2I) and they are NOT
interchangeable: the edit one rewrites an instruction about a picture that already exists, the
text one composes a scene from nothing. Empty means "let detection fill this direction", and a
missing edit half still falls back to the text one on the client. `PEDisabled` (`"none"`) is
how a user refuses: auto-pairing gave the empty string a second meaning, so clearing a field
with a detected candidate would be undone by the next regen. `resolveEnhancerIDs` preserves the
sentinel and `writeEnhancerKey` drops it, which is what makes the refusal stick.
- `Override.PromptEnhancerPrompt` / `PromptEnhancerEditPrompt` (the `ExtraImageModel` struct has
the same twins): the image model's OWN system prompt per direction, emitted by
`writeEnhancerPrompt` with the same `yaml.Marshal` + `indentYAML` treatment as the table,
because the prompt is written for the target model. One rewriter serves checkpoints that want
very different output (SDXL tag soup, a Flux paragraph, a Qwen instruction), so the prompt
belongs to the image model and outranks the settings row's at serve time. Skipped when the
model has no enhancer in that direction. `internal/config.ModelConfig` carries both too.
- `emitPromptEnhancers` (`generate_emit.go`) writes the block with **`yaml.Marshal`, not
`Fprintf`**. A PE system prompt is a multi-line document with quotes, colons and backslashes in
it; a hand-rolled block scalar corrupts the real thing. `indentYAML` re-indents the marshalled
document and uses `TrimSuffix`, not `TrimRight`: every newline before the final document
terminator is content, and trimming the run silently eats the blank line a `|+` keep-indicator
block exists to preserve (`TestEmitPromptEnhancers_RoundTrips` covers exactly this).
`writeEnhancerBlock` is the entry the emitters call (`emitImageModel`, `emitExtraImageModels`):
it resolves the pair, then writes both ids plus both per-model prompts. `writeEnhancerKey` no
longer requires a `settings.promptEnhancers` row. An id with no row, whether typed by hand or
detected from a filename, is emitted as typed (a row only fixes up the id's canonical casing),
because dropping it hid the button with nothing to explain why. So a dangling id is not an error
any more: it runs with the model's own prompt, else with no system prompt at all. The rewrite
itself never happens server-side; see `internal/server/CLAUDE.md` and the playground's
`lib/promptEnhance.ts`.
The encoder pool is the other half of the same classifier. `IsPromptEnhancerFile` drops a PE
gguf from `ScanEncoderPool` outright and bars a safetensors copy from the LLM role, because
`better()` breaks a same-arch tie on file size and a BF16 PE outranks the real Q8 encoder: the image model would condition on its
own prompt rewriter and render a confidently unrelated picture. Emit output changed, so
`genVersion` is v85.
## Gotchas / conventions
- **PowerShell parity is the spec.** Most functions are direct ports and say so in their doc
comments. `real_models_test.go` asserts parity against real ggufs
(`TestReadGgufMetadata_VsPowerShell`, `TestGetLoadPlan_VsPowerShell`,
`TestGetKvCostModel_VsPowerShell`). Documented divergences are called out at the call site.
- **Emit-logic changes need a `genVersion` bump.** The regen hash covers inputs (gguf files +
generate file + sidecar) but NOT the generator code, so a change to `buildCmdLines`/
`emitProfile` output alone leaves the hash stable → `EnsureConfig` skips regen → the stale
`config.yaml` survives even after a rebuild+restart. `genVersion` (folded into
`buildHashInput`) forces a one-time regen. **Bump it whenever the emitted YAML changes for
identical inputs.**
- **The VRAM budget is POOLED across GPUs, and that is only safe because of the split.** Since
issue #4 `freeVramGBFromStats`/`totalVramGBFromStats` sum every eligible adapter instead of
picking the largest one. Three things have to stay true together, or a two-card box OOMs on the
small card while the sizer reports a comfortable fit:
1. every device past the main one is charged `ExtraDeviceOverheadGB` into `prof.Overhead`;
2. `prof.TensorSplit` is derived from the FINISHED `Overhead` (`generate.go`, the compute-buffer
loop), because llama.cpp keeps the fixed costs (the output weight matrix, CUDA context) on
ONE device while it splits layers and their KV by the ratio. **That device is the LAST one
in the device list, and `--main-gpu` has nothing to do with it.** Measured on a Vulkan build
(`llama-server -lv 10 -sm layer -ts 0.5,0.5`, two adapters): reversing `--device` moved a
~146 MiB surplus in `load_tensors: ... model buffer size` from one card to the other, while
`--main-gpu 0` and `--main-gpu 1` at a fixed list produced byte-identical model, KV and
compute buffer lines. So the ORDER is the instruction. `GpuSet.MainLastOrder` puts the
sizer's main device (`PlanMainIndex`, the card with the most capacity, the one charged
`mainFixedGB`) at the end of the `--device` list, and `PermuteSplit` moves `--tensor-split`
with it so both flags still address the same cards. Two incidentals from the same log:
`token_embd.weight` falls to `CPU_Mapped` when it cannot use the host buffer type, and the
logits output buffer sits on `Vulkan_Host`, not on a device;
3. `-sm layer --main-gpu N --tensor-split a,b` is actually emitted, alongside
`env: CUDA_DEVICE_ORDER=PCI_BUS_ID`. Without that env the CUDA runtime's `FASTEST_FIRST`
default can reverse the pair and apply the ratio backwards, silently. None of this is
CUDA-only: the split flags are vendor-neutral llama.cpp arguments and are field-proven on
Vulkan (issue #4). Only the ORDER PIN is CUDA-specific, which is what `--device` replaces.
**`mainFixedGB` must not include `ExtraDeviceOverheadGB`.** `splitBy` already charges
`perDeviceFixedGB` to every non-main device; passing the pooled figure in as the main device's
fixed cost billed the secondary card's runtime twice, both times against the main card's side
of the ratio, and tipped layers onto the card least able to hold them. `generate.go` takes the
main figure before adding the extras; `EstimatePlan` reports it as `FixedGB` and adds the
extras to the pool afterwards, so the config and the spawn-time retune derive the same split.
Adapters below `minGpuVramGB` (3 GB) are dropped, measured against the device's TOTAL **after**
the shared pool is folded in (see the device-policy bullet below); `multiGpu: false` collapses
everything back to the single device `MainIndex` picks.
- **Which devices count is a POLICY (`GpuPolicy`), and it must be passed, never guessed.** The
same telemetry means opposite things on an APU and on a card: a 780M reports a 2 GB carve-out
plus a GTT several times larger, and that GTT *is* its GPU memory — while a discrete card
reports a host aperture that is slower than its own VRAM. There is no flag that tells them
apart, so `Settings.DevicePolicy()` (`SharedMemory`, `PoolIntegratedGpu`, `MinGpuVramGB`)
decides, and the sizer, the eligibility query and the router's spawn-time retune all take the
same struct. The READINGS come from the platform readers (rocm-smi's `GTT Total*` and the
kernel's own sysfs counters on Linux, DXGI's `SharedSystemMemory` + PDH `Shared Usage` on Windows,
nothing at all from nvidia-smi), but they are raw fields on `perf.GpuStat` — no platform decides
eligibility.
`SharedMemory: auto` (default) counts the pool only for a device that looks
integrated. Strongest signal first: `perf.GpuStat.Integrated`, which is KFD's
`local_mem_size == 0` (the kernel saying the device has no memory of its own, and the only thing
that classifies a Strix Halo, whose 16GB carve-out and unhelpful name look exactly like a card);
then a name marker (`ryzen`/`athlon`/`radeon graphics`, which cannot match the `amdgpu [1002:15bf]`
a sysfs read synthesizes); then dedicated ≤ 8 GB with shared ≥ 3× dedicated. A device under the floor used to be dropped outright, which on an APU-only box
hid the ONLY GPU in the machine from every budget in the program (issue #37); the floor still
drops it, but the pool is folded first, and "do not PAIR an iGPU with a real card" is now a
separate rule applied to the resulting SET (`dropUnpooledIntegrated`) — an all-integrated set is
kept. One pair of functions owns the answer: `effectiveGpuStat` folds, `gpuSetFromStats` builds —
every consumer (including `internal/server`'s OOM guard) goes through those, so the sizer and the
guard cannot disagree about the budget. `cmd/monitor-test` prints `DescribeGpuStats` for a bug
report from a machine we do not have.
- **Single-device backends need an explicit pin, and the ordinal in it is the BACKEND's.**
sd-server, tts-server, whisper and the embedding server have no split of their own, so on a
multi-GPU box `writeSingleDeviceEnv` pins each to the plan's main device. Without it the
runtime picks a card the sizer did not budget, and not necessarily the same one twice. SAM is
exempt: it runs on the CPU. The variable and its value both come from
`singleDeviceEnvFor` -> `backenddev.go`'s probe: a ggml device id ("Vulkan1") names the
enumeration to filter and the ordinal to give it, which is the only way to get the Vulkan or
ROCm value right, since the telemetry index is a different enumeration. CUDA alone has a
fallback when the probe fails, because `CUDA_DEVICE_ORDER=PCI_BUS_ID` makes the runtime's
ordinal agree with nvidia-smi's; everything else emits nothing rather than a guess.
- **The backend's device list is not ours, in order or in set** (`backenddev.go`). `--main-gpu`
and `--tensor-split` are positions in the BACKEND's list, but the plan derives them from the
telemetry ordinal, and an adapter we filtered out is still counted there. `DeviceFlagFor`
closes the gap by emitting `--device <ids>` so the list llama.cpp indexes into is the one we
handed it, and `--main-gpu` becomes a position in the resolved `GpuSet`. The probe doubles as
the capability check: a build without `--device` has no `--list-devices` either, so it cannot
be handed a flag it would reject. Everything refuses rather than guesses -- an unmatched
device yields no ids at all -- so a box that cannot be mapped emits exactly what shipped.
`DeviceFlagFor` returns the ORDER it built the list in as well as the ids, and the caller must
apply it to `--tensor-split`. **The probe runs under `CUDA_DEVICE_ORDER=PCI_BUS_ID`
(`probeEnv`), because the launch does.** The CUDA runtime defaults to `FASTEST_FIRST`, so a
listing read under the inherited environment numbers a mismatched pair the other way round,
and the id read off it names the OTHER card once handed to a bus-ordered process: the exact
reversal `--device` exists to prevent, arrived at by a different route. Vulkan and ROCm
ignore the variable. `retuneTensorSplit` re-derives that list from the LIVE main
device rather than trusting the baked one (the live main need not be the one generate picked),
rewrites `--device`, `--tensor-split` and `--main-gpu` together, and rewrites NOTHING when the
probe fails on an argv that carries `--device`: a split written in set order against a list in
main-last order would hand each card the other's ratio, which is worse than the stale one.
For the same reason `--device` is suppressed entirely when an override pins `tensorSplit`
(`pinsOwnSplit`, `generate_cmd.go`): a hand-written ratio is positional against the list the
backend would have enumerated by itself, so renaming the list under it re-points the user's
own numbers at different cards. Cost is bounded twice over: `backendProbeTimeout` caps one
probe, `backendProbeBudget` caps all of them together, since a generate can reach five
distinct backend binaries and the per-probe window is deliberately generous.
- **Sidecar SHADOWS the file row, it does not field-merge.** Override resolution is row-level
first-match (sidecar rows prepended), so a sidecar row replaces the matching file row
wholesale. A UI save must therefore write a *superset*: the config editor seeds the sidecar
row from `ResolveFileOverride` (the matched FILE override, sidecar excluded) before applying
the edited fields, so file-only knobs the UI doesn't model (`ctxVariants`, `quant`,
file-defined variants like `judge`) aren't dropped. Fleet-wide `settings.defaultVariants`
(e.g. `game`) are emitted independently of overrides, so they survive regardless — which is
why a buggy save used to lose the ctx tiers + `judge` but keep `game`.
- **Sidecar ownership** — `quartermaster-overrides.yaml` is fully owned by the UI and rewritten
whole on any edit, kept separate from the comment-rich hand-authored generate file.
- **Named variants INHERIT the model-wide override.** Each `<model>-<variant>` profile layers its
engine knobs over a copy of the resolved model override, so the spec/draft chain, kv quant,
reasoning budget, preserve-thinking etc. flow down at generate time; a variant's own
non-blank/non-zero field still wins. Was previously *standalone* over `Override{}`, which
dropped the draft chain + kv on every variant but the one the user hand-edited.
- **Drafters and projectors are FAMILY-wide, dir-local first.** A sidecar is published for a
model but downloaded next to one copy of it, so a second quant in its own folder — or a
finetune of a base that shipped an mmproj — used to see nothing. `inheritSidecars`
(`family.go`) fills only blank `DraftPath`/`MmprojPath`, so a model that ships its own drafter
always keeps it, and picks the closest donor: another quant of the same model (`ModelBaseKey`),
then the family's un-tuned base, then a peer finetune. **The name match alone never authorizes
a donation** — the donor's model and the recipient must also agree on arch, embedding length,
block count and vocab size (`sidecarCompatible`). That gate is not cosmetic: a drafter whose
vocab differs doesn't degrade, it aborts the launch (`tensor 'output.weight' has wrong shape`).
- **`Override.MmprojFile` names a projector outright, and CREATES the twin.** Discovery only
ever sees a projector in the model's own folder or in a family sibling's, so one shared
mmproj kept in a folder of its own is invisible: no twin, no explanation, and hand-typing
`--mmproj` into the launch box is swallowed by `IGNORE_VALUE` (`modelCmdForm.ts`) on purpose.
`mmprojFile` (model-wide, or on the reserved `vision` variant with the usual sentinel) is
resolved by `mmprojFor` (`family.go`) and beats discovery entirely; `mmproj: none` still
wins over it, since placement is checked past the twin gate. The path is `os.Stat`'d there
rather than read off `GgufRow.MmprojSizeGB` — that size describes DISCOVERY's file, and
pricing an override's projector at another file's size is how a twin ends up sized against
VRAM it will not have. **Automatic pairing is deliberately NOT widened to match**: a
projector is bound to one vision tower, so an auto-paired unrelated one would load clean and
then hallucinate on every image. `mmprojFileErr` (`configapi.go`) is the safety net that
replaces it — save rejects a path that is missing, a directory, or whose gguf header is not
`architecture == "clip"`, the same rule `MmprojSidecarForDir` pairs by.
Consequences to keep in mind: a model that never had a `-vision` twin can grow one, `-md`
appears where no drafter is visible in the folder, and anything summing `DraftSizeGB` across
rows must dedupe on `DraftPath` (one file, many rows — `internal/setup/probe.go`). Per-model
opt-out is the existing spec override (no `draft-*` → no `-md`) and marking the vision twin
unlisted. `ModelBaseKey`/`FamilyKey` mirror `baseKey`/`familyOf` in
`ui-svelte/src/lib/modelTable.ts` — change one, change both.
- **`settings.encoders` is now a PIN, not the source.** Diffusion component paths are
discovered structurally (`encoderpool.go`, see `classes.md`), and `fillEncoderSet` fills only
the roles left blank, so a declared path always wins and no existing config changes behaviour.
**Except a declared file that is GONE** (v118, `declaredGone`): not in the scan AND `os.Stat`
says not-exist (any other stat error keeps it). It is cleared before the fill. Kept, it read
as satisfied: the regen emitted a flag at a missing file and the model browser never offered
the part, because nothing was "missing". Found by moving the FLUX.2 VAE out of the models tree
with `flux2Vae` still pinned to it. Test fixtures declaring bare names need `onDisk(t, ...)`.
`qwenLlm` was the exception until v71: the scan ignored a declared pin whenever it found any
encoder of the DiT's width, so the Qwen3-4B vs Qwen3-VL-4B tie silently overrode the user.
`EncoderPool.Llm` now takes the declaration as `prefer` and returns it when it clears the same
width/vision gates, which is why configs with a pin regen once.
Two consequences worth remembering: the pool holds every ordinary chat gguf too (any decoder
can be a text encoder), so a change to how candidates are ranked can silently re-point a
working image model, and `resolveComponents` therefore takes both the pool and the DiT's
`CondHidden` - a caller that passes 0 gets the declared-only behaviour, which is exactly what
the older tests assert.
- **Video components are scoped by family, and the two audio VAEs do not cross-wire.**
`EncoderPool.AudioVae` takes a family first (empty matches any), `fillEncoderSet` fills the
single `VideoVae`/`AudioVae` pins with MiniMax-H3's `VaeFamilyVideo3D` pair only, and LTX
resolves its own `VaeFamilyLtx` pair in `videoComponents`: the two soundtrack decoders are
unrelated networks over unrelated latents, so a wrong pick fails the load. The text encoder is
the same trap one level up: LTX's is a Gemma-4-12B republished with the caption projection
grafted on, and a stock Gemma-3-12B is the same 3840 wide, which is why `resolveComponents`
feeds `pool.LlmHinted("ltx")` in as the `prefer` argument.
- **"A pin is declared" and "the pin applies here" are different questions.** `settings.encoders`
is ONE global set shared by every diffusion model on the box, and `EncoderPool.Llm` honours
`prefer` only for a candidate that clears the same width and vision gates: a `qwenLlm` aimed at
an image model (Qwen3-4B, 2560 wide) is silently dropped for a 3840-wide video one and decides
nothing. So a family carrying its own path hint must gate on `pool.llmCandidate(...)`, not on the
field being blank. Gating on blankness suppressed LTX's hint in the only configuration that
matters, a populated `encoders:` block, which every real install has, and the symptom is not an
error: LTX loads a stock Gemma of the right width and conditions on nothing.
- **A gguf can have a full tensor table and ZERO hyperparameter KVs.** That is not just MiniMax-H3
(no `general.architecture` at all): LTX-2.5's text encoder declares arch `gemma4`, ships 686
tensors, and writes no `gemma4.embedding_length`, because it was converted by ComfyUI tooling
that keeps tensors and treats llama.cpp's KVs as optional. `readTensorScan` therefore records
`embedWidth` from ne[0] of `token_embd.weight` OR HF's `model.embed_tokens.weight`, and
`ReadGgufMetadata` uses it only when the KV is absent. Without it the width reads 0, which
`ScanEncoderPool` interprets as "a diffusion model or something with no hidden width" and drops
the file from the pool entirely: no warning, no row, just a missing encoder. Same shape as the
`captionChannelsFrom` fallback, and for the same reason.
- **A new quant type needs THREE tables, and the header gate is why.** `quantRe` (`discover.go`)
names it in a FILENAME, `ggmlTypeSize` (`gguf.go`) sizes its tensors, `ggmlTypeName`
(`quantlabel.go`) names the TYPE id (a type that can be weighed but not named makes every file
carrying it unlabelled — `TestGgmlTypeTablesAgree` asserts the pair). Miss the second and every gguf in that
quant comes back with `VocabSize` 0 — which mis-sizes the logits buffer AND makes
`sidecarCompatible` refuse it every family sidecar, silently. The tensor walk no longer aborts
on an untabulated type (only the MoE expert share degrades to 0), but adding the type is still
the fix. MXFP4 was the case that found this.
- **`AppSettings` is a REPLACE, and `SettingsPatch` is a MERGE — do not unify them.** The dashboard
renders the process-level block as one form and PUTs all of it, and its fields have no meaningful
"unset from this section" state: clearing `adminAllow` must actually clear it, which a merge
cannot express. `mergeAppSettings` (file → sidecar) is written out field by field rather than
reflected, because "unset" differs per field (`""`, `0`, `nil`) and one blanket rule turns a
deliberately-emptied `AdminAllow` back into the old value. Precedence:
`argv > sidecar app block > settings.app > built-in default` — argv wins so
`quartermaster.exe -listen 127.0.0.1:1250` can always rescue an install whose stored address no
longer binds.
- **`ReplaceSidecarSettings` exists because `MergeSettingsPatch` can only SET.** A per-section
"restore defaults" (the advanced sizer knobs) has to nil that section's fields and store the
result verbatim; merging a patch full of nils is a no-op. Use it only after a read-modify-write
that preserves every other section's fields.
- **Empty `modelsRoot` is valid, not an error** — the server boots with an empty catalog so a
setup UI can point it at a folder later; discovery and hashing short-circuit on blank.
- **`Parallel` > 1: sized per slot, emitted as a pool.** `--kv-unified` makes `-c` ONE KV buffer
shared by all `--parallel N` slots, so N conversations of the sized window need N x that pool.
`Generate` multiplies the per-token KV cost by `profileParallel` **before** `sizeProfile` (so the
clamp shrinks the per-slot ctx to what VRAM holds) and `emitProfile` multiplies the result back
out into `-c ctx*parallel`. Both read the slot count through `EffectiveParallel`/`profileParallel`
(variant wins over the model-wide override, capped at `MaxParallelSlots`) — if one side ever reads
a different number, the server either over-commits VRAM or hands slot 1 the whole card.
`/api/models` divides `-c` back down so the UI reports the per-conversation window.
- **Determinism for tests** — `Generate` takes the timestamp as an argument (`DefaultNow()` is
separate) so output is reproducible.
- **Caching** — metadata by size+mtime (`metacache.go`); replacing a gguf invalidates its entry.
The config itself is cached via the `.modelhash` sidecar digest.
- `gguf.go` parses the tensor section too (for expert share); a tensor of unknown ggml type
leaves the share at 0 (arch-table fallback) rather than erroring the whole read.
### Sampler defaults
- **Only `--top-k` and `--min-p` get an arch baseline** (`defaultSamplerFor`,
`generate_cmd.go`): Qwen3-family models (`qwen3*`, all 3.x minors) emit `--top-k 20 --min-p 0`,
`muse-glimmer*` emits `--top-k 64` and NO min-p (its card pins one and not the other),
everything else emits nothing. The asymmetry is the point — neither knob has an OpenAI-API field, so a client
physically cannot set them and llama-server's defaults (top-k 40, min-p 0.05) are what every
request got, for every model, regardless of what the model card says. Temperature/top-p/
presence-penalty are sent by nearly every client, so a launch flag would be overwritten on
arrival; they stay opt-in per model. They are also the mode-dependent ones (Qwen documents
temp 1.0 thinking vs 0.7 instruct), and one process serves both modes, so there is no single
correct value to seed.
- **`Temp`/`TopK`/`TopP`/`MinP`/`PresencePenalty` are POINTERS, unlike the zero-gated knobs
around them.** 0 is a meaningful, non-default value for every one (`--min-p 0` disables
truncation, `--temp 0` is greedy), so `0 => inherit` would make the exact values model cards
recommend unexpressible. Everything downstream must follow: the variant merge tests `!= nil`,
the DTO patch tests `!= nil`, and the UI reads them with `??` not `||` (`advFromOverride`,
`vsamp`/`vsampSet` in `ModelConfigModal.svelte`).
- The UI's launch-command box parses these back (`parseCmdFields`), capturing them as a **delta
vs what the generator already emits** (`genDefaultNum`, same trick as `genDefaultKv` for
`-ctk`) — otherwise blurring the box for any unrelated reason freezes the arch baseline into
an explicit per-model pin that never tracks a future baseline change.
- **Do not add speculative baseline entries.** A survey of ~19 current local models (Llama
3.x/4, Mistral, Magistral, Gemma 3, Phi-4, DeepSeek R1/V3.1, GLM-4.6, gpt-oss, Kimi-K2,
Nemotron Nano, Qwen2.5/QwQ) found no top-k or min-p recommendation outside Qwen3 and Muse
Glimmer — those cards publish temperature and top-p, which clients send anyway. The empty
default is the researched answer, not an unfinished table. Add a case only with a model card
in hand, and match on the arch string llama.cpp actually registers (check the GGUF metadata
or the upstream PR — the card's prose name is not the arch name).
### Chat template & reasoning effort
- **No chat template is ever substituted automatically.** `--chat-template-file` is emitted
only when the model has a `chatTemplateFile` override. Chat templates are the user's to
download and wire up; shipping a curated replacement for one vendor's family played
favourites and silently dropped whatever the baked template supported that the replacement
did not (Qwen 3.8's reasoning-effort ladder, for one).
- **A variant can drop an inherited template with `chatTemplateFile: none`.** Empty means
"inherit the model-wide value" on every free-form string a variant carries, which left no
way to say "this profile runs on the gguf's baked-in template". `inherit.go` gives the off
state a name (`NoneSentinel`, matching the `mmproj: none` vocabulary) and owns the whole
rule: `inheritStr` at merge, `NormalizeNone` on every door an `Override` comes in
through so a sentinel written at MODEL level can never reach the emitter as a literal
path. **Both doors, not just the config file:** `LoadGenerateFile` covers the file, and
`applyOverrideDTO` covers the editor — `handleAPIModelCmdPreview` renders the
launch-command box straight from that DTO without ever reading the config, so a sentinel
left unresolved there shows `--chat-template-file "none"` and the preview silently
disagrees with what saving produces. Never normalize a `VariantSpec`; there the sentinel
is the point. It covers the free-form/path knobs
only — `chatTemplateFile`, `extraArgs`, `tensorSplit`, `overrideTensor`, `kvKDraft`/
`kvVDraft`, and the sd-server component paths. The enums are deliberately excluded: they
already spell out their own off state (`flashAttn: off`, `mmproj: none`, `ropeScaling:
none`), and a second spelling would be ambiguous. `ModelConfigModal.svelte` keeps hand-kept
twins (`inheritStr`/`deltaStr`) so the previewed command matches what the generator emits,
and so deleting a flag from a variant's launch box saves as `none` rather than silently
inheriting it straight back.
- **`scanChatTemplate` still reads the baked template**, but only for the effort ladder.
`ReadGgufMetadata` decodes `tokenizer.chat_template` and derives `ChatTemplateEffortLevels`
(plus `ChatTemplatePreservesThinking`, which nothing consumes today) — the flag and the level
list are kept, never the ~10–100 KB source. Note that arch cannot distinguish Qwen minors:
3.5, 3.6 **and** 3.8 all report `qwen35`/`qwen35moe`, which is why nothing here branches on
arch to decide template behaviour.
- **The ladder always describes the template that will actually run.** A
`--chat-template-file` replaces the baked template wholesale, so the gguf's levels say nothing
about the live renderer — `effortLevels` (`generate_emit.go`) therefore scans the **override
file** in the gguf template's place (memoized per path; one file is shared by every ctx variant
of every model using it). Whatever survives that scan becomes the emitted
`capabilities.reasoningEffort`. There is no longer a special case forcing `nil` for the old
bundled Qwen template, so a model with no override now advertises the ladder its own baked
template actually implements.
- **The effort ladder is read out of the template, never hardcoded.** Two shapes, in order:
- *Strict* — `effortLevelsRe` matches 3.8's `reasoning_effort not in ('xhigh', 'medium', 'low')`
validation line, so the advertised set is exactly what the jinja accepts.
- *Tolerant* — a template that reads `reasoning_effort` and folds it onto its own rungs without
validating it (the Qwen 3.8 drop-ins in circulation) declares no tuple, so `effortAssignRe`
reads the literals **assigned** to an effort-named variable (`set _initial_effort = 'xhigh'`),
not the ones compared against it — the comparisons are padded with OpenAI-ladder aliases the
template folds away. Gated on the template raising nothing about effort (`effortRaiseRe`,
which deliberately ignores the unrelated content raises every real template carries): with no
guard in play, a rung read wrong degrades to the template's own default rather than the 500
llama.cpp returns for an unknown kwarg value.
A template that reads `reasoning_effort`, names no rungs, and validates none advertises
**nothing**. This is what `internal/server/http-core.md` (`reasoning_effort.go`) snaps an
OpenAI-ladder value onto.
## Connections
- **Depends on:** `internal/perf` and `internal/logmon` (live GPU/VRAM telemetry in `vram.go`);
`gopkg.in/yaml.v3`.
- **Called by:** `cmd/quartermaster/quartermaster.go` at startup — `EnsureConfig` regenerates the config before
the router loads it, and `-watch-models`/reload paths use `CachedConfigHash`/`CurrentInputsHash`
to detect changes. `internal/server` config API reads/writes sidecar overrides and settings,
previews tunings via `EstimatePlan` + `ReadGgufMetadataCached`, and triggers `EnsureConfig` on
save (`internal/server/configapi.md`).
- Produces the YAML consumed by `internal/config` / the router/scheduler; its `groups`/`listeners`
output backs the fork's multi-listener + cross-port eviction features.