internal/autogen/generate_cmd.go
Rama main · réplica de lectura del repositorio original.
package autogen
// Launch-command rendering: turns a sized profile (ctx, offload split, KV
// types) into the llama-server / sd-server / tts-server argv lines that land in
// the generated config. RenderSoloCmd is the same path with a ${PORT}
// placeholder, used by the UI's cogwheel preview and ad-hoc commands.
import (
"fmt"
"strings"
)
// isVLModel reports whether a model is a dedicated vision-language model (its
// core identity is image chat: Qwen2/3-VL, InternVL, CogVLM), as opposed to a
// text LLM that merely ships an mmproj sidecar for optional vision. Only the
// former gets its vision twin's ctx capped to VisionCtx. VL archs carry "vl" in
// their gguf arch (qwen2vl/qwen3vl/qwen3vlmoe) and such models are conventionally
// named "*-VL"; a plain LLM-with-vision (gemma3, mllama, mistral) matches neither.
func isVLModel(name, arch string) bool {
return strings.Contains(strings.ToLower(name), "vl") ||
strings.Contains(strings.ToLower(arch), "vl")
}
// cmdPath renders a filesystem path as ONE cmd argument. NOT `%q`: the emitted
// cmd is re-split by shlex (Windows rules on Windows), where a backslash inside
// double quotes is a literal separator — so Go's escaping turned D:\Models\x
// into D:\\Models\\x and llama-server failed to open it. Separators are
// normalized to "/" (accepted by Windows too, and what slotKvPath already does),
// and quotes are kept for paths with spaces.
func cmdPath(p string) string {
p = strings.ReplaceAll(strings.TrimSpace(p), "\\", "/")
return `"` + strings.ReplaceAll(p, `"`, "") + `"` // a `"` can't appear in a real path
}
// corsOriginsFlag locks a spawned llama-server down to localhost origins.
//
// llama-server defaults to --cors-origins '*' WITH credentials, which means it
// echoes back whatever Origin it is given: any page you happen to visit can read
// http://127.0.0.1:<upstream port>/props (model path + full launch args) and
// /slots (live prompt and KV contents), and can drive generation. Binding to
// 127.0.0.1 does not help — the browser is already inside the loopback. It is
// also what llama-server warns about at init ("CORS is set to allow all origins
// ('*') and no API key is set").
//
// Nothing in a browser is supposed to reach an upstream directly: the UI and
// every API client go through quartermaster's own listener, which proxies
// server-side and has no Origin to speak of. So the lockdown costs nothing.
// 'localhost' is llama-server's special value for "reflect the Origin only if it
// is localhost". Emitted early, so a user's extraArgs can still override it.
const corsOriginsFlag = "--cors-origins localhost"
// loadMode renders llama-server's --load-mode, the single flag that replaced
// --mmap/--no-mmap, --mlock and -dio (all three now warn "DEPRECATED in favor of
// `--load-mode`" at startup). Returns "" or a flag with ONE trailing space, so
// it can be interpolated into the flag run without leaving a double space.
//
// Modes are an enum, not a set, so the three old booleans do not all survive
// contact: there is no dio+mlock. DirectIO wins that one — it is a streaming
// load path that bypasses the page cache, which is the opposite of what pinning
// pages in RAM is for, so the pair was already self-defeating.
//
// mmap alone stays UNSET on purpose. The default is 'auto' ("mmap, unless a
// device does not support it"), which is what we emitted before by omission;
// forcing 'mmap' would take llama.cpp's device check out of the loop.
//
// Note this makes a generated config require a llama-server new enough to know
// --load-mode. Every backend the installer fetches has it; a hand-pointed
// binary from before the flag landed would fail to spawn on an unknown arg.
func loadMode(mmap, mlock, dio bool) string {
switch {
case dio:
return "--load-mode dio "
case mmap && mlock:
return "--load-mode mmap+mlock "
case mlock:
return "--load-mode mlock "
case mmap:
return ""
default:
return "--load-mode none "
}
}
// defaultSamplerFor returns the arch-derived server-side sampler baseline for a
// model: the values an override leaves nil fall back to. nil/nil => emit nothing
// and take llama-server's own defaults.
//
// Only top-k and min-p are seeded, and deliberately so. Neither has an OpenAI-API
// field, so a client physically cannot set them through /v1/chat/completions —
// llama-server's defaults (top-k 40, min-p 0.05) are what every request gets, for
// every model, forever, unless the launch command says otherwise. Temperature and
// top-p are the opposite: near enough every client sends them, so a launch-flag
// value would be overwritten on arrival and would only ever act as a floor. Those
// stay opt-in per model.
//
// The Qwen3 family (3.x dense and MoE alike, all reporting "qwen3*" — llama.cpp
// has not split out a separate arch per Qwen minor, so 3.5, 3.6 and 3.8 are
// indistinguishable here) documents
// top_k 20 / min_p 0.0 across every model card, in both thinking and instruct
// mode, so it is safe to apply without knowing which mode a request wants
// (checked on the 3.6-27B and 3.8-27B cards; 3.6 additionally lists a coding
// preset, which likewise moves only temperature). The
// mode-dependent values (temperature, top-p, presence-penalty) are NOT seeded:
// one process serves both modes here, so there is no single correct choice, and
// they are the client-settable ones anyway.
//
// Muse Glimmer is the only other family found to pin a top-k. It asks for 64 —
// wider than llama's 40, and constant across all four reasoning-strength levels —
// but says nothing about min-p, so min-p stays nil rather than being invented.
// That asymmetry is exactly why this returns two independent pointers instead of
// a struct that would tempt a caller to fill both.
//
// A survey of ~19 current local models (Llama 3.x/4, Mistral, Magistral, Gemma 3,
// Phi-4, DeepSeek R1/V3.1, GLM-4.6, gpt-oss, Kimi-K2, Nemotron Nano) turned up no
// other top-k or min-p recommendation at all: the values those cards publish are
// temperature and top-p, which clients send anyway. Absence here is a finding,
// not a gap — for those models llama's own defaults are the documented state of
// the art, so do not add speculative entries.
func defaultSamplerFor(meta Metadata) (topK *int, minP *float64) {
arch := strings.ToLower(meta.Architecture)
switch {
case strings.HasPrefix(arch, "qwen3"):
k, p := 20, 0.0
return &k, &p
// Prefix, not equality, to match the convention llama.cpp uses for family
// members (qwen3 -> qwen3moe / qwen35moe): a future muse-glimmer MoE or minor
// would land on "muse-glimmer*" and should inherit the same top-k.
case strings.HasPrefix(arch, "muse-glimmer"):
k := 64
return &k, nil
}
return nil, nil
}
// samplerLines renders the sampler-default flags for a resolved override, layered
// over the arch baseline. An override value always wins, including an explicit 0
// (hence the pointers — 0 is a real, non-default value for each of these).
func samplerLines(meta Metadata, ov *Override) []string {
topK, minP := defaultSamplerFor(meta)
var temp, topP, presence *float64
if ov != nil {
if ov.TopK != nil {
topK = ov.TopK
}
if ov.MinP != nil {
minP = ov.MinP
}
temp, topP, presence = ov.Temp, ov.TopP, ov.PresencePenalty
}
var out []string
if temp != nil {
out = append(out, fmt.Sprintf("--temp %g", *temp))
}
if topK != nil {
out = append(out, fmt.Sprintf("--top-k %d", *topK))
}
if topP != nil {
out = append(out, fmt.Sprintf("--top-p %g", *topP))
}
if minP != nil {
out = append(out, fmt.Sprintf("--min-p %g", *minP))
}
if presence != nil {
out = append(out, fmt.Sprintf("--presence-penalty %g", *presence))
}
return out
}
// smallCardVramGB is the budget below which a high-ctx profile falls back to
// ub=512. The ub=1024 compute buffer (~0.5 GB larger) only fails to fit on a
// genuinely small card at 64k+; bigger cards keep ub=1024 everywhere (bench:
// ub=512 costs ~52% prefill vs 1024).
const smallCardVramGB = 12.0
// MaxParallelSlots caps --parallel. Slots are cheap in code but not in VRAM:
// every extra slot multiplies the KV pool sizing (see buildCmdLines), and the
// slot cache keeps per-slot bookkeeping. 8 is far past any local multi-agent
// setup; a typo of 400 would otherwise size a pool nothing can hold.
const MaxParallelSlots = 8
// profileParallel resolves the slot count for one profile the same way emit
// does: the variant's own parallel wins over the model-wide override. It exists
// so sizing can charge the slot multiplier BEFORE effOv is assembled.
func profileParallel(prof profile, ov Override) int {
if v := prof.Variant; v != nil && v.Parallel > 0 {
return EffectiveParallel(&Override{Parallel: v.Parallel})
}
return EffectiveParallel(&ov)
}
// EffectiveParallel resolves llama-server's --parallel (server slot count) for a
// model: the per-model/variant override, else 1, clamped to MaxParallelSlots.
// Sizing (generate.go) and command emission (buildCmdLines) MUST agree on this
// number — one charges N x KV per token, the other multiplies -c by it.
func EffectiveParallel(ov *Override) int {
if ov == nil || ov.Parallel <= 0 {
return 1
}
return min(ov.Parallel, MaxParallelSlots)
}
// effectiveUb picks the physical batch. 512 for dense models, 1024 for MoE.
// Drop to 512 at high ctx ONLY when the VRAM budget is small (< smallCardVramGB)
// — the auto-sized solo profile (Ctx=0) learns its real ctx after sizing, so
// ctx>=longCtxThreshold is checked alongside prof.IsLong. budgetGB<=0 (unknown)
// keeps the class default. An explicit Ub override always wins.
//
// The dense/MoE split is benched, and the two classes want opposite things:
//
// - Dense, fully GPU-resident: a bigger physical batch buys nothing. Measured
// on a 7900 XTX with Qwen3.8-27B-UD-Q4_K_XL (b=2048 held fixed), pp2048 was
// 682 / 679 / 667 t/s at ub 512 / 1024 / 2048 — flat-to-inverted, with 512
// ~2% ahead at every depth tested (d0 682 vs 667, d16384 543 vs 532, d65536
// 335 vs 327). Since computeBufferGB caps its vocab-scaled logits term at
// exactly computeLogitsTokens (1024), halving ub to 512 halves that term as
// well as activations — ~0.37GB of VRAM back on a 27B/151k-vocab model, for
// free-to-slightly-positive throughput. That headroom is worth more as ctx.
// - MoE with CPU expert offload: the opposite. A larger ub amortises the
// per-micro-batch PCIe expert fetch over more tokens — benched 380 -> 647
// t/s going ub 512 -> 1024 at b=2048 on the 8GB/32GB rig. Keep 1024.
//
// Keyed on meta.IsMoE rather than on whether offload was actually planned: a
// fully-resident MoE has no fetch to amortise and would likely behave like the
// dense case, but that is untested, so MoE keeps the conservative 1024.
func effectiveUb(meta Metadata, prof profile, ov *Override, ctx int, budgetGB float64) int {
ub := 512
if meta.IsMoE {
ub = 1024
}
if (prof.IsLong || ctx >= longCtxThreshold) && budgetGB > 0 && budgetGB < smallCardVramGB {
ub = 512
}
if ov != nil && ov.Ub > 0 {
ub = ov.Ub
}
if prof.Ub > 0 {
ub = prof.Ub
}
return ub
}
// buildCmdLines returns the generated launch command as a list of flag lines
// (no leading indentation), shared by emitProfile (which writes them as a YAML
// `cmd: >` block) and RenderSoloCmd (which joins them for the editor preview).
// The user's custom launch arguments are NOT applied here: callers pass the
// result through ComposeCmd, which drops the generated flags a custom knob owns
// and appends the user's text last.
// pinsOwnSplit reports whether the user hand-wrote the --tensor-split vector for
// this model, in which case we must NOT emit --device.
//
// A hand-written ratio is positional against whatever list the backend would
// have enumerated on its own, which is what it meant when the user typed it.
// --device replaces that list with ours, in main-last order, so the same string
// would silently start addressing a different pair of cards. The override is
// appended after the generated flags and wins on the last-flag-parsed rule, so
// it survives either way: the choice here is only whether it lands on the cards
// the user meant. Same refusal contract as everywhere else in this path, a split
// that cannot be mapped is not rearranged.
func pinsOwnSplit(ov *Override) bool {
return ov != nil && strings.TrimSpace(ov.TensorSplit) != ""
}
func buildCmdLines(s Settings, meta Metadata, row GgufRow, prof profile, ctx, ngl, ncpuMoe int, kvK, kvV string, kvInRam bool, ov *Override) []string {
cpuMoeFlag := ""
if ncpuMoe > 0 {
cpuMoeFlag = fmt.Sprintf(" --n-cpu-moe %d", ncpuMoe)
}
// mmap only helps when some weights live on the CPU: mmap gives lazy
// demand-paging + natural OS page-caching of the on-CPU tensors. With NO CPU
// offload the weights are copied straight into VRAM, so mmap only leaves a
// redundant file-backed copy in host page cache — waste. So default to
// mmap off and only keep it when CPU offload is actually happening:
// - --n-cpu-moe > 0 (experts on CPU), or
// - partial layer offload (ngl <= blocks => some layers on CPU).
// Unknown block count (parse miss) assumes GPU-resident => mmap off.
// Explicit Mmap:"on"/"off" always wins over this default.
cpuOffload := ncpuMoe > 0 || (meta.BlockCount > 0 && int64(ngl) <= meta.BlockCount)
mmapOn := cpuOffload
if ov != nil && ov.Mmap == "off" {
mmapOn = false
} else if ov != nil && ov.Mmap == "on" {
mmapOn = true
}
loadModeFlag := loadMode(mmapOn, ov != nil && ov.Mlock, ov != nil && ov.DirectIo)
kvoFlag := ""
if kvInRam {
kvoFlag = "--no-kv-offload "
}
fa := "on"
if ov != nil && ov.FlashAttn != "" {
fa = ov.FlashAttn
}
// One slot by default; a per-model override raises it for multi-agent harnesses
// (Qwen Code = main + memory subagent, an IDE plugin + a CLI on the same model)
// that would otherwise thrash one slot's preamble. The slot cache pins each
// conversation to a slot (id_slot) and saves/restores that slot, so >1 is a
// supported configuration, not a foot-gun — see internal/server/slotcache.md.
parallel := EffectiveParallel(ov)
threads := s.Threads
if ov != nil && ov.Threads > 0 {
threads = ov.Threads
}
ub := effectiveUb(meta, prof, ov, ctx, s.TargetVramGB)
// -b (logical batch) decoupled from -ub (physical/compute tile). A larger logical
// batch pipelines more ub-micro-batches per decode(), overlapping CPU expert-fetch
// with GPU compute on MoE offload — measured +38% pp at ub1024, +20% at ub512, for
// ZERO extra VRAM (the compute buffer is sized by ub, not b; b only sizes the
// token/pos arrays). 2048 is the plateau (bench: b past 2048 flat). Clamp >=ub and
// <=ctx. ponytail: fixed 2048, expose a per-model b override only if a model wants
// less pipelining depth.
bTok := 2048
if bTok < ub {
bTok = ub
}
if bTok > ctx {
bTok = ctx
}
spec := effectiveSpec(meta, ov, row.DraftKind)
if prof.Spec != "" {
spec = prof.Spec
}
reason := prof.ReasoningFmt
if reason == "" && ov != nil {
reason = ov.ReasoningFmt
}
rfmt := "auto"
reasoningFlag := ""
switch {
case reason == "off":
rfmt = "none"
reasoningFlag = " --reasoning off"
case reason != "":
rfmt = reason
}
modelPath := strings.ReplaceAll(row.FullPath, "\\", "/")
lines := []string{
s.ServerExe,
fmt.Sprintf("-m %s", modelPath),
"--port ${PORT}",
"--host 127.0.0.1",
corsOriginsFlag,
fmt.Sprintf("-ngl %d", ngl),
// -c is the TOTAL KV pool: --kv-unified makes all slots share one buffer, so
// N slots each keeping the sized per-slot context needs N x that pool. ctx is
// already the PER-SLOT window (sizing charged N x the per-token KV cost and
// clamped to VRAM), so multiplying here is what actually reserves it.
fmt.Sprintf("-c %d", ctx*parallel),
fmt.Sprintf("-ub %d -b %d", ub, bTok),
fmt.Sprintf("-fa %s -ctk %s -ctv %s", fa, kvK, kvV),
fmt.Sprintf("--parallel %d %s%s--kv-unified --no-warmup --no-ui --metrics --props", parallel, loadModeFlag, kvoFlag),
}
if ov.CpuDevice() {
// -ngl 0 alone still opens the GPU: a compute buffer and runtime context
// stay resident and op offload borrows it for large batches. --device none
// is what makes a CPU pin hold no VRAM, which the cpu pool assumes.
lines = append(lines, "--device none")
}
// Multi-GPU: split the layers (and, under -sm layer, their KV with them)
// across every eligible card, pinning the buffers that do NOT split to the
// device with the most free memory. Emitted only when the sizer actually
// resolved a split, so a single-GPU box and a box with no telemetry both
// produce byte-identical output to before.
//
// -sm layer, not row: row-split needs the interconnect to carry activations
// per layer and is a loss on consumer boards with no NVLink, which is what a
// mismatched desktop pair is.
if len(prof.TensorSplit) > 1 && ngl > 0 {
// --main-gpu is INERT under -sm layer. Measured, not inferred: at a fixed
// --device list, --main-gpu 0 and --main-gpu 1 produced byte-identical
// model, KV and compute buffer lines. It is kept because it costs nothing
// and is the correct pin for anyone who puts -sm none or -sm row in
// extraArgs, but nothing here should be read as controlling placement.
//
// What DOES control placement under layer split is the ORDER of the device
// list: llama.cpp puts the non-splittable output weight on the device
// listed last. MainLastOrder puts the plan's main device there, and
// PermuteSplit moves --tensor-split with it so both flags still address the
// same cards. That is what makes the main device's larger fixed budget
// (mainFixedGB, see splitBy) land on the card that was charged for it.
//
// Naming the devices is also what makes the positions mean anything:
// without --device they are positions in the BACKEND's list, which counts
// adapters we filtered out and need not be in our order, and Vulkan/ROCm
// have no CUDA_DEVICE_ORDER to pin it with. When the list cannot be
// resolved the flags stay as they were, which is what shipped and what
// issue #4 was tested on.
split := fmt.Sprintf("-sm layer --main-gpu %d", prof.MainGpu)
ts := prof.TensorSplit
if devs, order := DeviceFlagFor(s.ServerExe, s.GpuSetOrEmpty(), prof.MainGpu); devs != "" && !pinsOwnSplit(ov) {
ts = PermuteSplit(ts, order)
split = fmt.Sprintf("--device %s -sm layer --main-gpu %d", devs, len(order)-1)
}
lines = append(lines,
split,
fmt.Sprintf("--tensor-split %s", FormatSplit(ts)),
)
}
// Vision twin loads the projector for image input. --no-mmproj-offload keeps
// the CLIP tower on the CPU: no VRAM for the projector (the sizer already
// priced the twin that way), slower image encode, same token throughput.
// ov is the EFFECTIVE override here (model-wide with the vision variant's
// knobs merged in), so an explicit mmprojFile - from either level - is
// already resolved by the time the argv is rendered.
mmprojFileOv := ""
if ov != nil {
mmprojFileOv = ov.MmprojFile
}
mmprojPath, _ := mmprojFor(row, mmprojFileOv)
if prof.Vision && mmprojPath != "" {
lines = append(lines, fmt.Sprintf("--mmproj %s", strings.ReplaceAll(mmprojPath, "\\", "/")))
if prof.CpuMmproj {
lines = append(lines, "--no-mmproj-offload")
}
}
// spec is a "+"-joined list of backends (draft-mtp / draft-dflash / ngram-map-k4v
// / ngram-mod / none); llama-server accepts them chained, so emit one --spec-type
// each and the sub-knobs for whichever backends are present.
for _, st := range strings.Split(spec, "+") {
lines = append(lines, fmt.Sprintf("--spec-type %s", st))
}
// draft-mtp and draft-dflash both drive off a draft model checked each step
// against the main model, so share the -md/-ngld wiring; only the sane
// default draft length differs. DFlash proposes a whole diffusion block per
// pass so it tolerates a longer chain than single-token MTP, but 5 is the
// measured sweet spot on Qwen3.6-35B-A3B (own n-max sweep, 3/4/5/6): reasoning
// tg jumps ~15% at n=5 vs n=3/4 and ties n=6, while n=5 also edges out n=6 on
// creative tg — higher (6+, 12/15) over-drafts, accept rate falls faster than
// length rises, so the wasted GPU verify compute makes TG worse.
if isDraftSpec := specHas(spec, "draft-mtp") || specHas(spec, "draft-dflash"); isDraftSpec {
nmax := 2
if specHas(spec, "draft-dflash") {
nmax = 5
}
if ov != nil && ov.SpecDraftNMax > 0 {
nmax = ov.SpecDraftNMax
}
lines = append(lines, fmt.Sprintf("--spec-draft-n-max %d", nmax))
// A separate draft file (MTP sidecar like Gemma-4, or any DFlash drafter —
// DFlash has no baked-in variant, it's always separate): baked-in MTP
// models need no -md. Only attach the file when its kind matches the active
// draft backend — a draft-dflash gguf must NOT be loaded as a draft-mtp
// draft (arch mismatch: the sidecar is arch=dflash) and vice versa; a
// baked-in MTP layer (DraftKind=="") correctly emits no -md. Pin the draft
// fully to VRAM (-ngld 99): spec decode is serial (draft proposes, main
// verifies each step), so a CPU-resident draft stalls the GPU main model
// every round. DraftSizeGB charges this via draftOverheadGB.
mdMatches := (specHas(spec, "draft-mtp") && row.DraftKind == "mtp") ||
(specHas(spec, "draft-dflash") && row.DraftKind == "dflash")
if row.DraftPath != "" && mdMatches {
lines = append(lines, fmt.Sprintf("-md %s", strings.ReplaceAll(row.DraftPath, "\\", "/")))
lines = append(lines, "-ngld 99")
}
// Draft KV quant (-ctkd/-ctvd), defaulted to the model's OWN -ctk/-ctv.
// This is not "draft models only": a baked-in MTP head has no -md and
// still gets a full second context whose cache type comes from this same
// field, and llama's default for it is f16 no matter what -ctk says. On
// Qwen3.8-27B at 160k that lone nextn layer was paying 0.61 GB of f16 KV
// while the main cache ran q8_0; matching them halves it for free. Flash
// attention is global, so the same fa that licenses a quantized main KV
// covers the draft's. An override still wins on either side.
ovDraftK, ovDraftV := "", ""
if ov != nil {
ovDraftK, ovDraftV = ov.KvKDraft, ov.KvVDraft
}
dKvK, dKvV := draftKvPair(ovDraftK, ovDraftV, kvK, kvV)
lines = append(lines, fmt.Sprintf("-ctkd %s -ctvd %s", dKvK, dKvV))
}
if specHas(spec, "ngram-map-k4v") && ov != nil {
if ov.SpecDefault {
lines = append(lines, "--spec-default")
}
if ov.SpecNgramSizeN > 0 {
lines = append(lines, fmt.Sprintf("--spec-ngram-map-k4v-size-n %d", ov.SpecNgramSizeN))
}
if ov.SpecNgramSizeM > 0 {
lines = append(lines, fmt.Sprintf("--spec-ngram-map-k4v-size-m %d", ov.SpecNgramSizeM))
}
if ov.SpecNgramMinHits > 0 {
lines = append(lines, fmt.Sprintf("--spec-ngram-map-k4v-min-hits %d", ov.SpecNgramMinHits))
}
}
lines = append(lines, fmt.Sprintf("--jinja --reasoning-format %s%s", rfmt, reasoningFlag))
// Cap thinking tokens when a budget is set and reasoning is on.
if ov != nil && ov.ReasoningBudget > 0 && reason != "off" {
lines = append(lines, fmt.Sprintf("--reasoning-budget %d", ov.ReasoningBudget))
}
// Keep prior-turn <think> in history (Qwen3.6+); pointless when thinking is
// off. --reasoning-preserve is llama.cpp's own knob for it — it sets the
// template's preserve_thinking variable itself, and llama-server logs a nag at
// init when a template supports preservation and the flag is absent. Verified
// equivalent to the old --chat-template-kwargs '{"preserve_thinking":true}' on
// both template wordings (3.8's preserve-by-default and 3.6's opt-in), and
// safe on a template that does not support it at all: llama-server warns
// "--reasoning-preserve has no effect" and carries on.
// Default ON (nil), so a model with no override still keeps its thinking; only
// an explicit false strips. See Override.PreserveThinking.
if reason != "off" && (ov == nil || ov.PreserveThinking == nil || *ov.PreserveThinking) {
lines = append(lines, "--reasoning-preserve")
}
// Always emit so runtime matches our reserve (else llama-server defaults to 32).
ckptConstGB := GetKvCostModel(meta, kvK, kvV).ConstGB
ckptRecurrent := meta.FullAttnInterval > 0
ckpts := effectiveCtxCheckpoints(prof, defaultCtxCheckpoints(ckptConstGB, ckptRecurrent))
lines = append(lines, fmt.Sprintf("--ctx-checkpoints %d", ckpts))
// Spacing is charged per checkpoint by checkpointReserveGB, so emit whatever it
// assumed — ALWAYS, never "only when it differs from a default". llama-server's
// own default is 8192 (LlamaDefaultCheckpointMinStep), so omitting the flag
// launched every recurrent and plain-attention model 32x wider than the reserve
// charged for. An ov-only pin (a preview profile built without the field) still
// wins over the arch default.
if ckpts > 0 {
stepProf := prof
if stepProf.CheckpointMinStep == 0 && ov != nil {
stepProf.CheckpointMinStep = ov.CheckpointMinStep
}
lines = append(lines, fmt.Sprintf("-cms %d", effectiveCheckpointMinStep(stepProf, ckptConstGB, ckptRecurrent)))
}
// DRY sampler: defaults to settings.DryDefault (nil => off) and a per-model
// ov.Dry wins (nil => the default, false => off, true => on). Values default
// to 0.8 / 1.75 / 3; a non-zero override replaces each independently.
dryOn := s.DryDefault != nil && *s.DryDefault
if ov != nil && ov.Dry != nil {
dryOn = *ov.Dry
}
if dryOn {
mult, base, allow := 0.8, 1.75, 3
if ov != nil {
if ov.DryMultiplier != 0 {
mult = ov.DryMultiplier
}
if ov.DryBase != 0 {
base = ov.DryBase
}
if ov.DryAllowedLength != 0 {
allow = ov.DryAllowedLength
}
}
lines = append(lines, fmt.Sprintf("--dry-multiplier %g --dry-base %g --dry-allowed-length %d", mult, base, allow))
}
// Server-side sampler defaults (arch baseline + per-model override). These
// apply only to requests that omit the field, so a client stays in control of
// anything it actually sends.
lines = append(lines, samplerLines(meta, ov)...)
lines = append(lines, fmt.Sprintf("-t %d%s", threads, cpuMoeFlag))
// Slot KV persistence: expose llama-server's save/restore slot endpoints. Path
// is quoted (it lives under a per-user dir that may contain spaces) and matches
// the emitted slotCache.path the server's LRU uses. Per-model SlotCache is
// OPT-IN (nil => off): the master switch only arms the feature, and a model
// joins with slotCache:true from its config editor.
if s.SlotCache.Enable && ov != nil && ov.SlotCache != nil && *ov.SlotCache {
lines = append(lines, fmt.Sprintf("--slot-save-path %q", slotKvPath(s.SlotCache)))
}
// Chat template: ONLY what the user asked for. A model's baked-in gguf
// template is otherwise left alone: chat templates are the user's to manage,
// and substituting one silently drops whatever the baked template supports
// that the replacement does not. Quoted: user paths routinely contain spaces.
if ov != nil && strings.TrimSpace(ov.ChatTemplateFile) != "" {
lines = append(lines, "--chat-template-file "+cmdPath(ov.ChatTemplateFile))
}
// Advanced / power-user knobs. Each is gated on a set override field (zero/
// empty => omit), so a model with none of them set emits exactly as before.
if ov != nil {
if ov.ThreadsBatch > 0 {
lines = append(lines, fmt.Sprintf("-tb %d", ov.ThreadsBatch))
}
if ov.Prio > 0 {
lines = append(lines, fmt.Sprintf("--prio %d", ov.Prio))
}
if ov.NoOpOffload {
lines = append(lines, "--no-op-offload")
}
if ov.NoRepack {
lines = append(lines, "--no-repack")
}
if ov.CacheReuse > 0 {
lines = append(lines, fmt.Sprintf("--cache-reuse %d", ov.CacheReuse))
}
if ov.CacheRamMB > 0 {
lines = append(lines, fmt.Sprintf("-cram %d", ov.CacheRamMB))
}
if ov.LogVerbosity > 0 {
lines = append(lines, fmt.Sprintf("-lv %d", ov.LogVerbosity))
}
switch ov.CacheIdleSlots {
case "on":
lines = append(lines, "--cache-idle-slots")
case "off":
lines = append(lines, "--no-cache-idle-slots")
}
if ov.SwaFull {
lines = append(lines, "--swa-full")
}
switch ov.ContextShift {
case "on":
lines = append(lines, "--context-shift")
case "off":
lines = append(lines, "--no-context-shift")
}
if ov.SpecDraftNMin > 0 {
lines = append(lines, fmt.Sprintf("--spec-draft-n-min %d", ov.SpecDraftNMin))
}
if ov.SlotPromptSimilarity > 0 {
lines = append(lines, fmt.Sprintf("-sps %g", ov.SlotPromptSimilarity))
}
if ov.RopeScaling != "" {
lines = append(lines, fmt.Sprintf("--rope-scaling %s", ov.RopeScaling))
}
if ov.RopeScale > 0 {
lines = append(lines, fmt.Sprintf("--rope-scale %g", ov.RopeScale))
} else if ropeExtends(ov.RopeScaling) {
// Derive the factor from the ctx actually chosen. Emitting the type
// alone leaves llama.cpp on the model's own factor (1.0 unless the
// publisher fine-tuned for extension), i.e. a longer window over
// untrained positions.
if f := ropeFactor(meta, ctx); f > 0 {
lines = append(lines, fmt.Sprintf("--rope-scale %g", f))
}
}
if ov.RopeFreqBase > 0 {
lines = append(lines, fmt.Sprintf("--rope-freq-base %g", ov.RopeFreqBase))
}
if ov.YarnOrigCtx > 0 {
lines = append(lines, fmt.Sprintf("--yarn-orig-ctx %d", ov.YarnOrigCtx))
}
if ov.SplitMode != "" {
lines = append(lines, fmt.Sprintf("-sm %s", ov.SplitMode))
}
if ov.TensorSplit != "" {
lines = append(lines, fmt.Sprintf("-ts %s", ov.TensorSplit))
}
if ov.MainGpu > 0 {
lines = append(lines, fmt.Sprintf("-mg %d", ov.MainGpu))
}
if ov.OverrideTensor != "" {
lines = append(lines, fmt.Sprintf("-ot %s", ov.OverrideTensor))
}
// LoRA adapters, one flag each. Last so a user's extraArgs (appended
// after this whole block) can still add more: --lora is Additive in the
// flag table, so a custom occurrence does NOT drop these.
lines = append(lines, llmLoraLines(s, ov, row.FullPath)...)
}
return lines
}
// RenderSoloCmd previews the full launch command for a candidate override,
// reusing the solo-profile sizer so the editor's launch-parameters box matches
// what a save would emit. Returns the command on one line (with `${PORT}` intact).
func RenderSoloCmd(s Settings, meta Metadata, row GgufRow, ov Override) (string, error) {
cc, err := RenderSoloCmdLayers(s, meta, row, ov)
if err != nil {
return "", err
}
return cc.Effective, nil
}
// plainCmd wraps a rendered command that has no composed layers: the non-llama
// classes keep their emitter's own custom-text handling until their forms get
// the same treatment (see ui-svelte/launch-args.md, non-goals).
func plainCmd(cmd string) (ComposedCmd, error) {
return ComposedCmd{Generated: cmd, Effective: cmd}, nil
}
// RenderSoloCmdLayers is RenderSoloCmd with the layered view the config editor
// needs: the generated command, the effective one a save would produce, and
// per-token provenance for rendering it.
func RenderSoloCmdLayers(s Settings, meta Metadata, row GgufRow, ov Override) (ComposedCmd, error) {
// SAM models render a sam3_server command (no metadata; matched by IsSam).
if row.IsSam {
return plainCmd(strings.Join(samCmdLines(s, row, &ov), " "))
}
// TRELLIS.2 packages render a trellis2-server command (a directory, no
// metadata, matched by IsTrellis).
if row.IsTrellis {
lines, err := trellisCmdLines(s, row, &ov)
if err != nil {
return ComposedCmd{}, err
}
return plainCmd(strings.Join(lines, " "))
}
// HF folders render a vllm command or nothing: the same refusal emitHFModel
// makes when no vllm backend is registered. meta is the caller's
// ReadHFMetadata, since the folder has no gguf header.
if row.IsHF {
be := resolveBackendPreferring(s, &ov, "llm", "vllm")
if !strings.EqualFold(be.Kind, "vllm") {
return ComposedCmd{}, fmt.Errorf("%s is a safetensors model folder and needs a vllm backend; llama.cpp loads gguf only", row.FileName)
}
return plainCmd(strings.Join(vllmCmdLines(s, row, &ov, row.FullPath, be, meta), " "))
}
// Video DiTs render the same sd-server command shape as an image model, but
// have to be tested FIRST: their arch (when they declare one at all) still
// reads as an image arch, and the video branch is what adds --audio-vae and
// the frames/fps profile.
if vid := videoInfoFrom(meta); vid.is() {
// The sd path composes inside imageCmdLines, so the layered view comes
// back already carrying custom/generated/effective and per-token
// provenance: the editor's panes light up for a diffusion model exactly
// as they do for llama-server, instead of falling back to plainCmd's
// bare string, which owns no knobs and badges nothing on the form.
cmd, _, _, _, _ := imageCmdLines(s, row, &ov, effectiveImageArch(meta), row.FullPath, meta.CondHidden, vid)
cmd.Issues = validateComposedWith(SdFlags, cmd, imageExe(s, &ov))
return cmd, nil
}
// Diffusion models render an sd-server command, not a llama-server one.
if imgArch := effectiveImageArch(meta); isImageArch(imgArch) {
cmd, _, _, _, _ := imageCmdLines(s, row, &ov, imgArch, row.FullPath, meta.CondHidden, videoInfo{})
cmd.Issues = validateComposedWith(SdFlags, cmd, imageExe(s, &ov))
return cmd, nil
}
// Decision models render their /v1/systemone command, ahead of the embedder
// check for the same reason as in emitModel.
if IsDecisionModel(meta) {
return plainCmd(strings.Join(decisionCmdLines(s, row, &ov, meta), " "))
}
// Embedders render a minimal --embeddings command (no KV/spec sizing).
if IsEmbeddingModel(meta) {
return plainCmd(strings.Join(embeddingCmdLines(s, row, &ov, meta), " "))
}
// audio.cpp models render an audiocpp_server command. Tested ahead of the
// two legacy speech engines: an audio.cpp gguf names its own family in the
// header, so the match is exact and neither of the others could read it.
if IsAudioCppModel(meta) {
return plainCmd(strings.Join(audioCppCmdLines(s, row, &ov, audioCppClass(meta.AudioFamily)), " "))
}
// Speech models render a tts-server command: qwentts (talker + paired codec)
// or TTS.cpp (--model-path), whichever engine the model resolves to.
if IsTTSModel(meta, row.FileName) {
return plainCmd(strings.Join(ttsCmdLines(s, row, &ov, meta), " "))
}
// Parakeet ASR models render a parakeet-server command (model + port only).
if IsASRModel(meta, row.FileName) {
return plainCmd(strings.Join(asrCmdLines(s, row, &ov), " "))
}
// vllm-backed LLMs render a vllm command; a chosen llama build swaps the exe.
be := resolveBackend(s, &ov, "llm")
if strings.EqualFold(be.Kind, "vllm") {
// Same refusal the emitter makes: vllm loads only the shard it is given,
// so there is no command to preview for a split set.
if isSplitGguf(row) {
return ComposedCmd{}, fmt.Errorf("vllm cannot load split gguf shards; merge %s with llama-gguf-split --merge or pick a llama backend", row.FileName)
}
return plainCmd(strings.Join(vllmCmdLines(s, row, &ov, row.FullPath, be, meta), " "))
}
if be.Exe != "" {
s.ServerExe = be.Exe
}
// Custom launch arguments size the preview, not just its argv (pins.go): a
// pinned -c has to show up in the sized window and the memory numbers, or the
// box and the panel would disagree about the same command. ov is a local copy.
if pins := PinsFromArgs(ov.CustomArgsText()); !pins.Empty() {
ov = pins.ApplyToOverride(ov)
}
// Same default as emitModel — this used to hardcode q8_0 with no MoE branch,
// so the editor previewed a KV type the emitted config never used.
soloTarget := s.TargetVramGB
if ov.VramTargetGB > 0 {
soloTarget = ov.VramTargetGB
}
soloSpec := effectiveSpec(meta, &ov, row.DraftKind)
soloDraftGB := matchedDraftSizeGB(soloSpec, row.DraftKind, row.DraftSizeGB)
kvDef := defaultKvQuant(s, meta, soloTarget, s.VramOverheadGB+draftOverheadGB(soloSpec, soloDraftGB))
kvK, kvV := kvDef, kvDef
if ov.KvK != "" {
kvK = ov.KvK
}
if ov.KvV != "" {
kvV = ov.KvV
}
if !ValidKvPair(kvK, kvV) {
kvK, kvV = kvDef, kvDef
}
perTokGB, kvConstGB := 0.0, 0.0
if m := GetKvCostModel(meta, kvK, kvV); m.OK {
perTokGB, kvConstGB = m.SlopeGB, m.ConstGB
}
// Rope scaling lifts the trained-length ceiling; without it this is nativeCtx.
modelMax := ropeCeiling(meta, ov.RopeScaling, ov.Ctx)
target := s.TargetVramGB
if ov.VramTargetGB > 0 {
target = ov.VramTargetGB
}
specOh := draftOverheadGB(soloSpec, soloDraftGB)
prof := profile{
Name: "preview",
Target: target,
Overhead: s.VramOverheadGB + specOh,
Ctx: ov.Ctx,
CpuOffload: ov.CpuOffload,
KvK: kvK,
KvV: kvV,
IsLong: ov.Ctx >= longCtxThreshold,
CtxCheckpoints: ov.CtxCheckpoints,
CheckpointMinStep: ov.CheckpointMinStep,
}
// The pin's ctx/placement belong on the profile (per-slot window, exact
// rounding): the override copy above only carried the model-wide scalars.
pinsForProfile(&prof, ov, meta)
soloUb := effectiveUb(meta, prof, &ov, prof.Ctx, s.TargetVramGB)
// ...plus the second llama_context a baked-in MTP drafter runs, which
// allocates a graph of its own at that same ub (see mtpDraftComputeGB).
prof.Overhead += computeBufferGB(meta, soloUb, s.ComputeBufFactor) +
mtpDraftComputeGB(meta, soloSpec, soloDraftGB, soloUb, s.ComputeBufFactor)
// The baked-in MTP drafter's own KV scales with the window, so it belongs in
// the slope the sizer solves ctx against (see mtpDraftSlopeFor).
sdKvK, sdKvV := draftKvPair(ov.KvKDraft, ov.KvVDraft, kvK, kvV)
prof.DraftSlopeGB = mtpDraftSlopeFor(meta, soloSpec, sdKvK, sdKvV, soloDraftGB)
// Multi-slot: every slot carries its own window over the one shared -c pool,
// so the sizer solves the per-slot ctx against N x the KV — same as emitModel.
// Sizing one slot here previewed a command that over-committed the pool.
if slots := profileParallel(prof, ov); slots > 1 {
perTokGB *= float64(slots)
kvConstGB *= float64(slots)
prof.DraftSlopeGB *= float64(slots)
}
ctx, plan, kvReserve, _, err := sizeProfile(meta, s, prof, perTokGB, kvConstGB, modelMax, ov.KvInRam)
if err != nil {
return ComposedCmd{}, err
}
ngl, ncpuMoe := forceLowActiveMoE(meta, plan, prof, kvReserve)
if ov.CpuOffload > 0 || prof.CpuOffloadSet {
ngl, ncpuMoe = applyForcedOffload(meta, prof.CpuOffload)
}
lines := buildCmdLines(s, meta, row, prof, ctx, ngl, ncpuMoe, kvK, kvV, ov.KvInRam, &ov)
cc, err := ComposeCmd(lines, ov.CustomArgsText())
if err != nil {
return cc, err
}
cc.Issues = validateComposed(cc, be.Exe)
return cc, nil
}
// specHas reports whether a "+"-joined spec list contains backend b.
func specHas(spec, b string) bool {
for _, s := range strings.Split(spec, "+") {
if s == b {
return true
}
}
return false
}
// formatCtxTag renders a short ctx tag: 8192->"8k", 131072->"128k", 1048576->"1m".
// effectiveSpec resolves the spec-type: MTP-capable models (baked-in nextn
// layer or a paired mtp-*.gguf sidecar) default to draft-mtp+ngram-mod (the
// chain beats mtp alone), everything else to ngram-mod. A paired DFlash drafter
// is never auto-selected — see below.
// An explicit override spec wins (set spec: "ngram-mod" to force ngram on a
// drafted model, or spec: "draft-dflash" to opt into DFlash).
//
// A DFlash sidecar sitting in the model's dir is a deliberate choice (the user
// downloaded it) but is NOT auto-selected: it wins a short flat-prompt bench
// (GPU-bound: +17% vs mtp) but its resident draft weights + own full-context KV
// crowd an already-tight VRAM budget over a long real session, cratering to a
// GPU-mem-oversubscription cliff mtp never hits (mtp is a baked-in head with no
// separate weights or KV). Confirmed in real multi-turn agent use on
// Qwen3.6-35B-A3B-100k: switching qwen3.6-35b-a3b-ud-q4_k_s-100k off dflash back
// to mtp fixed a production slowdown. An explicit `spec: draft-dflash` override
// still works — this only removes it as the automatic pick.
func effectiveSpec(meta Metadata, ov *Override, draftKind string) string {
// Explicit override always wins.
if ov != nil && ov.Spec != "" {
return ov.Spec
}
switch {
case meta.IsMTP || draftKind == "mtp":
// draft-mtp CHAINED with model-less ngram-mod: benched better than mtp
// alone. The MTP head drafts short high-confidence continuations; ngram-mod
// fills the gaps mtp misses (verbatim repeats — code, quoted context, tool
// echoes) at zero extra VRAM (no draft weights/KV). llama-server verifies
// both proposals each step, so the union lifts accept rate over either solo.
return "draft-mtp+ngram-mod"
}
return "ngram-mod"
}