internal/hub/CLAUDE.md

Rama main · réplica de lectura del repositorio original.

# internal/hub

## Purpose

Model browser + downloader: search a model hub from inside quartermaster, read a repo's file list, and pull a quant into the models folder, where autogen discovers it like any hand-copied GGUF. Before this, acquiring a model was entirely out-of-band (browser → manual download → drop in the folder).

Two halves, deliberately separable: a **Source adapter** (what a hub knows) and a **download Manager** (how bytes get to disk). The interface exists so a second hub — ModelScope, phase 2 — is an adapter, not a fork of the downloader.

## Key files

| File | Role |
|---|---|
| `hub.go` | Package doc, the `Source` interface, wire types (`Model`, `File`, `ModelDetail`, `Query`, `Page`), and the filename parsing every adapter shares: `classify` (quant tag, shard number, **group key**) and `IsModelFile`. |
| `hf.go` | The Hugging Face adapter. `Search` (`/api/models?filter=gguf`), `Detail` (`/api/models/{id}?blobs=true` — `blobs=true` is what makes siblings carry a size), `readme` (raw README.md), `FileURL` (`/resolve/main/…`), `CheckURL` (host pinning), `Authorize` (bearer token). Plus `hubHTTPError`/`AuthError` and the repo-id / repo-path validators. 10-minute response cache, bounded at 128 entries. |
| `header.go` | `FetchRange`/`FetchHead` — Range-fetch a window of a repo file (its GGUF header) without downloading it, so a candidate can be sized before it is committed to. Bytes only: parsing lives in `internal/server`, since this package must not depend on `internal/autogen`. |
| `download.go` | The `Manager`: job list, admission, the pause/resume/cancel verbs, and the resumable transfer (`fetchFile` retry loop → `fetchOnce` single attempt). `RepoDirName` owns the on-disk layout; `SweepPartials` clears orphaned `.part` files; the `.quartermaster-download.json` journal + `Restore` bring in-flight jobs back after a restart. |

## Important types & functions

- `Source` — `ID`/`Name`/`Search`/`Detail`/`FileURL`/`CheckURL`/`Authorize`. Implementations must be concurrency-safe.
- `Manager` (`NewManager(modelsRoot func() string, log, sources...)`) — `Start`, `Pause`, `Resume`, `Cancel`, `Jobs`, `Job`, `Source`, `Sources`. `OnComplete` is the caller's hook to regenerate + hot-reload. `launchLocked` is the shared spawn used by both `Start` and `Resume`, so a resumed job is the same job (same id, same files) rather than a new one pointed at the same repo.
- `FetchRange(ctx, src, repo, path, off, n)` / `FetchHead` (= `off 0`) / `MaxHeaderBytes` (`header.go`) — the pre-download sizing primitive. Runs the same two validators and the same **per-redirect-hop** host pinning as the download path (a `/resolve/` URL is answered by a CDN redirect), accepts a `206` **or** a `200` (a server that ignored `Range`), and reads through an `io.LimitReader` so that 200 can't turn a size probe into a 40 GB read. A short read is **not** an error: only the caller parsing the prefix can tell whether what arrived was enough, which is what lets `internal/server` extend the prefix on `io.ErrUnexpectedEOF`. **The offset is why that extension is cheap** — a caller that guessed short fetches the remainder rather than re-requesting from zero, and a real header is 6-8 MB (the tokenizer vocab dominates it), close enough to any sensible first guess that re-fetching would cost more than the header itself. `off > 0` on a `200` **discards** that many bytes before reading, since the caller is appending to a prefix it already holds and must not be handed the head of the file under a false offset.
- `Job` / `JobFile` — phase (`queued`→`checking`→`downloading`→`registering`→`done`|`error`|`canceled`, plus `paused` off to the side), byte counters, `Gated`. Copied out under the lock, never handed out by pointer (same contract as `internal/backends.Job`).
- `LocalFiles(repo)` — repo-relative path → `LocalFile{Size, OID}` for everything already under that repo's folder, which is what puts the **"downloaded"** badge on a picker row (`handleAPIHubModel` marks `File.Local` with it; `File.Local` is filled in by the HTTP layer, not an adapter, because the disk is a property of this installation, not of the hub). A `.part` is deliberately excluded — half a file is not a model, and that row should stay a download button. The manifest is excluded for the same reason it is dotted: it is our bookkeeping, not a repo file, and the download journal is skipped alongside it. Sizes rather than a bool because only the caller knows what the hub said the file weighs: a short file is a truncated copy, not a finished download. `OID` is the recorded identity and is returned **only while the entry still matches the size on disk** — a file replaced behind our back has an id describing bytes that are gone, and reporting it would vouch for the wrong thing. A missing folder is an empty map, never an error.
- **The repo manifest** (`.quartermaster-hub.json`, `manifest`/`manifestEntry`, `readManifest`/`recordManifest`/`forgetManifest`) — per repo-relative path, the hub's content id **at the moment we fetched it**, written next to the bytes as each file lands. It exists because *a filename is not an identity*: publishers re-upload a quant in place — a fixed chat template, a corrected tokenizer, a rebuild against a newer converter — under the same name, at a byte count that is often within rounding of the old one or exactly on it. Before this, `have >= want` read the new revision as "already downloaded" and the **only** way to get it was to rename or delete the file by hand. It is **trust-on-first-write on purpose**: verifying on-disk bytes against the id means hashing tens of gigabytes to render a picker row, so the manifest describes what this downloader put there, not what is there now. Writing it is best-effort — the bytes are on disk and usable, and failing a finished 40 GB download over a bookkeeping file would be absurd; the cost of a miss is only a fall back to the size check.
- `haveCurrent(dir, f, man, force)` — the one place that decides whether a file on disk may be skipped, used by **both** passes of `run()` (the free-disk precheck and the transfer loop) so they cannot disagree about which files the job is fetching. Complete **and** current, or refetch. When either side states no id the size test stands unchanged: without an id we cannot tell "the same file" from "a different file of the same length", and re-pulling 40 GB on a guess is the worse error. `force` short-circuits it.
- **The download journal** (`.quartermaster-download.json`, `downloadRecord`/`downloadJournal`, `readJournal`/`writeJournal`/`recordDownload`/`clearDownload`, `Restore`/`restoreJob`) — per repo folder, the jobs that have **not** finished. The job list itself is deliberately in memory, but a `.part` is invisible to the UI: an interrupted transfer left a resumable stub that no page ever mentioned again, and re-picking the file from Browse was the only way to continue it. `Start` writes the record **before** the job can run (a tiny file could otherwise finish and clear the record first, and a record written after that clear would resurrect a finished download), `run()` clears it on `done` and `discard` on cancel, and **pause and error keep it** — both left bytes on disk. `Restore(maxPartialAge)`, called once from `StartHubDownloads` (main, after the autogen admin and with it the models root exist), walks the models tree for journals and re-registers each record as a **paused** job, recomputing every file's `Done`/`Skipped`/`Replaced` from disk exactly as `run()` would judge it — no byte counter is trusted; the disk is what survived. Records with no bytes anywhere are cleared, ones whose files are already complete are dropped as done, and the age gate passed in is the same one `SweepPartials` uses so a `.part` the sweep will delete is not counted as progress. Progress is deliberately not persisted: a journal that stopped being written at the moment of a kill is still correct, because the `.part` sizes are the truth. A repo may hold several records (two quants, or a model and its projector, share one folder), which is why the file is a list, and a job that is only *queued* has a record too — nothing is downloaded yet, so `Restore` clears it.
- `File.Stale` / `StartRequest.Force` / `Job.Force` — the two halves of the answer. `Stale` is the case the server **can** see (both ids known, and they differ): the picker renders "update available" and a plain download refetches exactly the shards whose id moved. `Force` is the case it **cannot** — a file swapped outside quartermaster, or one downloaded before the manifest existed — and is the UI's always-available re-download.
- `RepoDirName` — the on-disk layout, **`<owner>/<name>`**, mirroring the hub's own namespacing (`unsloth/Qwen3-8B-GGUF` → `…/Models/unsloth/Qwen3-8B-GGUF`) rather than the flat `owner__name` it used to emit. Every segment goes through `sanitizeSeg` **before** it is joined, and an all-dots segment (`.`, `..`) is rewritten to dashes: nesting is what turned a `..` owner into a directory climb, where the old `__` glue had made it inert. No migration — repos already on disk keep their flat folder and still load.
- `ParamsB` / `WithinParams` — the size filter behind `Query.MaxParamsB` (the Max size knob in the Browse page's Filters menu, 120B by default). A **name** parser, because the list endpoint returns no parameter count and the real number is in a GGUF header we'd have to download the model to read. Two rules: the **first** size token wins (a MoE is named `…-235B-A22B`, total then active — taking the last would read a 235B model as 22B, the exact mistake the filter exists to prevent), and an unparseable name is **unknown, so it is kept** — hiding a repo because its author skipped the convention is worse than showing one whose own file sizes will say it's too big. `capParams` in `hf.go` applies it to the response and the search **over-fetches 3x** first, since HF has no size filter of its own and a capped page would otherwise come back nearly empty. The cache stores what the hub said, unfiltered, so a capped search can't poison an uncapped one.
- `Page` / `Query.Skip` — search is **paged, not capped**: the browser loads more as the list is scrolled, so `Search` returns `{Models, NextSkip, HasMore}` and the caller pages by `NextSkip`. That number counts the **hub's own rows**, not the surviving ones — the adapter over-fetches and then filters (size cap, age), so advancing by `len(Models)` would re-request rows already shown or skip rows never shown. `hfPage` deliberately does **not** trim to `Limit` for the same reason: everything that survived the caps is returned, and `Limit` is only how much is asked of the hub. `HasMore` is short-page detection against what was actually fetched. HF's own `skip=` is stable enough to page with, but not enough to guarantee no overlap, so the UI dedups by id.
- `Query.MaxAgeDays` / `capAge` (`hf.go`) — the **Trendy** filter (**off** by default in the browser: it answers "what came out recently", which is a question the user has to ask, and defaulting it on hid every established model from a search-by-name): keep only repos **created** within N days, which paired with the existing sort is "popular releases that are new". Judged on `Model.Created`, never `Updated` — `lastModified` moves for a README fix, so a two-year-old repo touched yesterday would read as a new release. A repo stating no creation date is KEPT, same posture as `MaxParamsB`. Applied client-of-the-hub-side for the same reason the size cap is: HF has no created-after filter, which is also why `MaxAgeDays` joins `MaxParamsB` in triggering the 3x over-fetch. **`createdAt` is not in the plain listing** — it takes `expand[]`, which is opt-in *per field* (asking for one drops all the others), so `hfExpand` has to name every field `Model` carries.
- `searchFilters` (`hf.go`) — maps `Query.Kind` (the Browse page's category tab, ids taken from the UI's own `MODEL_CATEGORIES` plus `any`) onto HF filter tags. **`any` is the Browse page's "All" tab** and drops the filter entirely: every other tab pairs `gguf` with one pipeline tag, so a repo tagged with something else — or with nothing — is invisible in all of them, and "All" is the escape hatch that finds it. Its cost is honest and accepted: it also lists repos carrying no file this project can load, which open with "this repo carries no GGUF files". The API **repeats and ANDs `filter`**, so each category is `gguf` + a pipeline tag (`text-to-image`, `text-to-speech`, `automatic-speech-recognition`, `feature-extraction`, and `mask-generation` for segment — that, not `image-segmentation`, is what the SAM/BiRefNet GGUF repos carry). `""`/`llm` is plain `gguf` at the hub, then **`keepText`** (in `hfPage`) drops rows that declare a non-LLM task (a non-chat `pipeline_tag`, or for an untagged repo a task in its plain tags). It is an exclusion on our side, not an include filter at the hub, because many popular LLM quant repos (unsloth's, lmstudio-community's) carry no `pipeline_tag` at all, and VLMs/omni models are `image-text-to-text`/`any-to-any`; without it audio.cpp, transcribe.cpp and Wan GGUF repos were listed as LLMs. `any` drops the filter entirely, anything else is passed through as a raw tag. `keepText` also drops anything tagged `decision-model`/`typed-decisions`/`system-one`, checked before the pipeline tag, because many decision repos declare `text-generation`. See `searchFilterSets` for `video`, `music` (`text-to-audio` + `audio-to-audio`, the latter for separators) and `decision`, the categories that need more than one query (decision models have no pipeline tag of their own, so that tab unions `gguf` + each of those three tags). Narrowing at the hub rather than locally is the whole point: a 30-row page filtered client-side would render most tabs empty. The filters are part of the cache key, since they are part of the encoded query.
- `searchFilterSets` / `searchOnce` / `sortModels` (`hf.go`) — a **union** of filter sets, because `video` is the one category no single HF query describes. The exact pipeline tags are split across the catalog (Wan's T2V repos carry `text-to-video`, its I2V repos `image-to-video`) and HF ANDs its filters, so either alone hides half the models; the broad `video` tag is not the answer either, since every video-*understanding* VLM carries it (GLM Flash, MiniCPM-V) while generation repos often do not (`lightx2v/Minimax-h3-Turbo` is tagged `image-to-video` and nothing else). So video asks twice, merges, dedups by id and re-sorts — concatenating two sorted lists does not preserve the order. **`gguf` is deliberately absent from the video sets**, as it is from `3d`: the useful video repos are LoRAs, fp8 variants and safetensors component sets, and requiring a GGUF hid every one. Every other category is a single set and unchanged. Each set is asked at the same `Skip` and cached on its own, and `NextSkip` advances by the **longest** sub-query, never by the merged length — advancing by the deduped total would skip rows in whichever set returned fewer. One set failing out of several is a thinner page, not a failed search.
- `AvatarSource` — an **optional** extra a `Source` may implement (`Avatar(ctx, author) (string, error)`), type-asserted in `handleAPIHubAvatar`. Kept out of `Source` on purpose: it is decoration, and a row without a picture falls back to a monogram. HF needs two lookups because it has orgs and users in one namespace with no endpoint covering both (`/api/organizations/{x}/avatar`, then `/api/users/{x}/avatar`) — and a **miss is cached like a hit**, or a namespace that is neither is asked twice for every row of every search.
- `SelectFiles` / `setManifest` — the per-repo listing policy, now **three** cases, not two. A repo with any quant file is a QUANT repo (only those files). A repo with none but **with a set manifest** (`pipeline.json`, `config.json`, a `ckpts/*.json`) is a COMPONENT SET, every file under one group key so the picker downloads it whole. A repo with **neither** is LOOSE WEIGHTS — a LoRA collection, a row of fp8/bf16 variants — where the files are *alternatives to each other*, not parts of each other, so each keeps its own group key. Before the manifest test, `lightx2v/Minimax-h3-Turbo` rendered as one row that downloaded all seventeen LoRAs.
- `MarkSelection` — what `Detail` actually returns: **the whole repo**, with `SelectFiles`' verdict *recorded* on each file rather than applied to it. A curated file keeps its group; everything else comes back `Aux`, grouped by its own path (one row each). The picker shows the curated set and puts the rest behind a "show all N files" toggle. The curated list is the right default view but was the wrong *whole* view: it is a set of rules about file names, and those are permanently behind the publishers — an image model's external VAE, a text encoder, a projector shipped as safetensors. Hiding them made the picker the reason a model could not be assembled, with no recourse but a browser. It also widens the **download allowlist**, since `Manager.Start` admits only paths the hub's own listing named.
- `classify` — turns a repo path into `Shard`/`Shards` + `Group` + `Projector`. **`Group` is load-bearing:** every shard of a multi-part GGUF shares one group key, which is what makes N files one logical download — and since the picker labels a row by its group's base name, that key doubles as the row's displayed name (the shared filename minus `-00001-of-00003`).
- **There is no quant parser, deliberately.** `File.Quant` and `quantRe` are gone; the picker pastes each file's whole name instead. Isolating the tag was shorter but wrong twice over: a miss was *silent* (a recipe marker or suffix the pattern didn't know mislabelled the row rather than failing), and two different files could reduce to the same tag — `mmproj-F16.gguf` rendered as a bare "F16", indistinguishable from the model's own F16 weights. The authority for what a file is has always been the GGUF header, which autogen reads once the file is on disk.
- `Projector` — an mmproj companion, matched by `projectorRe` (`mmproj`/`mm_proj`/`mm-proj` anywhere in the base name; publishers write it both as a prefix and as a `.mmproj.` infix). The whole filename now says so on its own, so the flag exists for the other three behaviours: the **badge**, sorting projectors **below** the models (as the smallest file in a repo it otherwise sorted first, where it read as the cheap option), and showing **"companion"** in place of a VRAM verdict — a projector is charged on top of whichever file you pick, so sizing it alone answers the wrong question.

## Gotchas / conventions

- **`internal/backends.download()` is NOT reused, on purpose.** It `os.Create`s and streams with no resume — fine for a 200 MB backend zip, useless for a 40 GB quant on a flaky line. This package re-implements the transfer with `Range` resume, retry with backoff, cancel, a free-disk precheck, a `.part` file and an atomic rename. The *job/phase/progress vocabulary* is deliberately mirrored from that package so the two managers read the same way.
- **The path is user-chosen; the host must not be.** `CheckURL` pins to `huggingface.co` / `*.hf.co` / `*.huggingface.co` over https only, and it is re-applied **on every redirect hop** (`CheckRedirect`) — a `/resolve/` request is answered with a CDN redirect, so checking only the initial URL would check the one URL that was never going to fetch the bytes. Same posture as `validAssetURL` in `internal/backends`.
- **Two validators, both before anything is used as a URL or a filename.** `validRepoID` (owner/name, hub-legal chars only) and `validRepoPath` (no absolute, no `..`, no backslash, no empty segment). `Start` additionally refuses any file **not present in the hub's own listing** — which both blocks a crafted path and guarantees every job file has a size, the denominator the disk check and the progress bar need.
- **A `.part` file is never renamed into place until the transfer succeeds.** Model discovery walks the models folder; a half-file with a `.gguf` name would be published as a model and fail at load.
- **A refetch discards the old `.part` and keeps the old file.** `JobFile.Replaced` marks a file being downloaded over one already on disk. Its `.part`, if there is one, was written for the revision being replaced, and resuming into it would splice two different files together — so it goes, and it does not count toward the free-disk precheck's "already downloaded". The **finished** file stays put until the rename at the end, which is what makes a canceled or failed update leave the working copy intact.
- **`discard` forgets what it deletes.** A manifest entry for a file that is no longer there would be judged against the next download of the same name, which is a different file entirely.
- **`ClearFinished` clears ROWS, not bytes.** The panel's Clear button: every terminal job
  (done, error, canceled) leaves the list, and running/paused/queued jobs stay. It deletes
  nothing on disk — cancel already discarded its partials and a finished job's files belong
  where they are, so the only job with anything left is an **errored** one, whose `.part` and
  journal record are kept on purpose. Destroying resumable bytes from a tidy-up button, when
  Cancel is right there and says what it does, would be the wrong default; the honest
  consequence is that such a job comes back as a paused row on the next `Restore`.
- **Pause keeps the bytes; cancel discards them.** They were one operation and it could only be one of the two. `Pause` stops the transfer and leaves every `.part` where it is (`paused` is a **non-terminal** phase — the job stays in the list and `Resume` re-launches it under the same id). `Cancel` deletes the job's partials **and any file this job itself completed**, then the (now empty) repo dir and its owner dir. Finishing one shard of a multi-part GGUF and leaving it behind publishes a broken model into the catalog, which is why cancel takes the whole set — but a `Skipped` file (already on disk before the job started) is never touched, since the job did not put it there. That difference is also what earns the UI's confirmation dialog.
- **A running job's files are deleted by `run()`, not by `Cancel`.** `Cancel` only records the intent (`stop[id]`) and cancels the context; the goroutine's `stopped()` does the removal once it has released the file handle. Windows refuses to delete an open file, so deleting from the caller's side would half-work and leave the `.part` behind.
- **Orphan sweep, age-gated.** `SweepPartials(root, maxAge, log)` walks the models tree and removes `.part` files not touched for `maxAge` (`hubPartialMaxAge`, 14 days, in `internal/server/hubapi.go`), run in the background at startup. The age gate is the point: a `.part` is resumable, so deleting a fresh one throws away real work — only a `.part` nothing has resumed in two weeks is garbage. A fresh partial is now also backed by a journal record, so it comes back as a paused row on the next start (`Restore` takes the same age gate, which is what keeps the two from disagreeing about which bytes count); the sweep is for partials nothing has touched in two weeks, journal or no journal.
- **A 401/403 is terminal, not retryable.** `fetchFile` breaks out of the retry loop on an `AuthError`, because retrying a license wall five times only delays telling the user to accept the license on the model page. The HTTP layer maps it to 403 with that wording.
- **The stall watchdog is not optional.** A TCP connection that goes quiet without erroring would hang the job forever, and "downloading, 41%" frozen for six hours is the worst failure mode this package has. `fetchOnce` cancels its own attempt after `stallTimeout` (2 min) of no bytes; the retry loop then resumes from the `.part`.
- **No client timeout.** A 40 GB file legitimately takes hours. What bounds a hung transfer is the per-attempt context, `ResponseHeaderTimeout`, and the watchdog — never `http.Client.Timeout`.
- **One job per repo, several repos in parallel; files within a job are sequential.** Three concurrent 20 GB pulls on one line finish no sooner and make every progress bar useless.
- **A second pick for a busy repo is QUEUED, not refused.** It used to be an error, which was wrong about the common case: two quants of one repo, or a model plus its projector, are the ordinary reason to click twice, and "come back in forty minutes and click again" is not an answer. `Manager.wait` is the admission FIFO (jobs sitting in `PhaseQueued`), and `pumpLocked` — called from the job goroutine's own defer, the only place a queued job is ever launched — starts whatever the freed repo was blocking. `Resume` admits the same way, so resuming into a busy repo queues rather than failing. `Cancel` on a still-queued job drops it from the queue and **discards nothing**: a `.part` under that repo belongs to whatever is running or paused there, not to a job that never opened a file. `Pause` on one parks it in `PhasePaused` for the same reason `Resume` is the single restart verb. Membership in `wait` (`dequeueLocked`), never `Phase == PhaseQueued`, is what tells a waiting job from a launched one — a launched job is briefly `queued` too, and canceling *that* one from the queue path would leave a live transfer behind a canceled row.
- **`ModelsRoot` is a func, resolved per call.** It is a config value a live reload can change, and a stale root would drop a 40 GB download in the wrong place.
- **The job goroutine recovers its own panics.** `launchLocked` wraps `run` in a `recover()` that logs the stack and lands the job in `error`. A bare goroutine has nothing above it that recovers — unlike an HTTP handler, where `net/http` turns a panic into a dropped connection — so a bug anywhere in the transfer, and especially in `OnComplete` (which regenerates the config and hot-reloads over a models tree that just changed), would take the whole process down at the exact moment a 40 GB download landed. The same code reached from a request survives as a 500; reached from here it killed quartermaster. Note this catches *panics only* — a `fatal error:` (concurrent map write, OOM) is still unrecoverable by design; what it has gained is a witness: the persistent log file (`-log-file`) re-points fd 2 at itself on Windows, so the runtime's fatal dump lands in the log instead of the console a GUI launch never had.
- **`OnComplete` failure does not fail the job.** The files *are* on disk; only the config refresh failed. The job ends `done` with an explanatory `Err`, because reporting `error` invites the user to re-download 40 GB that is already there.
- **Progress callbacks are throttled to 1 MiB.** Each one takes the manager lock.

## Not built yet (Feature 11 remainder)

- **A stored, UI-editable token.** `hubToken` (`internal/server/hubapi.go`) reads `HF_TOKEN` / `HUGGING_FACE_HUB_TOKEN` / `HUGGINGFACE_TOKEN` from the environment. The sidecar is the natural home for a managed one.
- **ModelScope adapter.** Mostly a CN mirror, so it earns its keep only for users HF is blocked for.
- **Image-model sibling sets: BUILT, outside this package.** A split DiT's companions (VAE, text encoder, tokenizer) are planned by `autogen.PlanImageCompanions` against the curated `internal/autogen/companions.json`, joined to the listing and the disk by `GET /api/hub/companions` (`internal/server/hubapi_companions.go`), and queued by the browser as ordinary jobs, one per companion repo. This package gained nothing for it on purpose: it stays autogen-free. Still open: video families, ERNIE (no verified ungated Ministral-3B source) and the vision mmproj an edit model wants.

## Connections

Depends on: `github.com/shirou/gopsutil/v4/disk` (free-space precheck) and the standard library only — deliberately no dependency on `internal/autogen`, so the downloader stays testable without a models tree.

Called by: `internal/server/hubapi.go` (the admin-gated `/api/hub/*` routes) — which supplies `ModelsRoot` from the autogen settings and wires `OnComplete` to `regenReload` — and `Server.StartHubDownloads`, which `main` calls once the autogen admin (and with it the models root) is attached, for the boot-time `Restore` plus the orphan sweep.