docs/maps/documents-and-collab.md
Rama main · réplica de lectura del repositorio original.
# Documents & Collaborative Editing Map
The dual-state storage model, the Yjs WebSocket protocol, and every path that
can write a document's content. This is the subsystem where "it looked right in
the editor" and "it is right in the database" most easily diverge.
---
## 1. Dual-state storage
A `logs` row carries **three** representations of the same document, plus one
generated column:
| Column | Type | Written by | Read by |
|---|---|---|---|
| `html_content` | `MEDIUMTEXT` | REST save, publish, restore, explicit WS save, GitHub pull | everything: rendering, export, search (via generated column), GitHub push |
| `markdown_content` | `MEDIUMTEXT` | REST save when the client sends it, WS save, GitHub pull/resolve/import | GitHub push (`github.js:1035-1043`), markdown-mode editing |
| `ydoc_state` | `LONGBLOB` | collab autosave and explicit save (`collab.js:126-132`) | collab session restore only (`collab.js:84-91`) |
| `plain_content` | generated `STORED` | MySQL, from `html_content` (`init.sql:293`) | the FULLTEXT index |
`plain_content` is `REGEXP_REPLACE(html_content, '<[^>]+>', '')`, computed by
MySQL on every write of `html_content`. It is the *only* thing search matches on
besides the title, which means **a document edited only over the CRDT is not
searchable until an explicit save writes `html_content`.**
### The invariant, and where it breaks
The intended invariant is that `html_content` and `ydoc_state` describe the same
document. It is maintained by convention, not enforced. The debounced autosave
is explicit about this (`collab.js:135-137`):
> HTML is NOT updated here. Clients send HTML during explicit save/publish.
So a 3-second-debounce autosave writes **only** the blob. Between autosave and
the next explicit save, `html_content` is stale, and every consumer of
`html_content` sees the older document: search, all four export formats,
GitHub push, and the version snapshot taken by REST publish
(`documents.js:216`, which reads `pg.html_content` straight from the row).
**`markdown_content` semantics:** `undefined` leaves the column untouched, an
explicit `null` clears it, a string replaces it (`documents.js:129-140`,
`collab.js:573`). Rich-text-mode saves send `null` to signal "HTML is now the
canonical source"; markdown-mode saves send the raw markdown. `localMarkdown()`
in the GitHub path (`github.js:1035-1043`) prefers `markdown_content` when
non-empty and otherwise round-trips the HTML through turndown, so a stale
`markdown_content` silently wins over fresher HTML on push.
### Every writer of document content
| Path | Writes | Clears `ydoc_state`? |
|---|---|---|
| `POST /api/save-document` (`documents.js:84`) | `html_content`, optionally `markdown_content` | no |
| `POST /api/document/:logId/publish` (`documents.js:173`) | `version`, `versions` row from existing `html_content` | no |
| `POST .../versions/:versionId/restore` (`documents.js:426`) | `html_content`, `version`, new `versions` row | **yes**, `NULL` (`documents.js:465`) |
| WS `{type:'save'}` (`collab.js:555-623`) | `html_content` if changed, `markdown_content`, `ydoc_state` | no |
| WS `{type:'publish'}` (`collab.js:654-731`) | `html_content`, `ydoc_state`, `version`, `versions` row | no |
| WS `{type:'title'}` (`collab.js:629-652`) | `title` only | no |
| collab autosave (`collab.js:139-149`), and the shutdown flush (`flushPendingSaves`, `collab.js:165-187`) | `ydoc_state` only | no |
| GitHub pull / resolve / overwrite (`github.js:1183-1192`, `1225-1234`, `1407-1416`) | `html_content`, `markdown_content` | **yes**, `NULL` |
| GitHub bulk import (`github.js:1597-1606`) | new `logs` row | n/a |
| First-boot seed (`bootstrapInstance`, `routes/admin.js`) | new `logs` row, the welcome document | n/a |
Clearing `ydoc_state` to `NULL` is the deliberate mechanism for "the HTML just
changed underneath the CRDT": on next connect the Y.Doc starts empty and the
first client's Tiptap `Collaboration` extension re-initialises it from the
REST-loaded HTML (`collab.js:92-94`). Any external writer of `html_content`
**must** null the blob or live editors will keep resurrecting the old content.
`POST /api/save-document` notably does not, which is correct only because it is
the editor's own save path.
## 2. The collab WebSocket
`services/collab.js`. One `Map` of `logId -> entry` (`collab.js:45`), where an
entry is `{ doc, conns, saveTimer, saving, cleanupTimer, lastSavedHtml, credits, logId }`
(`collab.js:71-81`).
### Constants (`collab.js:53-59`)
```
SAVE_DEBOUNCE_MS 3000 autosave debounce after any Y.Doc mutation
CLEANUP_DELAY_MS 30000 drop the in-memory doc this long after the last close
MAX_MESSAGE_SIZE 5 MiB ws maxPayload
MAX_HTML_SIZE 2 MiB largest HTML accepted in a save/publish message
MAX_CONNECTIONS_PER_USER 10 across all documents, tracked in userConnectionCounts
RATE_LIMIT_WINDOW_MS 1000 per-connection
RATE_LIMIT_MAX_MESSAGES 60 per window, applies to binary and text alike
```
### Session setup
`setupDocSession` (`collab.js:441-780`) runs after auth succeeds:
1. `getOrCreateDoc(logId)` (`collab.js:67-121`) loads `ydoc_state` and applies it
to a fresh `Y.Doc`, and caches `html_content` into `entry.lastSavedHtml`.
2. Cancels any pending cleanup timer (`collab.js:452-455`).
3. Assigns a cursor colour round-robin from a 10-entry palette
(`collab.js:297-307`). `colorIndex` is a module global, so colours are
assigned per-connection in arrival order, never per-user.
4. Sends **sync step 1** then **sync step 2** immediately
(`collab.js:469-479`). Step 2 is proactive so a joining client gets content
without a round trip.
5. Sends a JSON `{type:'sync', canWrite, user}` frame carrying permissions and
identity. **No HTML travels over this frame**, despite the older protocol
comment at `collab.js:315` saying it does. The doc arrives as binary CRDT
state.
6. Broadcasts awareness to everyone.
`canWrite` is computed once at connect time (`collab.js:430`) and never
re-checked. Revoking someone's write access does not take effect until they
reconnect.
### Message taxonomy
Binary frames are Yjs sync protocol; text frames are JSON. The JSON `type` is
allow-listed to exactly five values (`collab.js:526-528`):
| Type | Gate | Effect |
|---|---|---|
| `cursor` | `canWrite` | position is validated and coerced to safe integers (`collab.js:532-543`), then broadcast to others |
| `save` | `canWrite` | immediate save, see below |
| `publish` | `canWrite` | permission-checked snapshot, see below |
| `comment` | `canWrite` | relays only ids, never content (`collab.js:736-750`); the actual CRUD is REST |
| `title` | `canWrite` | updates `logs.title`, broadcasts, and logs `log.rename` |
All five are gated the same way. `title` was the exception until 2026-08-09
(B6 in `open-questions.md`): a read-only participant could rename the document,
and the `log.rename` activity that followed also mailed every watcher.
`comment` broadcasts are id-only by design: the receiving
client refetches over REST, which keeps the WS from becoming a second
authorisation surface for comment content.
Binary frames are dropped outright for read-only clients (`collab.js:508`).
Updates are echoed to every peer except the origin, using the sender's `ws` as
the Yjs transaction origin (`collab.js:106-117`, `collab.js:225`), which is what
prevents an echo loop.
**Image credits.** Every client's save carries the whole shared document, so
the client that sends a save is not necessarily the writer who added what it
carries. `applySyncFrame` (`collab.js:217-231`) scans each binary frame's bytes
for `/doc-images/<hash>.webp`, and only for a frame that names one renders the
`default` fragment before and after applying it. `entry.credits`, a
`DocImageCredits` from `routes/helpers/images.js` opened with the entry
(`collab.js:79`, seeded with the stored HTML and any loaded CRDT state at
`collab.js:101`, closed on cleanup at `collab.js:258`), credits a reference to
the frame's writer only when their own frame named it, it newly appeared, the
stored HTML does not already show it, and nobody holds the credit yet. Saves
then record references per credited writer, never per sender
(access-control.md section 3f).
### Save (`collab.js:555-623`)
Cancels the debounce, encodes the CRDT state, sanitises the client HTML through
`sanitizeHtml` then `extractImagesFromHtml`, and writes. It only writes
`html_content` when the HTML actually changed against `entry.lastSavedHtml`
(`collab.js:576`); otherwise it writes the blob alone. On an HTML change it then
records the images (`recordLiveDocImages`, `collab.js:239-246`, which logs its
own failure so nothing after it is skipped), and runs `processMentionsOnSave`
and `logActivity('log.update')` (`collab.js:599-618`).
### Publish (`collab.js:654-731`)
Loads the log's squad context, calls the shared `canPublish`
(`collab.js:670`), bumps `logs.version`, writes HTML plus blob plus version,
inserts the `versions` row, fires mentions and `log.publish` activity, then
broadcasts `{type:'published', version, title}` to **all** connections including
the publisher. Title is capped at 255 chars, notes at 5000
(`collab.js:656-657`).
### Lifecycle and teardown
On close (`collab.js:753-768`): drop the connection, decrement the per-user
count, rebroadcast awareness, and if this was the last connection schedule both
a final save and cleanup. Cleanup after 30s destroys the `Y.Doc` and removes the
map entry, but only if no one reconnected (`collab.js:251-261`).
**Sign-out closes the sockets of one session (W6-CDX-6).** Each socket records the digest of
the session it authenticates with (`ws.sessionId = digest`, `collab.js:431`, the SHA-256 of the
token it presented) before its auth read's first await, so a sign-out that lands at any point finds
it. After the read, a socket whose sign-out already closed it (`sessionClosedRecently`,
`services/closed-sessions.js`, a five-minute set) or that is no longer open is closed 4004 before it
registers. `closeSocketsForSessions(digests)` (`collab.js:197`) closes, with 4004 "Signed out",
exactly the sockets whose digest is in the set, across every collab server, and records the digests. `POST /api/logout` calls it for the one session it deleted, and the
back-channel receiver calls it for every row a logout token ended (`routes/auth.js:608`,
`routes/oidc.js`, section 3 of `request-lifecycle.md`). The same user's other devices stay open,
which the tests pin. The user channel has the same closer (`services/user-channel.js:88`, socket
recorded at `user-channel.js:183`). Rotations (password and email changes) close the sockets of
every session they end, through the same closers (`routes/auth.js:57`).
**The open sockets are re-checked every minute (W6-CDX-8).** Auth is read once, on a socket's first
message, so a session ended later (a deactivation, a row deleted by hand) would keep its socket. The sweep
(`services/socket-sweep.js`, started at `server.js:293`) runs every 60 s, collects the digests of the
authenticated sockets on both servers (`openSessionDigests`, `collab.js:196`, `user-channel.js:87`),
asks `liveSessionDigests` (`socket-sweep.js:30`) which of them still hold a live session for an active user
(OIDC only on an OIDC-only instance, batched 500 at a time), and closes the rest with 4004 through
`closeSocketsForSessions` (`collab.js:212`). A failed pass is logged and the timer keeps running.
**All of this is per-process, in-memory. Restarting the server drops every
in-memory doc; state survives only through `ydoc_state`. A graceful stop
(SIGTERM or SIGINT) writes every pending debounced save first
(`flushPendingSaves`, `request-lifecycle.md` section 7), so `ydoc_state` is
current after one; only a crash or a SIGKILL can leave it up to 3 seconds
stale. This is the load-bearing reason the single-process architecture is not
negotiable: a second replica would maintain a second, divergent `Y.Doc` for the
same log, and the instance lock (`services/instance-lock.js`) now makes a second
process on the same schema refuse to start.
### The REST side-channel
`broadcastToDoc(logId, message)` (`collab.js:787-795`) lets REST handlers push
arbitrary JSON to live editors without going through Yjs. Its only current
callers are the GitHub pull and resolve routes, which emit
`{type:'github-pulled', ...}` (`github.js:1193`, `1235`, `1417`). It returns
`false` when no one has the doc open.
Three read-only accessors feed the admin and presence surfaces:
`getActiveDocCount` (`collab.js:800`), `getActiveUsers(logId)`
(`collab.js:807`), `getAllPresence()` (`collab.js:820`).
## 3. Versions
`versions` rows are snapshots of `html_content` with an optional title and
release notes (`init.sql:360-375`). Four operations:
- **Publish** bumps `logs.version` and inserts a row. Two entry points, REST
(`documents.js:173`) and WS (`collab.js:654`), sharing `canPublish`.
- **List / fetch** (`documents.js:350`, `documents.js:386`) behind read access.
- **Restore** (`documents.js:426`) writes the old HTML back as a *new* version,
so history is append-only and nothing is lost. It nulls `ydoc_state`.
- **Delete** (`documents.js:500`) allows admin or the version's own author
first, then falls back to `squad_members.can_delete_version` or squad
`role = 'owner'`.
The publish route also accepts `create_github_release`, `target_repo`, and
`tag_name` (`documents.js:179`), tying a version to a GitHub Release through
`versions.github_release_id` / `github_tag_name` / `github_target_repo`.
## 4. The editor
`src/pages/Editor.jsx` (1356 lines) is the client. The Tiptap extension stack
is assembled at `Editor.jsx:341-361`:
`StarterKit` with `codeBlock`, `underline` and `undoRedo` disabled
(`Editor.jsx:342`, undo/redo because the `Collaboration` extension supplies its
own CRDT-aware history), plus `ResizableImage`, `Placeholder`, `Underline`,
`TextAlign`, `Link`, the four table extensions, `CodeBlockWithLanguage`
(lowlight), `DrawioBlock`, `GitHubCodeEmbed`, `GitHubIssueEmbed`, `Mention`,
and `Collaboration.configure({ document: ydoc })`.
Initial content is deliberately empty (`Editor.jsx:362-365`); the
`Collaboration` extension populates the editor from the shared Y.Doc. Its
`onUpdate` fires for **both** local and remote changes (`Editor.jsx:393-407`),
which is why state updates there are deferred rather than applied inline.
`useCollab(logId, onRemoteUpdate, onRemoteComment, onPublished, onRemoteTitle)`
(`src/hooks/useCollab.js:31`) owns the socket: it memoises the `Y.Doc`
(`useCollab.js:49`), connects to `${proto}//${host}/collab?logId=${logId}`
(`useCollab.js:71-72`), sets `binaryType = 'arraybuffer'`, sends the auth frame
on open (`useCollab.js:103`), queues local updates until the server's sync step
completes and flushes them afterwards (`useCollab.js:83`, `useCollab.js:126-129`),
and auto-reconnects on close. It exposes `sendCursor`, `sendSave`, `sendPublish`,
`sendTitle`, `sendCommentEvent`. `sendPublish` waits for the `published` frame
with a 10-second timeout (`useCollab.js:258`).
Mobile visitors are redirected away from `/editor/:logId` entirely by
`MobileEditorGuard` (`src/App.jsx:43-47`); on mobile, documents are view-only
through the archive view.
---
## Related
- [access-control.md](access-control.md) for what `canWrite` at
`collab.js:430` actually resolves.
- [notifications-and-activity.md](notifications-and-activity.md) for
`processMentionsOnSave` and `logActivity`, called from every save path here.
- [github-integration.md](github-integration.md) for the pull/push paths that
write `html_content` from outside the editor.