sync-rules-spec.md
Rama main · réplica de lectura del repositorio original.
# Clipman Sync Rules Specification
This document is normative for every Clipman client that implements machine-scoped
sync rules ("sync channels"). All clients on all platforms must implement exactly
these derivations, schemas, and algorithms so that the same buckets, routes, and
merge outcomes are produced everywhere. The implementation plan lives in `sync.md`;
this file is the contract.
Binding constraints:
- Server code must not change. Everything here works against every deployed 2.x
Clipman Server using only `HEAD/GET/PUT /api/v1/database/{id}`.
- The `CLIPDB2` container format, PBKDF2 parameters (150,000 iterations,
HMAC-SHA1), and AES-256-CBC + HMAC-SHA256 layout must not change.
- The `ClipEntry` wire schema gains no new required fields. Routing uses only
existing fields.
- `ClipmanCli/internal/` and `ClipmanLinuxBackend/internal/` are forks; shared
logic changes land in both trees in the same change.
- The feature is opt-in and off by default. With rules disabled, client behavior
is identical to a client without this feature.
- Old clients must never lose data: they sync the core channel only and must
never delete or corrupt channel buckets or channel files.
## 1. Concepts
- A **sync channel** is a named partition of the shared history. Each entry lives
in exactly one channel at a time. The default channel is **core**, stored in the
existing bucket (server mode) or the existing `clipman-history.clipdb` file
(shared-folder mode). Enabling the feature moves nothing until a rule routes
entries elsewhere.
- A **routing rule** is attached to each channel and decides which entries live
there, computed only from existing entry fields (`Group`, `SourceMachine`,
rich-text image presence). First matching channel in list order wins; no match
means core.
- A **device subscription** lists which channels a named device downloads. Core is
always downloaded by everyone. A device not listed in the rules document
subscribes to everything.
- The **rules document** holds channels, routes, and device subscriptions. It is
stored in its own bucket (server mode) or its own file (shared-folder mode),
never inside the history database, because the Windows client drops unknown
JSON fields on save.
## 2. Bucket identity
All inputs are trimmed of leading and trailing whitespace before derivation,
matching the existing database-id derivation. `base64url_nopad` is RFC 4648
base64url with padding stripped.
```
key = SHA256(UTF8(trim(token)))
databaseId = base64url_nopad(HMAC-SHA256(key, UTF8("Clipman.ServerDatabaseId.v1\n" + password))) [existing]
channelId(k) = base64url_nopad(HMAC-SHA256(key, UTF8("Clipman.ServerChannelId.v1\n" + password + "\n" + k)))
syncRulesId = base64url_nopad(HMAC-SHA256(key, UTF8("Clipman.ServerSyncRulesId.v1\n" + password)))
```
`k` is the normalized channel key (Section 3). All ids are 43 characters and are
valid database ids on every deployed server. A blank token or blank password
yields an empty id (channel sync requires server mode's mandatory password, or
shared-folder mode where ids are not used).
Cross-client test vectors (token `example-token`, password `example-password`):
| Purpose | Input | Expected id |
|---|---|---|
| database id (existing) | - | `l4GLcFU7RrlmkGXoRyQ7-zVG5D5S0VmfwO6-dGNmebU` |
| sync rules id | - | `j5Z6kOIWgsJMqS0IRzNJEq38aqJ-iA8e6yzyX0W71WQ` |
| channel id | `work` | `F0MZBlui50Vd37JVf-JcvOWvoHV71IDlQE5OnfVBKeA` |
| channel id | `images` | `K_hH97mxfF4_DQvN90Orzu_HUz7MOcKYoL3-6nY-TbQ` |
| channel id | `desktop only` | `02tgOt5QC_sWY2RmoI2pqII9MocLQ7-XIMHaSRVBE1o` |
Shared-folder mode maps channels to sibling files in the data folder:
```
clipman-history.clipdb core (existing name, unchanged)
clipman-channel-<key with spaces as dashes>.clipdb
clipman-sync-rules.clipdb
```
## 3. Channel keys and name matching
- Channel display name: what the user typed, stored as `Name` in the document.
- Channel key: `lowerInvariant(trim(Name))`. Validation, enforced by every rules
editor: the key must match `[a-z0-9]([a-z0-9 _-]{0,30}[a-z0-9])?` (ASCII only,
1-32 chars, no leading/trailing space, dash, or underscore). Non-ASCII names are
rejected so no Unicode normalization library is required anywhere.
- Reserved keys, rejected as channel names: `core`, `all`, `pinned`, `sync-rules`.
- Storage-name collisions: spaces fold to dashes in shared-folder file names
(Section 2), so the distinct keys `my work` and `my-work` would share
`clipman-channel-my-work.clipdb`. Every rules editor rejects a document in
which two channel keys fold to the same storage name. This is an edit-time
rule only: readers accept an already-saved document with such a collision and
route with it, because a client must not drop to disabled rules over a
document other clients are still routing by.
- Device-name matching (both `Devices[].Name` against the local device name and
`Route.SourceDevices` against `Entry.SourceMachine`) compares
`lowerInvariant(trim(x))` on both sides. This is deliberately stricter and more
portable than the UI's culture-aware device filter; the difference is
acceptable because routing must be identical on every platform.
- Group matching (`Route.Groups` against `Entry.Group`) uses the same
`lowerInvariant(trim(x))` comparison, consistent with the existing cross-client
case-insensitive group folding.
## 4. The rules document
Stored as JSON inside a standard container: `CLIPDB2` encrypted when a history
password exists, `CLIPDB1` compressed otherwise (identical container selection to
the history database). The payload is this document, not a ClipDatabase:
```json
{
"Clipman": "sync-rules",
"Version": 1,
"Enabled": true,
"UpdatedUnixMs": 1757200000000,
"UpdatedBy": "Desktop",
"Channels": [
{ "Name": "Images", "Route": { "Kind": "RichTextImages" } },
{ "Name": "Work", "Route": { "Groups": ["Work", "Standup"] } },
{ "Name": "Desktop only", "Route": { "SourceDevices": ["Desktop", "Work-PC"] } }
],
"Devices": [
{ "Name": "Desktop", "Channels": ["*"] },
{ "Name": "Jeff-iPhone", "Channels": ["work"] },
{ "Name": "Work-PC", "Channels": ["work", "desktop only"] }
]
}
```
Field semantics:
- `Clipman`: always the string `sync-rules`; readers reject other values.
- `Version`: document format version, currently 1. A client that sees a higher
version treats the document as read-only: it applies what it understands, must
not rewrite the document, and syncs core plus the channels it can resolve.
- `Enabled`: `false` means every client behaves exactly as without the feature,
even if channels are defined. This is the instant global off switch.
- `Channels[].Name`: display name; its normalized key (Section 3) identifies the
channel and feeds the id derivation. Keys must be unique within the document.
- `Channels[].Route`: conditions are ANDed within one route; a route must contain
at least one condition. `Groups` and `SourceDevices` are lists matched per
Section 3. `Kind` currently allows only `"RichTextImages"`, which matches when
the entry's `RichText` is present and its `HtmlFragment` contains the ordinal
substring `data:image/`.
- Routing is evaluated top to bottom over `Channels`; the first matching route
wins; an entry matching no route lives in core. Evaluation STOPS at the first
matching route even when that route's channel key is unresolvable (possible
only under a future-version document): the entry then routes to core rather
than falling through to later routes the client only partly understands.
- Read-only documents never trigger relocation: under a future-version document
a client treats every resident entry's current channel as authoritative (no
misrouted-entry migration, no channel re-routing), because a client that
cannot fully evaluate the rules must not fight newer clients over placement.
Only NEW captures are routed (first-match-stops rule above), with write-through
for resolvable unsubscribed targets.
- `Devices[].Name`: the device's user-visible device name.
- `Devices[].Channels`: channel keys, or the single element `"*"` meaning all
channels. Core is implicit and always synced. A device whose name is not
listed subscribes to everything (safe default: behaves like a full sync).
- Concurrency: the rules bucket uses the normal `If-Match` flow. Concurrent edits
merge by whole-document last-writer-wins on `UpdatedUnixMs`; ties break toward
the higher `UpdatedBy` string by ordinal comparison.
- Caching: every client caches the last-seen rules document locally. If the rules
bucket returns 404 but a cache exists, the cache stays in effect and the client
re-uploads it with `If-None-Match: *`. If neither exists, rules are disabled.
Exception: a cached FUTURE-VERSION document (Section "Version" above) is kept
for display only - it must never be re-uploaded, so it does not arm the 404
re-upload fallback; on a genuine rules-bucket loss such a client falls back to
disabled rules until an up-to-date client restores the document.
- Registry behavior: after rules are enabled, an updated client whose device name
is missing from `Devices` adds itself with `Channels: ["*"]` on its next
successful sync.
## 5. Multi-channel sync algorithm
Definitions: the **view** is the merged in-memory database shown to the user;
**residence** maps each entry id to the channel it was loaded from.
Download / poll (each poll tick):
1. `HEAD` the rules bucket. If its revision changed, `GET`, decode, merge
(Section 4 LWW) into the cached document, and recompute the subscription set.
2. For each subscribed channel (core plus subscriptions), `HEAD`; `GET` only
channels whose revision changed.
3. Assemble the view across channels by entry `Id` (core first, then channels in
rules-document order). Cross-channel assembly deliberately does NOT use the
entry-level field merge or text-fallback matching: within a channel the
existing merge applies as today; across channels an `Id` collision (move
race) is resolved by dropping the copy with the lower `ModifiedUnixMs`
wholesale, and on a tie the earlier channel in assembly order wins. The same
text under different ids in two channels stays duplicated (see Section 8).
Tombstones apply only within their own channel, with one exception: a
tombstone with a non-empty `TextHash` also suppresses matching-text entries
in other channels, subject to the same entry-changed-before-deletion rule the
entry-level merge uses. A relocation marker (empty `TextHash`) must never
suppress a live entry with the same id in another channel. Manual order is
channel-local. To form the view, sort each channel by its own `ManualOrder`
(then `CreatedUnixMs` and `Id`), and merge those sequences by choosing the
available head with the earliest positive `CreatedUnixMs`; ties follow
channel assembly order. Treat a missing or nonpositive creation time as
later than a valid time. Renumber the resulting view from 1. This preserves
deliberate order within every channel while preventing a newly created
channel entry at local position 1 from jumping ahead of older history.
4. The dropped loser of an `Id` collision is repaired (rewritten to its routed
channel) on the next save.
Upload (each local mutation):
1. Recompute `target = route(entry)` for every entry in the view.
2. Build one database per channel: entries whose target is that channel, plus the
tombstones that belong to that channel.
3. Relocation: when residence differs from target, the entry moves into the
target channel's database and a **relocation tombstone** is added to the source
channel: `{Id: <entry id>, TextHash: "", DeletedUnixMs: <now>, SourceMachine:
<this device>}`. The empty `TextHash` distinguishes "moved" from "deleted";
merge implementations must only match `TextHash` when it is non-empty, and
normalization must not back-fill an empty `TextHash` on such markers.
4. Dirty detection: serialize each channel's plaintext JSON deterministically
with the database-level `UpdatedUnixMs` field zeroed, and compare its SHA-256
to the hash recorded at the last transfer. Zeroing `UpdatedUnixMs` makes the
hash durable across poll cycles (normalization restamps that field on every
pass). Only dirty channels are encrypted and `PUT` (with per-channel
`If-Match`). The comparison must happen on plaintext: ciphertext differs on
every encode because the IV is fresh.
5. Upload ordering (add-then-remove two-phase): a partial failure must never
leave an entry deleted from its source channel without having been committed
to its target. To guarantee this per entry, including for channels that both
gain and lose in one save and for relocation chains or cycles, uploads happen
in two phases. Phase 1: every channel is uploaded with its rebuilt content
EXCEPT that entries departing it are still included and its new relocation
markers are withheld (targets receive additions; sources do not yet drop
departures). Phase 1 carries the copy of a departing entry that the source
channel was fetched with, not the locally modified one, so a channel whose
only change is a departure is typically byte-identical to what the server
holds and skips its phase-1 upload (this is an optimization, not an
invariant - ordering renumbering can still dirty such a channel; ports must
not assert on the skip); the target receives the modified copy, whose higher
`ModifiedUnixMs` wins the duplicate resolution of download step 3 for the
transient window in which both channels hold the id. Consequence of the
fetched-copy rule: when the target's upload fails, an edit made to the
departing entry in the same save is committed nowhere - the mutation is
reported as failed and the entry keeps its pre-mutation value at the source;
nothing previously committed is lost. Phase 2: only after every phase-1 upload has committed, each
losing channel is rebuilt with departures removed and relocation markers
added, and uploaded again. A phase-2 failure is safe: the entry exists in
both channels, and view assembly (step 3) resolves the duplicate until the
next save repairs it. Channels that lose nothing are uploaded once, in phase
1, subject to the dirty check. On a 409 for one channel: `GET` that channel,
merge, rebuild for the current phase, retry.
Salt sharing: a channel blob created for the first time copies the core
database's salt, so one PBKDF2 derivation serves every channel through the
existing derived-key caches. The rules blob does the same.
Channel deletion (rules editor action): the editing client re-routes that
channel's entries (they fall through to the next matching rule or core), uploads
the affected channels, uploads the now-empty channel database, then removes the
channel from the rules document. The empty bucket remains server-side and can be
pruned with existing server admin tooling.
Rules edits: the device that edits the rules performs the re-route immediately.
Editors must be subscribed to every channel a given edit affects; rules UIs
disable edits that the local device cannot see. Additionally, any client that
notices a misrouted resident entry during upload migrates it; moves are
idempotent and convergent under the entry-level LWW merge.
## 6. Write-through to unsubscribed channels
A device may capture an entry that routes to a channel it does not subscribe to.
The writer performs a one-shot fetch-merge-put of the target channel: `GET` (404
means new; then `If-None-Match: *`), merge the new entry, `PUT` with `If-Match`,
then discard the channel from memory. The entry does not appear in the local
view; the UI announces once: "Added to <channel name> for your other devices."
If the write-through fails (offline), the entry is kept in a small local pending
store and retried after the next successful poll.
## 7. Mixed fleets and compatibility
- Old servers need no changes; channels are ordinary buckets.
- An old client (server or folder mode) syncs only core. It never sees channel
entries; nothing is lost or corrupted. Its new captures land in core and are
re-routed by updated clients on their next save.
- Guidance shown to users: enable sync rules only after every device runs a
version that supports them; older devices keep syncing the main history only.
- An old client deleting a core entry writes a core tombstone, which updated
clients apply normally.
## 8. Known limitations (accepted)
- Quick Paste hotkeys referencing an entry in an unsubscribed channel behave as
for a deleted entry.
- Deleting an entry on one device while another concurrently moves it between
channels can resurrect it; converges on the next edit or delete.
- Capture-time text dedupe consults only subscribed channels, so identical text
captured on devices with disjoint subscriptions can exist once per channel.
- Retention limits evaluate against the subscribed view only; pinned and named
entries remain exempt as today.