docs/semantic-language.md

Rama main · réplica de lectura del repositorio original.

<!-- authority:non-specification -->

# First-class semantic language

Exomem keeps Markdown readable while making small observations, richer typed
units, retrieval filters, ranking evidence, and authored graph relations
addressable through the public product commands. Markdown remains the source of
truth; lexical, vector, and graph stores are rebuildable sidecars.

<!-- BEGIN GENERATED SEMANTIC AUTHORING CONTRACT -->
<!-- exomem-semantic-authoring:v11 sha256:6ab1c7e3ef04f6ad089520f568fe613e159860bf813cb25227b52cfd239067a8 -->
## Semantic authoring contract

Choose homes by future use and accumulated content, not mentions or broad scope. Keep coherent long notes; reuse or create a home for independent material.

Live compiled creates, replacements and transitions to live need a valid nonempty semantic unit; pending pages may remain unit-free. Either compact or rich form satisfies the minimum; compact is preferred, and a valid rich unit does not need a duplicate compact restatement.

Semantic roles:

- Category: One primary open-vocabulary label describes what a unit is about; rich category defaults to its governed kind unless explicitly overridden.
- Tag: Zero or more optional secondary retrieval labels refine lookup and never replace category or determine kind.
- Kind: The governed semantic form: compact units always use `observation`; rich units use their recognized heading kind.

Compact grammar: `- [category] content #tags (context) ^anchor`. Parse valid compact observations anywhere outside fenced code blocks. Exomem writers use `-` under the canonical `## Observations` section. Parser bullet markers are `-`, `*`, `+`; the canonical marker is `-`. Parse from the end by taking anchor, then context, then trailing tags; the authored display order remains tags, context, anchor. Category uses open vocabulary.

- Compact category: the unit's one primary open-vocabulary subject or domain label. After trimming, use 1-64 Unicode code points; begin with a Unicode letter; then use only Unicode letters or digits, spaces, `_`, or `-`. Apply Unicode NFKC and casefold, then collapse runs of spaces, `_`, and `-` to one `_`. Registry alias resolution is separate from authored canonicalization; an unseen valid category needs no registry write.
- Compact content: the unit's substantive observation. Use non-empty content that remains on one Markdown line. Escaped parentheses, embedded hashes, and non-trailing tag-like text remain content.
- Compact tags: zero or more optional secondary retrieval labels; tags do not replace the primary category or governed kind. Write `#slug`. Use 1-64 Unicode letters or digits, `_`, `-`, or `/`; begin with a letter or digit; do not use empty path segments or a trailing `/`. Use one contiguous trailing run after content and before optional context and anchor.
- Compact context: one optional authored qualifier for the observation. Write `(<context>)`. Use one balanced, unescaped parenthesized suffix preceded by whitespace.
- Compact anchor: one optional stable authored unit identifier. Write `^anchor`. Use 1-64 ASCII letters, digits, or hyphens and begin and end alphanumeric. Place it at the end of the line.
- Compact exclusions: observation-shaped rows inside fenced code blocks; task labels `[ ]`, `[x]`, `[X]`, and `[-]`; reserved or punctuation-bearing bracket labels outside category grammar. Compact units do not carry typed unit relations; use a canonical note-level relation or the rich form.
- Rich: write `## <Governed Kind>` with optional leading metadata `- category: <open category>`, `- id: <stable-id>`, `- tags: <comma-separated tags>`, `- context: <context>`, `- relations: <relation-type>: [[Target]]`, `- verdict: <epistemic outcome>`, `- check_by: <ISO revisit date>`, `- as_of: <ISO start date>`, `- until: <ISO last valid date>`. Metadata rows are optional and leading; the canonical writer emits category, id, tags, context, relations, verdict, check_by, as_of and until; category defaults to the governed kind when omitted. Accepted metadata order is flexible while rows remain leading. After optional leading metadata, add a blank line and a substantive Markdown body. Typed unit relations require the rich form. A `#anchor` target addresses a unit.
- Epistemic corrections: Inspect affected claims and evidence. Distinguish attributed accounts from established facts. Correct suitable existing units with guarded writers, justified verdicts and exact links; verify current state. Refutation alone changes neither lifecycle nor rank. Writers retain prior interpretations; request read_instructions with profile=full for history guidance. Keep an anchored unit's identity when its wording changes. Preserve a false historical proposition and author its verdict; add a distinct corrective unit when useful. Refutation and contradiction do not imply supersession or reduce rank. The semantic writer retains prior interpretations atomically. Read revision_history with read_page(part=history), optionally for unit_ref, using history_limit and history_continuation in options. When it carries lookup=stale, repeat the same call shortly instead of reading absence or paging on. Existing history remains the page's operation log. Entities and legacy pages use their existing edit owner; Records use guarded row corrections and history. Corrections need no dated child, special user nudge, or confirmation beyond the active engagement policy.
- Rich boundary: A heading at level N owns content until the next non-fenced heading at level N or shallower; deeper headings remain in its body. `empty_rich_unit` means a recognized rich heading has no substantive body; Add substantive body content or remove the empty recognized heading.
- Exact applicability: obtain `read_instructions(profile='compact', section='authoring')` and read `semantic_authoring.minimum_semantic_unit` for current structural applicability and the status-class rule. Apply `compiled_intent(after_state) = canonical_compiled_destination(path) OR the normalized type's note-type role is compiled`. Shipped compiled types and their default folders: `research-note` in `Notes/Research`, `insight` in `Notes/Insights`, `failure` in `Notes/Failures`, `pattern` in `Notes/Patterns`, `experiment` in `Notes/Experiments`, `production-log` in `Notes/Productions`, `procedure` in `Notes/Procedures`. More may be registered; `inspect_memory(operation='schema.inspect', arguments={'subject': 'note-types'})` lists the live set. Reject missing, invalid, or mismatched compiled frontmatter before evaluating the minimum-unit predicate. A page in a type folder needs a compiled type; a compiled type may sit in any Notes folder, and a compiled page outside Notes is refused. Folders are defaults. `create_note` files research notes in their `project` home. Other compiled notes use the first `topics` entry with a home. `inspect_memory(operation='schema.inspect', arguments={'subject': 'homes'})` lists homes. The draft's `placement` gives the folder, basis and call to make a topic a home. The minimum predicate applies when the path and normalized compiled type structurally match; the result is writable managed Markdown in the governed subtree; the result is outside Sources, Evidence, and trash; no activation exclusion applies; the admitted effective status class is live. Check live creates, replacements and transitions to live; pending compiled notes may remain unit-free until activation.
- Existing active pages: A post-activation compliant page cannot lose its final valid semantic unit.
- Exempt content: arbitrary non-compiled Markdown, dataset cards, Evidence artifacts, hubs, indexes, logs, non-Markdown files, schema and admin artifacts, snapshots, Sources, templates, trash.
- Routes: use `create_note` for a new compiled note, `supersede_note` for a replacement, `add_observation/edit_note` for one unit, and `edit_note` for a small edit or activation. Tier 2 manage_memory `files.create` and `files.append` receive the same semantic precommit contract on the complete resulting compiled Markdown; prefer create_note or supersede_note when their typed route fits.
- Findings: `missing_semantic_unit` means an applicable live-class compiled result has no valid non-empty unit; `empty_rich_unit` means a recognized rich heading has no substantive body. Add substantive body content or remove the empty recognized heading.
- Compact remediation: Add `## Observations` and `- [operating constraint] Keep retries bounded #reliability`.
- Rich remediation: Alternatively add `## Decision`, a blank line, and a substantive body.
- Semantic-unit coverage and relation-review disposition are independent obligations.
- Validation review: Validation evaluates the proposed result without saving content or review state; the request includes relation assessment. Inspect the current page and candidate evidence. If no truthful link applies, preview again with a bounded evidence-based unlinked_reason. Missing edges prove nothing; preserve typed relations and metadata. Report insufficient evidence or unresolved material choices. Commit needs separate write authority.
- Portable categories: Choose exactly one primary category: prefer a meaningful epistemic or operational role and put the domain in tags, but if the role would only be a generic fact, finding, or observation and the domain is the durable lens, use a domain category instead. The category vocabulary is open: these core keys are a shared starting point, not a closed list. When no core key is a good primary fit, author a new meaningful category rather than forcing an ill-fitting one. Use exactly one primary category; kind is the governed form, tags are secondary facets, and relations are typed edges. The rich form's category defaults to its kind, so `category: decision` is redundant when the kind is Decision. Create multiple distinct semantic observations and typed relations when the source genuinely supports them, but never multiply units or relations to satisfy a quota and never duplicate the same fact. Core keys are `action`, `assumption`, `code`, `config`, `constraint`, `decision`, `design`, `fact`, `finding`, `insight`, `preference`, `problem`, `question`, `requirement`, `risk`, `technique`. Core aliases are `actions` → `action`, `assumptions` → `assumption`, `configs` → `config`, `configuration` → `config`, `configurations` → `config`, `constraints` → `constraint`, `decisions` → `decision`, `designs` → `design`, `facts` → `fact`, `findings` → `finding`, `insights` → `insight`, `open_question` → `question`, `open_questions` → `question`, `preferences` → `preference`, `problems` → `problem`, `questions` → `question`, `requirements` → `requirement`, `risks` → `risk`, `techniques` → `technique`. Role example: `- [decision] Relocate to a coastal city next spring #life ^relocation`. Domain example: `- [nutrition] Evening protein improves adherence #experiment ^evening-protein`. Breadth example (life, finance, legal/travel, and career alongside one retained code line):

```markdown
- [constraint] Keep retry windows bounded #code ^retry-windows
- [risk] Variable-rate mortgage payments could spike #finance ^mortgage-rate-risk
- [question] Does the destination require a visa before travel #legal #travel ^visa-requirement
- [career] Weigh a sabbatical before the next promotion cycle #growth ^sabbatical-timing
```

Rich example:

```markdown
## Decision
- id: commit-to-morning-training
- tags: health
- relations: supports: [[Knowledge Base/Notes/Health/Morning training]]

Commit to a fixed 6am training block on weekdays so consistency compounds and health stays the durable lens for this decision.
```
<!-- END GENERATED SEMANTIC AUTHORING CONTRACT -->

For rich units without explicit `- category:`, the heading supplies the raw and
normalized category identities before reviewed category-alias resolution; the
resolved public category falls back to the governed kind when no category alias
applies. Rich `tags` and `context` are first-class retrieval fields, not merely
generic metadata. Category, kind, tags, context, and authored relations remain
independent axes.

Rich units also carry two governed epistemic-loop metadata keys. `check_by` is
one exact ISO calendar date (`YYYY-MM-DD`) naming the day a claim should be
revisited; it is filterable as the typed date field `unit.check_by`, so "what is
due" is an ordered query rather than a scan. `verdict` records how the claim
turned out — exactly one of `confirmed`, `refuted`, `qualified`, `inconclusive`,
or `abandoned` — and is filterable as `unit.verdict`. Both appear on unit hits
when present and are omitted when absent, so an unjudged unit is distinguishable
from a judged one.

Both are answered from the maintained catalogue rather than by reading pages, so
a due-date query costs an index read on a served cell. One filter field has no
index answer: `page.frontmatter:/<pointer>`, which is open-ended by construction.
A served request naming it is refused with `UNSUPPORTED_FILTER_FIELD` rather than
silently falling back to a corpus scan, while an offline CLI caller keeps the
exact scan. `unit.dimension` (below) has no column either, and a served request
answers it without a scan. A date bound is answered exactly when it names a whole
day, so `updated_after`, `updated_before`, `recency_days` and a bare ISO date can
all be negated with `$not`; a bound carrying a timestamp is refused under the same
code when nothing else in the filter narrows it, because a day column cannot
decide the hour and a wrong negation would silently drop pages.

`verdict` is state, not supersession, and not a score. A refuted unit keeps its
page's active standing and its ordinary rank: nothing replaced it, the question
simply got an answer, and the negative result stays retrievable with the
evidence that refuted it. A claim carries no confidence score and a verdict is a
judgment in words, so a probability or percentage is never a valid `verdict`; the
parser rejects one outright. A degree of belief that its holder stated is a
`## Belief` unit; "Recording a degree of belief" in the scaffold's
`references/page-types.md` gives its form. The
`prediction` governed kind names a unit that is a claim about the future, and
pairs naturally with both keys. None of this adds a relation kind: attach the
refuting or confirming material with the governed vocabulary you already have
and let `verdict:` carry the judgment itself.

Use `observe_memory(operation="add"|"update"|"remove"|"validate")` for one
unit. Update and remove operations must carry both the current parent
`content_hash` and the unit fingerprint. Compact observations cannot author
typed unit relations: use a rich unit or a canonical note-level relation under
`## Relations`.

## Retrieve pages, units, or both

`search_memory` and `find` accept `result_level="page"`, `"unit"`, or `"mixed"`.
`auto` preserves the normal page-first product behavior unless unit predicates
require a unit-capable result.

```json
{
  "query": "battery policy",
  "categories": ["config"],
  "kinds": ["observation"],
  "result_level": "mixed"
}
```

The `categories` and `kinds` lists are shortcuts. The general `filters` object
uses a bounded typed expression over reserved `page.*` fields, RFC-6901
frontmatter paths, and the closed `unit.*` field set. Lists are OR within an
axis; category/kind/text/filter axes combine with AND. Invalid or over-limit
plans fail before candidate retrieval and never silently widen.

```json
{
  "query": "",
  "filters": {
    "$and": [
      {"page.type": {"$eq": "insight"}},
      {"unit.category": {"$in": ["config", "rule"]}},
      {"page.frontmatter./status": {"$ne": "superseded"}}
    ]
  },
  "result_level": "unit"
}
```

An empty query plus filters is a filter-only lookup. It uses the documented
filtered-most-recent sort tuple; it does not fabricate a text score.

`unit.dimension` filters units by epistemic dimension, with `$eq`, `$in` and
`$ne`. It takes a dimension key or a label in any language, and a name that no
registry entry holds is refused with the known keys. A unit belongs to a
dimension when its kind is a member, when it carries a member key, or when it is
an end of a typed edge whose family is a member. A unit's own `relations` row
starts an edge at that unit, and a link with an anchor, such as `[[#before]]`,
ends one at the unit the anchor names. An edge written in frontmatter, in a
page's relations section or as a link with no anchor starts or ends at the whole
page, and that end reaches every unit of the page. A vault adds a dimension in
`_Schema/semantic-language-registry.yaml`, and the filter takes it with no code
change. A vault may also remove a dimension it added: no page stores a dimension
key, and a filter that still names it is refused with the keys that exist.

```json
{"filters": {"unit.dimension": {"$eq": "currency"}}, "result_level": "unit"}
```

A dimension with a family member reads the edge graph in one query for the
units it tests. While the graph sidecar is missing or stale the request raises
the typed warming outcome and never answers as if no edge existed. A caller sees
an edge only when it may see both of its pages.

A served request answers `unit.dimension` without reading pages. A dimension
with no family member, whose member keys each have an index column, becomes the
equivalent kind and key predicates, anywhere in the filter. Any other dimension
is checked on the best units the search found, with one graph read for them,
so the predicate must sit at the top level of the filter or under a top-level
`$and`; under `$or` or `$not` it is refused with `UNSUPPORTED_FILTER_FIELD`.
The limit applies before that check, so when it drops units the reply's
`user_summary` says that the filter ran only on the best matches, how many of
them it kept, and that more may exist. The offline CLI evaluates every dimension on every unit.

### Dimension groups

A unit search asks for dimension groups through the same filter, with no other
argument. When `unit.dimension` sits at the top level, alone or under a top-level
`$and`, `$eq` returns one group and `$in` returns one group for each named
dimension, in the order given. `$ne` asks for no group. The limit counts
distinct units, and it cuts the hits by rank before the groups order them. Each
unit hit in a group carries `dimension`, the key of its
group, and `display` when the unit has a mark. The reply's `dimension_groups`
lists each group with its `dimension`, `label`, `count`, mark counts and `units`
in display order. A unit that belongs to two requested dimensions is in full in
the first group, and a later group lists only its ref, with `shown_in` naming
the first group and `display` holding the later group's marks. A deep search's pack carries the same `display` on its units
and a `groups` list with each group's unit refs. The reply's `user_summary`
names each group by its `en` label, else by its first label in language-tag
order, and gives its counts, for example
"4 units in 'Uncertainty'; 1 is open."

The display behaviours that a dimension names set its group's order and marks.
Each one reads only relation behaviours, kind traits and governed keys, so a
vault dimension that names one needs no code change:

- `flag_open` marks a unit `open` when its kind has the `closable` kind trait,
  it has no `verdict`, and no edge whose relation has the `settles` behaviour
  ends at the unit or at its page.
- `lead_with_current` puts current units first and marks each replaced unit
  `replaced`, with `by` naming its successor when one is known. A unit is
  replaced when its page's lifecycle class is history, or when an edge whose
  relation has the `replaces` behaviour ends at the unit or at its page.
- `show_origin` gives each unit an `origin`. `by` is its `asserted_by` or
  `credence_holder`. `from` lists up to three targets of the edges that start at
  the unit or at its page and whose relation has the `sources` behaviour, and
  `more` counts the rest.
- `order_in_time` sorts the group by `occurred_at`, earliest first, with undated
  units after the dated ones. Each edge between two members whose relation has
  the `earlier_than_target` or `later_than_target` behaviour then orders its two
  ends. A unit that no such edge orders keeps its place in the date order
  wherever the edges allow. The edges inside a cycle decide nothing: its units
  keep the date order, and the group lists them in `order_conflict`.

A display behaviour changes only order and marks. It never removes a unit or
changes the unit's fields. Marks read only the edges that the caller may see, so
`open` means that nothing the caller can see settles the unit. When the edges
are unknown, no unit is marked `open`, no edge orders the group, the group
carries `edges: "unknown"`, and the summary says that its links could not be
read. The automatic working set (`recall_context`) does not group or mark units
yet.

The category report (`schema_memory` with `operation="infer"` and `subject="categories"`, which counts units by category) lists every
dimension with its members and display behaviours, and with `unit_count` and
`page_count` for the units it scanned. Its `edges` value says how the count was
made: `not_needed` when no edge decides the dimension, `counted` when the graph
answered, and `unavailable` when it could not. In an `unavailable` row, a count
that edges decide is `null` instead of a number that leaves out the edges. A row
also counts each mark that its dimension's display behaviours produce: `open`
where it names `flag_open`, and `replaced` where it names `lead_with_current`. While the graph is missing or
stale, the report schedules its rebuild, and the row's `repair` names the
`maintain_memory` call that runs the rebuild now. A dimension a vault adds appears
there with no code change.

## Explain ranking without inventing confidence

Set `explain=true` only when ranking interpretation is useful. The response
adds a versioned, bounded `retrieval_profile` and per-hit
`ranking_explanation`; the default response remains unchanged when explanation
is off.

The profile reports requested/effective modes and result levels, normalized
filters, participating and unavailable lanes, backend/model/metric metadata,
fusion constants, rerank decisions, and final ordering policy. Per-hit evidence
contains only lanes that actually participated.

The values are deliberately separate:

- BM25 is a backend relevance measurement with the direction/range declared by
  the profile.
- cosine is vector similarity, not probability.
- RRF contribution is exact rank-fusion math and exists only when fusion ran.
- reranker raw and adjusted values are separate when reranking ran.
- final rank follows the recorded boost, rerank, and deterministic tie-break
  chain.

None of these is a confidence score. A disabled, unavailable, warming, or
nonparticipating lane is reported at profile level and is never represented as
a fabricated per-hit zero.

## Read and traverse exact units

Unit-level recall returns an exact `unit_ref`. Pass it to `read_page` (in `options`) to read
the current unit with bounded parent context. Missing, stale, ambiguous, and
superseded references are explicit statuses; Exomem never substitutes a nearby
unit.

`connect_memory(operation="graph-context")` accepts an exact `unit_ref` or
registry-resolved `categories`/`kinds` filters. Compact and rich units appear as
derived graph nodes, but traversal follows authored edges only. Exomem does not
infer a typed edge from a compact observation's category or prose.

## Reviewed creation and adoption

For a validate/review/commit creation flow, call the intended writer with
`validate_only=true`, retain the returned `draft_id`, `draft_hash`,
`draft_token`, candidate, and semantic feedback, then commit the unchanged draft
through the same writer. If validation returns `reviewed_none_required=true`,
commit with `relation_disposition="reviewed_none"`, the returned
`relation_review_hash`, and an explicit `relation_review_reason`. Never invent a
reviewed-none decision from an empty Relations section. The legacy advertised
spelling `reviewed-none` is accepted at the boundary but stored canonically as
`reviewed_none`.

`adopt_vault(mode="scan-only")` includes a bounded read-only semantic census:
compact/rich counts, raw/canonical/resolved categories, collisions, malformed
candidates, contract/schema debt, relation dispositions, coverage, and safe
next actions. It never rewrites originals or requires categories.
`compile-selected` may copy selected legacy text into governed Sources and then
returns a proposal only. Review it and hand it to `create_note` (the `note()`
writer); normal semantic precommit remains mandatory.

Default initialization, scanning, reading, registry upgrades, and index
rebuilds do not fabricate observations, categories, relations, page IDs, or
review decisions. Existing pages are activation-snapshotted by stable ID or
portable path/hash fallback without rewriting their Markdown.

## Relation vocabulary workflow

Before authoring a typed relation, use
`connect_memory(operation="resolve-relation")` to inspect registered meanings.
Choose the most specific registered relation that is truthful. `relates_to` is
an honest generic connection; no edge is the right result when there is no
durable relationship. Resolution is evidence for the caller, never automatic
semantic selection or edge authoring.

For a durable missing distinction, use
`schema_memory(subject="relations", operation="propose-relation")`, review its
delta and `content_hash`, then submit the reviewed delta through
`schema_memory(subject="relations", operation="save-relations")` with
`expected_hash` and `why`. Save is hash-guarded; proposal and resolve are
read-only. For example, a reviewed synthetic `applies_to` relation can remain a
proposal until its explicit guarded save. Registry reuse, generic fallback, and
abstention do not require a registry change.