docs/architecture.md

Rama main · réplica de lectura del repositorio original.

# Architecture

kiwiki is a FastAPI application with a server-rendered Jinja2 front end. There is no SPA framework by design — the UI stays cheap to operate on small devices and friendly to incremental improvement. This document describes how a request flows, where code lives, and the conventions helpers follow. Read this before changing anything in `app/templates/`, `app/static/`, or the UI endpoints in `app/main.py`.

## High-Level Layout

```
Browser ──HTTP──► FastAPI (app/main.py)
                    │
                    ├── Middleware: request limits/IDs, rate limiter, security headers, CORS
                    ├── Auth: cookie session OR Bearer API key  (app/auth.py)
                    ├── Tenancy: ContextVar sets per-user namespace (app/tenancy.py)
                    │
                    ├── UI routes (/ui/*) → Jinja2 templates  (app/templates/)
                    ├── REST routes (/api/*) → JSON
                    ├── MCP routes (/mcp, /mcp/sse, /oauth/*)  (app/mcp_server.py)
                    ├── MCP Git sandbox helpers                (app/mcp_git.py)
                    └── Static files (/static/*)              (app/static/)
```

## Tenancy

Every request that touches storage runs inside a user namespace. `app/auth.py` calls `tenancy.set_user_ns(username)` after authenticating, which sets a `ContextVar` consumed by `app/storage.py` and `app/search.py`. This is why handlers can call `read_file(path)` without qualifying the path — the path is resolved against `/data/<username>/`.

Two users can never read each other's files, search each other's index, or share MCP state.

## Request Flow Example: Note View

1. Browser issues GET `/ui/file?path=notes/demo.md`
   (ohne `HX-Request`-Header und mit `text/html`-Accept antwortet der Endpoint
   mit 307 auf `/?file=…` — HTMX-Partials haben kein Layout, Deep-Links schon)
2. `_session_user(request)` (in `app/main.py`) reads the `kiwiki_session` cookie and returns a `User`
3. `set_user_ns(user.username)` is applied via the auth dependency
4. `read_file("notes/demo.md")` resolves to `/data/<user>/notes/demo.md` via `tenancy.user_root()`
5. The Markdown passes `_linkify_wikilinks` (`[[Ziel]]` → interne Links, Obsidian-Konvention:
   relativ zum Ordner, `.md`-Ergänzung, fehlende Ziele als `.missing`), then sanitized
   through `nh3` and rendered to HTML via `markdown`
6. `templates.TemplateResponse` renders `partials/file_view.html` with the context:
   - `title`, `title_redundant` (True, wenn der Frontmatter-Titel der ersten Body-H1
     gleicht — dann entfällt die View-H1, Breadcrumb + Meta bleiben), `path`,
     `updated`, `owner`, `tags`
   - `rendered` — sanitized HTML
   - `user` — the authenticated user (used by the template's role guard)
   - `can_delete` — write-or-admin flag for the delete button
   - `svg_edit`, `svg_trash` — pre-rendered icon strings
7. The partial is swapped into `#main-content` by HTMX (`loadFile()` in `kiwiki.js`)

The `user` key is the historically lost piece that caused the v2.1.1 "edit button missing" bug — the guard `{% if user and user.role in ['write','admin'] %}` evaluated to `Undefined -> False`. Always include `user` in templates that perform role-based rendering.

## Template Hierarchy

```
layout.html                   Base: <head>, header, sidebar slot, <main>, motion bundle
├── index.html                Home dashboard (hero, step grid, recent, FAB, file tree)
├── editor.html               /editor — Toast UI editor (own extra styles, own save flow)
├── settings.html             /settings — admin-only user management
└── login.html                /login — standalone (does NOT extend layout.html)
```

`layout.html` defines the `{% block sidebar %}`, `{% block content %}`, and `{% block extra_styles %}` slots used by index/editor/settings. `login.html` keeps its own CSS because it loads before any session exists.

## Startseite (`index.html`)

Layout-Reihenfolge im `{% block content %}`:

1. **Hero-Block** — kiwiki-Titel, Tagline („Dein persönlicher Wissensspeicher…"), „Neue Notiz"/„Tags"/„Suchverlauf"
2. **Dashboard** — Zwei Panels: „Zuletzt bearbeitet" + „Zuletzt erstellt" (je bis zu 8 Dateien, rekursiv via `list_all_files`)
3. **`<details class="home-help">`** — eingeklappt: „Erste Schritte" (4 Info-Karten) und „MCP-Verbindung"
   (POST Streamable HTTP + GET SSE)

Die Reihenfolge ist Absicht: wer die Startseite öffnet, will seine Notizen sehen, nicht erklärt bekommen, was
Markdown ist. Das Onboarding bleibt vollständig erhalten, nimmt der täglichen Arbeit aber nicht den Platz. Das
frühere Statuspanel (Format `.md`, Suche `FTS5`, Zugriff `<rolle>`) ist entfallen — drei Fakten, die sich nie
ändern und die Rolle steht ohnehin im Account-Menü.

HTMX-Endpoints für Dashboard:
- `/ui/recent-edited` → `partials/recent_edited.html` (sortiert nach `updated`-Frontmatter)
- `/ui/recent-created` → `partials/recent_created.html` (sortiert nach `created`-Frontmatter)

Sidebar (`{% block sidebar %}`):
- Dateibaum mit Filter + Multi-Select Toolbar (write/admin)
- Account-Menu Partial

Der Dateibaum ist die Hauptnavigation und wird deshalb offen gerendert. Sein Zustand liegt im Cookie
`kiwiki_sidebar` (`open`/`closed`), das `openSidebar()`/`closeSidebar()` im Desktop-Zweig setzen und
`template_language_context()` als `sidebar_open` in jedes Template gibt. Damit rendert der Server gleich
richtig — ohne Cookie wäre der Baum nach jedem Seitenwechsel wieder eingeklappt, und ein reines JS-Aufklappen
nach `DOMContentLoaded` wäre als Sprung sichtbar. Auf Mobile hält CSS die Sidebar unabhängig davon geschlossen
(`transform: translateX(-100%)`); `kwSyncSidebarForViewport()` setzt dort `inert`/`aria-hidden` nach.

## Partials (HTMX swaps)

| Partial | Rendered by | Swapped into |
|---|---|---|
| `partials/file_tree.html` | `GET /ui/files?path=…` | `#file-tree` |
| `partials/file_view.html` | `GET /ui/file?path=…` | `#main-content` |
| `partials/search_results.html` | `POST /ui/search` | `#search-results` |
| `partials/recent_edited.html` / `partials/recent_created.html` | `GET /ui/recent-edited`, `GET /ui/recent-created` | `#recent-edited`, `#recent-created` (mit `excerpt` + `tags` je Eintrag, max. 8, Dateien >100 KB ohne Ausschnitt) |
| `partials/tag_cloud.html` | `GET /ui/tags?compact=1` | `.home-tags` (Top-12-Chips; leer → `""`, Sektion blendet sich per `:empty` aus) |
| `partials/tags_overview.html` | `GET /ui/tags` | Subview mit Breadcrumb + Dateilisten |
| `partials/sidebar_account.html` | included by `index.html` and `editor.html` | `.sidebar-account` |

Partials must be self-contained — they cannot rely on `<script>` tags or external `<style>` from their parent page. All interactivity for partials lives in `kiwiki.js`, which is loaded once in `layout.html`.

## Static Assets

| File | Role |
|---|---|
| `app/static/kiwiki.css` | Single stylesheet, one `:root` token source, mobile breakpoints |
| `app/static/kiwiki-polish.css` | Feinschliff-Schicht: Primitive (Spacing, Control-Höhen, Fokus) und gezielte Korrekturen. Lädt nach `kiwiki.css` und überschreibt dort, wo die Basis nicht angefasst werden soll |
| `app/static/kiwiki.js` | All UI logic: sidebar, tree state, dialogs, toasts, swipe, FAB |
| `app/static/kiwiki-motion.bundle.js` | Built by `npm run build:motion` (Animate API entrance animations) |
| `app/static/kiwiki-fonts.css` | Local font-face declarations |
| `app/static/vendor/` | Vendored htmx, Toast UI, fonts and upstream license notices |

The cache-busting query strings in `layout.html` (`?v=20260701-a11y`) must be bumped whenever `kiwiki.css` or `kiwiki.js` change semantically. Otherwise users still get the old version from cache and report "works after hard refresh" bugs.

## Search

`app/search.py` stores one FTS5 database per user under `/data/<username>/.kiwiki/index.sqlite`. The `search()` function accepts plain queries and the special prefix `tag:<value>`:

```python
# Plain full-text search
search("markdown rendering")

# Prefix-triggered tag search (LIKE on the tags column)
search("tag:python")
```

The prefix path sidesteps FTS5 column filters, which are brittle in SQLite's FTS5. Use it whenever you need structured tag filtering from the UI.

`_clean_snippet(content, query)` builds the excerpt shown in the result list: it strips frontmatter, heading
markers, list bullets, emphasis, table pipes and link targets, then centres a window on the first occurrence of
the query term. The previous implementation returned `content[:200]`, which usually showed frontmatter and never
the part that actually matched.

Die Live-Suche im Header hängt an `hx-trigger="input delay:400ms, submit"`. Der Modifier `changed` darf dort
**nicht** stehen: htmx vergleicht dafür `elt.value` des Trigger-Elements, und ein `<form>` hat keines — der
Vergleich ist immer `undefined === undefined` und verwirft jedes Event. Das hat die Live-Suche bis v4.0.0
vollständig stillgelegt; `tests/test_ui_feinschliff.py` sichert die Trigger-Definition ab.

## Optional Knowledge Engine

`app/indexing.py` is the single mutation facade for derived indices. It updates the
existing FTS database synchronously and, when `KIWIKI_KNOWLEDGE_ENABLED=true`, adds
an idempotent job to the tenant's `.kiwiki/knowledge.sqlite` queue. The background
worker explicitly sets and resets the tenant `ContextVar`, processes users fairly in
bounded batches, and never participates in `/readyz`.

Markdown remains the source of truth. `app/knowledge/extract.py` derives document,
tag and link facts deterministically; every relation stores its source path and
revision. `PRAGMA user_version` drives forward-only schema migrations, while an
unknown newer version fails closed without modifying the database. On first enable
or restart, reconciliation compares workspace revisions, queues missing/up-to-date
operations, and never changes Markdown bytes. See
[`knowledge-engine-plan.md`](knowledge-engine-plan.md) for rollout and rollback details.

`GET /api/knowledge/graph` projects this derived data into a bounded visualization
contract (maximum 800 nodes and 2,000 edges). `/knowledge` renders it locally through
`app/static/knowledge-graph.js`; no graph payload leaves the browser origin. The
renderer uses a deterministic 3D layout projected onto an accessible canvas, with a
DOM-based inspector for readable metadata and source navigation. The UI remains
functional as an explicit empty state while the engine is disabled or backfilling.
On every visit the atlas plays a short showcase (build-up, data packets along edges,
a decorative synapse field around small graphs) and then fades to rest; motion
rules and tuning constants are documented in [`ui-system.md`](ui-system.md#neuronaler-atlas-bewegung).

## Design-Rollen: Typografie und Abstände

Beide Maßsysteme liegen als Tokens in `app/static/kiwiki-polish.css` unter `:root`. **Nicht nach `kiwiki.css`
verschieben** — `login.html` lädt `kiwiki.css` nicht (es bringt seine Farben selbst mit und bindet nur das
Polish-Stylesheet ein) und fällt sonst auf Browser-Defaults zurück.

| Rolle | Größe | Wofür |
|---|---|---|
| `--text-xs` | 11 px | Zähler, Badges, Verb-Chips, Initialen |
| `--text-sm` | 12 px | Meta, Section-Labels, Pfade, Captions |
| `--text-md` | 13 px | UI-Standard: Buttons, Dateibaum, Listen |
| `--text-lg` | 15 px | Lesetext, Fließtext |
| `--text-xl` | 18 px | Abschnitts- und Paneltitel |
| `--text-2xl` | 24 px | Titel einer Notiz, Dialogtitel |
| `--text-3xl` | 32 px | Seitentitel: Einstellungen, Login, `h1` |
| `--text-display` | `clamp(1.9rem, 3.4vw, 2.5rem)` | Hero |

Abstände: `--space-0-5` (2 px), dann `--space-1` bis `--space-16` als Vielfache von 4 (4/8/12/16/24/32/48/64).

Zwischen `--text-md` und `--text-lg` sowie zwischen `lg` und `xl` sitzt bewusst eine Lücke. Vorher trugen die
Stylesheets **37 Schriftgrößen** und **58 Abstandswerte**, 71 % davon zwischen den Stufen — `rem`-Brüche wie
`0.45rem` ergeben krumme Pixel, und acht Größen zwischen 10.4 und 13.44 px sind keine Hierarchie, sondern
Streuung. Schriftgewichte sind auf 400/500/600/700 beschränkt; Werte wie 620 oder 650 sind Artefakte einer
variablen Schrift, die niemand bewusst wählt.

`tests/test_design_tokens.py` hält die Skala geschlossen: rohe `font-size`-, `padding`-, `margin`- und
`gap`-Werte lassen die Suite fehlschlagen. Zwei begründete Ausnahmen stehen dort explizit:

- `font-size: 16px` auf Eingabefeldern — die Schwelle, unterhalb der iOS beim Fokussieren hineinzoomt.
- Icon-Abstände in Feldern als `calc(14px + 2 * var(--space-3))` — die Rechnung bleibt sichtbar, statt als
  krumme `rem`-Zahl zu erscheinen.

Das CSS des Toast-UI-Editors gehört nicht zu diesem System und wird nicht angefasst.

## Zeitstempel in Templates

Frontmatter liefert naive lokale Zeitstempel. Roh ausgegeben (`2026-09-19T17:42:00`) sind sie in Listen unlesbar
und optisch dominanter als der Notiztitel daneben. `app/i18n.py` stellt dafür zwei Funktionen bereit, die
`app/main.py` als Jinja-Filter registriert:

| Filter | Ergebnis |
|---|---|
| `{{ f.updated\|kwdate }}` | „Heute, 09:31", „Gestern, 18:25", „vor 3 Tagen", „19. Sep.", „7. März 2021" |
| `{{ f.updated\|kwdate_full }}` | `19.09.2026, 17:42` — für `title=` neben der Kurzform |

`kwdate` ist ein `@pass_context`-Filter und liest `lang` aus dem Template-Kontext. Unparsbare Werte gibt er
unverändert zurück: Frontmatter darf beliebigen Text enthalten, der darf nicht verschwinden. Neue Templates, die
ein Datum anzeigen, nutzen `<time title="{{ x|kwdate_full }}">{{ x|kwdate }}</time>`.

## JS Helper Conventions

All global helpers in `kiwiki.js` are namespaced with the `kw` prefix (`kwToast`, `kwDialog`, `kwNewNote`, `kwSearchTag`, `kwToggleSelect`, …). Legacy helpers (`loadFile`, `openEditor`, `toggleFolder`, `deleteFile`) keep their original names for backward compatibility with templates but should not be extended — prefer `kw*` for new helpers.

Helpers that need to change frontmatter (tags, metadata) must go through `PATCH /api/file/frontmatter` (`storage.update_frontmatter()` server-side) instead of reconstructing the frontmatter block client-side with string/regex manipulation. `kwBatchTag()` used to rebuild the whole `---\n...\n---` block via regex directly on `fc.content` — it matched inconsistently (the `/api/file` response already strips frontmatter from `content`) and, when it did match, wiped out every other frontmatter field (`title`, `created`, `updated`). The server-side merge only touches the keys it's given.

State-sensitive HTMX reads use `kwHtmxGet(path, target)`. It creates a request-specific source element and resolves
from `htmx:afterRequest.successful`, because the promise returned by HTMX 1.9 may resolve before a queued request has
actually run and also resolves for HTTP error responses. File URLs, active markers and persisted open-folder state
must only change after `kwHtmxGet()` reports success. The global `htmx:responseError` handler is responsible for
visible retry/error feedback.

Tree state (open folders, active file, scroll position) is persisted in `localStorage` under these keys:

| Key | Purpose |
|---|---|
| `kiwiki:openFolders` | Array of open folder paths |
| `kiwiki:activeFile` | Last opened file path |
| `kiwiki:treeScroll` | Tree last scroll position |
| `kiwiki_sidebar_w` | Desktop sidebar width (resize handle) |

Offen/Zu der Sidebar liegt bewusst **nicht** im `localStorage`, sondern im Cookie `kiwiki_sidebar` — nur so kann
der Server das Markup beim ersten Byte richtig rendern.

## Roles & Visibility

Role checks happen in three layers and must align — a missing layer causes silent UI bugs:

1. **Server**: `Depends(require_role("admin"))` on the FastAPI route (HTTP 403)
2. **Template**: `{% if user and user.role in ['write','admin'] %}` (HTML visibility)
3. **Client**: `kwCanWrite()` / `kwCanAdmin()` read from `window.KIWIKI.roleLevel`, set in `layout.html` from the session cookie

If any layer is skipped, users see UI they cannot use (or vice versa). The `user` key in the template context is the bridge between server and template — always pass it.

## Rate Limiting

Browser fragments have their own `ui` tier (`KIWIKI_UI_LIMIT`, default 240 requests/minute per client IP). It covers
GET requests below `/ui/` and the read-only live-search POST `/ui/search`. Mutating UI POSTs remain in the `write`
tier, while REST/MCP reads use `read`. This separation prevents the initial file tree, dashboard fragments and
persisted folder restoration from exhausting the lower programmatic-read budget.

Behind a reverse proxy, enable `KIWIKI_TRUST_PROXY=true` only together with `KIWIKI_TRUSTED_PROXY_CIDRS`; otherwise
all users share the proxy peer IP and therefore one limiter window.

## OAuth Token Lifecycle

OAuth tokens are self-contained signed tokens (no server-side session): access tokens live 24h
(`KIWIKI_OAUTH_TOKEN_TTL_SECONDS`), refresh tokens 30 days (`KIWIKI_OAUTH_REFRESH_TOKEN_TTL_SECONDS`).
Refresh is single-use with reuse detection — each successful refresh returns a rotated refresh token and the
presented one is consumed (tracked by `jti` in `_oauth_refresh_seen`, RAM-only like codes and DCR clients).
Re-presenting a consumed token yields `invalid_grant`. Only the success path consumes; failed client/resource
binding checks do not burn the token.

Dynamic client registration (`POST /oauth/register`) is intentionally open but bounded twice: 128 clients per
process (`KIWIKI_OAUTH_MAX_CLIENTS`, 24h TTL) and 16 registrations per source IP per hour
(`KIWIKI_OAUTH_MAX_REGISTER_PER_IP`), so one source cannot fill all slots and lock legitimate clients into 503.
`KIWIKI_BASE_URL` must be the public URL — without it, MCP `search`/`fetch` citation URLs stay relative
(`/ui/file?path=...`) and are useless to OpenAI connectors; the app logs a startup warning when it is unset.

## Testing

| Area | Tests | Runner |
|---|---|---|
| Auth, storage, search, MCP | `tests/test_*.py` (per module) | `pytest` |
| MCP schema conformance | `tests/test_mcp_conformance.py` | `pytest` (needs `jsonschema`) |
| UI rendering regression | `tests/test_ui_file.py` | `pytest` (uses `TestClient`) |
| Browser/responsive regression | `tests/browser_smoke.py` | Playwright Chromium |
| Lint | All `app/` and `tests/` | `ruff check app tests` |
| Frontend bundle | `frontend/motion/` | `npm run build:motion` |
| Container | `Dockerfile` | `docker build -t kiwiki:test .` |

`tests/test_mcp_conformance.py` calls every MCP tool once and validates the result against that tool's own
`outputSchema`. It exists because the older tests only asserted that a schema was *present* — several tools
returned fields their schema forbade, and since protocol revision 2025-06-18 clients validate `structuredContent`
and discard the whole response on the first mismatch. The seeded workspace deliberately uses awkward but legal
frontmatter (`title: 2026` is an int, `tags: python` a scalar, unquoted dates become `datetime.date`, `.nan` is
not representable in JSON), and results are parsed with a `parse_constant` hook so `NaN`/`Infinity` literals fail.
`_TOOL_ARGS` must cover every tool — adding a tool without an entry fails the suite, so the sweep cannot rot.

UI regression tests combine fast template assertions with a real Chromium smoke path. The browser gate covers mobile sidebar focus/inert behavior, zoom, stable note deep links and titles, responsive settings, and overflow. Add browser assertions whenever behavior depends on layout, focus, history, or JavaScript timing.

## Adding a New Helper

1. **Name**: `kw<Action>` in `app/static/kiwiki.js`
2. **Role check**: `if (!kwCanWrite()) return;` at the top if the action needs write/admin
3. **Feedback**: Use `kwToast(msg)` or `kwToast(msg, {type: 'error'})` for all outcomes (never `alert()`)
4. **Localization**: Every new or changed user-facing string, ARIA label and dynamic
   browser message ships in German and English. Explicit choice is persisted, with
   `Accept-Language` as fallback; code identifiers remain English.
5. **Test**: Add a regression test in `tests/test_*.py` — render the partial and assert on the rendered HTML
6. **Cache-bust**: Bump `?v=…` in `layout.html` for both `kiwiki.css` and `kiwiki.js`
7. **Docs**: Update [docs/ui-accessibility.md](ui-accessibility.md) if the change affects keyboard, ARIA, or touch behavior

## Deployment

- Local: `docker compose up -d` (builds the image, mounts `./data`, exposes `:8082`)
- Helm: `charts/kiwiki/` for Kubernetes; review `values.yaml` before production
- Docker Hub: `natorus87/kiwiki:<tag>` — follow the `docker-push` skill workflow

For releases, the `bereitstellung` skill defines the full release flow: clean workspace → tests → build → changelog → tag → push → GitHub release.

## Related Docs

- [ui-accessibility.md](ui-accessibility.md) — WCAG, keyboard model, touch targets, verification checklist
- [../CHANGELOG.md](../CHANGELOG.md) — version history
- [../CONTRIBUTING.md](../CONTRIBUTING.md) — workflow, commit format, UI checklist for PRs
- [../README.md](../README.md) — user-facing overview, configuration, MCP tools