BookOfTlamatini.md
Rama main · réplica de lectura del repositorio original.
<!--
═══════════════════════════════════════════════════════════════════
✦ T L A M A T I N I ✦ — "one who knows"
Created by Angela López Mendoza · @angelahack1
Developer · Architect · Creator of Tlamatini
Tlamatini Author Banner — do not remove (Angela's name is kept in every build)
═══════════════════════════════════════════════════════════════════
-->
# Tlamatini
> Latest released version: **[v1.92.4](docs/changes/2026-11-08-v1.92.4.md)**.
## v1.92.4 — Matching builds, explicit release identity and readable Markdown
The combined builder predicts both editions’ actual emitted version before building, pins one matching number, runs Private/Public pairs in parallel with `--self-modify`, and archives verified ZIPs by edition, type and minute. The application log identifies its embedded release type. Dependency setup stays in isolated, application-owned caches and stops safely when no healthy environment exists. Readable Markdown with occasional control characters opens without losing its original bytes; binary content is still refused. See the [1.92.4 release notes](docs/changes/2026-11-08-v1.92.4.md).
> **The Book of Tlamatini** — a step-by-step guide to running, using, and mastering a locally-deployed AI developer assistant with RAG, Multi-Turn tool orchestration, ACPX external-CLI delegation, an Unreal MCP client for driving Unreal Engine 5 from chat or canvas, a visual workflow designer, 87 drag-and-drop agent types, and a backend Flow Compiler that turns the live canvas — or a chat-generated tool-call log — into a registry-validated, secret-redacted, source-and-frozen-portable workflow.
>
> Visit our site at **https://xaiht.org**, or get a one-minute taste of Tlamatini on YouTube: **https://youtu.be/a51miZ1JIe0**.
>
> 💬 **Join the community on Discord:** **https://discord.gg/WFQsrskgc** — get help, show what you build, report bugs, and help shape the roadmap.
---
## ⚠️ CLEAR DISCLAIMER — USER CONTROL, JURISDICTION, AND RESPONSIBILITY FOR AGENTS
Every agent in `Tlamatini/agent/agents/` is intentionally provided as a **plain-Python program** so its operating code can be read, audited, edited, restricted, or disabled by the user. This transparency is a user-control mechanism, **not a warranty that an agent is secure or suitable for a particular environment**. The agents do not have independent authority or jurisdiction: the user alone decides whether, where, how, and with which permissions they run.
When you enable, configure, modify, chain, or execute an agent, **that agent and its execution are under your control and your jurisdiction**. You are solely responsible for reviewing its code and configuration; protecting and limiting its secrets, credentials, and permissions; selecting and authorizing every file, folder, network target, browser, shell, API, external MCP server, machine, hardware device, and downstream system it can access; supervising its output; and complying with every law, policy, license, contract, and authorization that applies to your use.
**BY RUNNING AN AGENT, YOU ACCEPT RESPONSIBILITY FOR ITS ACTIONS AND CONSEQUENCES. TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, ANY SECURITY BREACH, DATA EXPOSURE OR LOSS, UNAUTHORIZED ACTION, CREDENTIAL LEAK, UNSAFE AUTOMATION, POLICY OR LEGAL VIOLATION, SYSTEM COMPROMISE, DEVICE DAMAGE, FINANCIAL LOSS, OR OTHER HARM ARISING FROM YOUR USE, CONFIGURATION, MODIFICATION, OR EXECUTION OF AN AGENT OR AGENT WORKFLOW IS THE RESPONSIBILITY OF THE USER WHO RUNS IT.** Tlamatini's orchestration, documentation, examples, and guardrails do not authorize access to third-party systems and cannot replace the user's own security review, permission controls, monitoring, or legal compliance.
---
## ⭐ Start here — five steps to a cloud-powered Tlamatini
Before any of the deep chapters, here is the whole journey on a single page. It is the first and most important thing you will do with Tlamatini, so it comes first.
Tlamatini itself is free and open-source, but the complete system was designed and coded primarily around the larger cloud models exposed through Ollama. **To use Tlamatini with its complete intended functionality, an active Ollama Pro plan — or a higher Ollama plan such as Max — is mandatory.** A free Ollama account or an all-local model configuration may be enough to launch the application and experiment with limited work, but neither is the supported baseline for the full Multi-Turn, long-context, parallel-vision, and agent-orchestration workload described in this book.
> [!IMPORTANT]
> **Ollama Pro is a Tlamatini system requirement, not a sponsorship.** XAIHT/Tlamatini is not sponsored, endorsed, affiliated with, funded by, or compensated by Ollama. We receive no referral fee or commission. The subscription is purchased directly from Ollama; it is specified here only because Tlamatini's architecture depends on the practical cloud usage and concurrency that Pro-or-higher service provides.
Ollama currently advertises Pro at **$20/month or $200/year**, with substantially more cloud usage and concurrency than a free account. Pricing and limits are controlled by Ollama and can change, so verify the current terms on **[Ollama's official site](https://ollama.com)** before subscribing.
Five steps take you from a bare machine to a Tlamatini that can flash a board, drive an engine, and run a whole workflow unattended.
### Step one — Install Tlamatini
There are two ways into Tlamatini, and they suit two different readers. The app itself is **free** — you never pay us a cent; the required external service cost in this chapter is paid directly to Ollama, three steps from now. Optional third-party APIs and external coding-agent services may have their own independent charges. Choose **one** road.
**🟢 Option A — the release installer (recommended; no Python required).** This is the gentle path, and the right one for most people. Open the **[Releases page](https://github.com/XAIHT/Tlamatini/releases)**, download the latest installer, and run it. It carries its own Python 3.12.10 and every dependency inside, so there is nothing else to install. Launch Tlamatini from its Start-menu shortcut and your browser opens at `http://127.0.0.1:8000/`, behind the default login **user / changeme**. When a new version lands, you update from inside the app — **About ▸ Check for updates** — and it keeps your config, database, and keys intact.
**🔵 Option B — from source (for developers).** Take this road if you mean to read, modify, or contribute to her code; it asks that **Python 3.12.10** and **git** already be on your machine. Six commands suffice:
```bash
git clone https://github.com/XAIHT/Tlamatini.git
cd Tlamatini
python -m venv venv && venv\Scripts\activate
pip install -r requirements.txt
python Tlamatini/manage.py migrate
python Tlamatini/manage.py runserver --noreload
```
> **`--noreload` is optional (since 2026-07-11):** plain `python Tlamatini/manage.py runserver` now boots clean and auto-reloads on code edits. It used to double-start the MCP helper ports `:8765` / `:50051` and crash with `WinError 10048`; fixed by a reloader-aware gate in `agent/apps.py`.
Then open `http://127.0.0.1:8000/` and log in with **user / changeme**.
*(Part I, §3–§7 walks both routes in full; this page is the map, not the territory.)*
### Step two — Install Ollama
Tlamatini never talks to a model directly; she speaks through **[Ollama](https://ollama.com/download)**, the engine that serves every model — the small embedding model that lives on your own GPU, and the large chat models that live in the cloud. Install it once.
### Step three — Activate Ollama Pro or higher (required for complete operation)
Sign in at **[ollama.com](https://ollama.com)** and activate **at least Ollama Pro**; Max also satisfies the requirement. Ollama may allow limited cloud use on a free account, but Tlamatini does not treat that allowance as a completely functional production configuration: a single Multi-Turn task can make repeated model decisions and may involve several cloud-backed subsystems. Pro is therefore the minimum intended capacity tier. Then introduce your machine to your account:
```bash
ollama signin
```
### Step four — Download the models
Pull the small embedding model onto your own disk, and reach for the cloud chat models you intend to use:
```bash
# Local — small, runs on your own GPU/CPU
ollama pull nomic-embed-text
# Cloud — use a signed-in Pro-or-higher account for full Tlamatini operation
ollama pull deepseek-v4.1-flash:cloud
ollama pull jcyhsiao/qwen3.5cloud:latest
```
Any cloud model will do; the two above are the current recommended pair (screenshots that follow may still show earlier model names).
### Step five — Point Tlamatini at the models
The final step lives inside Tlamatini's own interface. In the navbar, open the **Config** menu — it holds three doors: *Models*, *URLs*, and the *Access Keys Wizard*.
Behind **Config ▸ Models** are 38 settings in six searchable categories: Core, Vision, Speech, Workflows, Documents, and Monitoring & messaging. Set the models for all 21 model-backed agents, including Talker and Whisperer. Ollama choices use catalog suggestions; Whisper and cloud-provider models use their own identifiers. Save, reconnect chat, and restart running agents as needed. Agent YAML values marked `@config` inherit these choices; explicit per-agent values remain overrides. See [the full model configuration contract](docs/model_configuration.md).
The current dialog has six category tabs, searchable fields and dedicated engine/voice selectors. [Open the complete Models guide](docs/model_configuration.md#use-the-dialog).
Behind **Config ▸ Access Keys Wizard** the cloud connection is sealed — and here a single, important distinction decides whether you type anything at all. **If Ollama runs on your own machine (the usual `localhost`), you need no Ollama token — leave that field blank;** a local Ollama answers without a password. **Only when Ollama lives on a remote server — a rented GPU box on [Vast.ai](https://vast.ai), say — do you paste an Ollama token here,** so Tlamatini can authenticate to it across the network. Add any cloud-CLI keys you care to in the same place; blank fields are left untouched, so you only ever type what you mean to change. **Save**, and the wizard tallies how many of its slots are filled.
That is the whole setup. Tick **Multi-Turn** in the chat toolbar and hand Tlamatini her first real task. Everything past this chapter is depth — but you are already running.
---
## A note on the settings you will never touch
Underneath Config ▸ Models sit a handful of numbers nobody normally sees, and one of them was quietly breaking Tlamatini until September 2026. It is worth one page, because the story explains why you should trust the values that ship and leave them alone.
The symptom was an answer that said nothing: *"The tool-calling model returned an empty final response."* It arrived after about six minutes of silence, which felt like a crash but was not one. The cause was `ollama_repeat_penalty` — the dial that tells a model how strongly to avoid reusing words it has already used. It shipped at **1.9**, and nothing anywhere recommended that. Ollama's own default is 1.1, the reference implementations use 1.0, and the tuning guide everyone cites says to raise it gradually and stop at about 1.2.
Why did a too-high dial produce *silence* rather than strange prose? Because the models Tlamatini talks to now are **reasoning** models. They think privately first, in a stream you never see, and only then write an answer. Push the dial too hard and the thinking can never settle — every word that would round off a thought is the very word being penalised. So the model thinks, and thinks. One measured run produced **196,003 characters** of private reasoning in 287 seconds and then stopped with nothing written. A later one produced **3.1 million characters over fifty-one minutes**. From the outside: a long pause, then an empty reply.
Finding it took longer than it should have, for a reason worth remembering. The failure is **intermittent**. Six test runs of the bad setting came back perfectly clean; eight runs of the identical setting failed seven times. Nothing had changed between them. These are enormous models served from the cloud, and which machines answer your request varies from call to call. Four different theories were tested and discarded — the size of the tool list, the difficulty of the question, the context window, even the penalty itself measured the wrong way — each of which looked proven on a single run and each of which was wrong. **One run proves nothing here.**
The penalty now ships at **1.2**, and on the flagship model that took failures from seven in eight down to one in eight. Two neighbours were corrected at the same time: `ollama_repeat_last_n`, which decides how far back the penalty reaches, was never actually reaching Ollama at all — it is now wired, and set to a value that made the same model six times faster — and `ollama_num_ctx`, which now reads **1,048,576** instead of a smaller number that described nothing real. That last one is honesty rather than power. On a cloud model the server enforces its own limit regardless of what Tlamatini asks for, and it enforces it *well*: exceed it and you get a clear error naming both your size and the maximum, never a conversation quietly trimmed behind your back. While checking, Tlamatini was asked to find one buried fact inside a prompt of **660,000 tokens**. She found it every time.
The lesson generalises past these three numbers. A default with no reason recorded beside it is a bug waiting for someone to inherit it — so every one of these values now carries its measurement in the project's history, and the one remaining gap is written down too: the library Tlamatini uses to reach Ollama throws away the model's private reasoning, which is exactly why a runaway looks like an empty answer instead of an honest *"still thinking"*.
---
## How to read this book
Tlamatini does a lot. This book is organized so you can stop reading at the depth you need.
- **⭐ Start here** (the chapter just above): the entire five-step setup — install, Ollama, Ollama Pro, models, config — on a single page. *If you read nothing else, read this.*
- **Part I — Getting Tlamatini Running**: prerequisites, Ollama, the **Pro-or-higher requirement for complete functionality**, cloud models, installation, and first login. *Read this once.*
- **Part II — Using the Chat**: the five toolbar checkboxes (Multi-Turn, Exec Report, ACPX, Ask Execs, internet) walked through one by one. *This is the dummy-friendly heart of the book.*
- **Part III — The Visual Workflow Designer**: drag-and-drop flows, FlowCreator, FlowHypervisor, Parametrizer.
- **Part IV — The Tlamatini Bestiary**: compact reference covering all 87 workflow agents, with one authoritative row per live agent directory.
- **Part V — The Tool Surface**: every LLM-facing tool the chat can call, organized by family.
- **Part VI — Inside Tlamatini**: architecture, RAG, the embedding-memory pre-flight guard, Multi-Turn pipeline, ACPX runtime mechanics. *For the curious.*
- **Part VII — Configuration Reference**: every `config.json` knob.
- **Part VIII — Deploying & Packaging**: build, installer, frozen mode.
- **Part IX — The Command Deck**: WebSocket protocol, HTTP endpoints.
- **Part X — Survival Guide**: troubleshooting, `tlamatini.log`, common issues.
- **Bonus chapter §57** — Driving Unreal Engine 5 from Tlamatini (the Unrealer agent + Unreal MCP plugin). Read this if you build games or simulations in UE5 and want a chat / canvas surface for the editor.
- **Bonus chapter §59** — Sculpting in Blender from Tlamatini (the Blenderer agent + the official Blender MCP add-on). Read this if you make 3D art / assets in Blender and want a chat / canvas surface for the editor — and to see why Blender's *code-execution* protocol differs from Unreal's verbs.
- **[The Prompt Flow Panel: draw a chain of prompts, then press Play](#the-prompt-flow-panel-draw-a-chain-of-prompts-then-press-play)** — the 2026-09-25 sibling of the Agentic Control Panel: eight executable operations, including Terminater, plus static review bubbles, `.fpmt` files (and why `.pmt` stays with the system prompt), running, pausing and stopping, and what a run may and may not touch.
- **[Enable Tlamatini as a Blue-hat agent](#enable-tlamatini-as-a-blue-hat-agent)** — the complete Windows defensive-toolkit runbook: asset validation, persistent host changes, detect-only baselining, armed/watch modes, evidence review, false positives, response rollback, and operator responsibility.
- **[Voice commands: your words become the prompt](#voice-commands-your-words-become-the-prompt)** — the catalog's new first section: speak your instruction instead of typing it, why it needed the silence gate to exist, and the two promises (she reads it back; if she did not hear you she says so).
- **[Whisperer listens: the silence gate](#whisperer-listens-the-silence-gate)** — why a fixed recording length was the wrong question, how the gate hears you, the countdown in the console bar, and the one thing that would have made the whole feature invisible.
- **[Terminals you can watch: Terminater](#terminals-you-can-watch-terminater)** — the visible Executer: real CMD and PowerShell terminals in the Terminals area, typed key by key like a person, with a hand-over, an Esc that gives you the keyboard back, hung-command detection and a safe three-card tour in the catalog. *New in v1.88.0.*
- **[Avatar animation repair and reproducible visible tests](#avatar-animation-repair-and-reproducible-visible-tests)** — frame assets, the transparency-flash diagnosis, the source fix, real Django/Windows-voice validation, and the Python test launcher.
- **Appendix A** — Keyboarder key reference.
- **Appendix B** — Glossary.
- **Appendix C** — Full changelog (preserved verbatim).
- **Appendix D** — Acknowledgments / Contributing / License.
If you only have ten minutes, read Part I §3–§7 (install + first login), then Part II §12 (Multi-Turn).
---
## The Prompt Flow Panel: draw a chain of prompts, then press Play
### Why a second canvas
The Agentic Control Panel wires **agents** together: a Starter, an Executer, a Shoter, an Ender. But a lot of real work is a *conversation* with a shape — ask something, look at the answer, go one way or another, ask a follow-up, stop and check with a human. The **Prompt Flow Panel** (2026-09-25) is a canvas for exactly that: you draw the conversation once, and Tlamatini walks it for you.
Open it from the chat with **Panels ▸ Prompt Flow Panel**, or from the Agentic Control Panel with **File ▸ Prompt Flow Panel**. It opens in its own tab.
### Eight operations and static review bubbles
| Operation | Shape | What happens when the flow reaches it |
|---|---|---|
| **Prompt** | rectangle | Sends its text to Tlamatini. You can switch on **Multi-Turn** (and, with it, **ACPX**) for that one step, so a step can use every tool and agent she has. |
| **Programmed Prompt** | rectangle with a clock | The same, but first waits a number of seconds, or until a date and time you choose. |
| **Decision** | diamond | Takes the **Y** or **N** branch: does the last answer *contain*, *not contain* or *equal* some text, or is it *empty*? Or it can simply ask you. |
| **Feed embeddings** | triangle pointing up | Adds reference text to this run, so the following prompts can draw on it. |
| **Flush embeddings** | triangle pointing down | Takes that reference text away again, keeping the conversation. |
| **Clean History** | trapezoid | Forgets this run's conversation so far, keeping the reference text. |
| **User Input** | notched top and downward point | Stops and asks you something; your reply joins the conversation. |
| **Torrenter** | Inspect a magnet, select files, and follow an account-owned Jobs transfer to its verified manifest. [Guide](docs/torrenter.md). |
| **Terminater** | parallelogram | Runs one action in your visible CMD or PowerShell terminal; its output becomes `{{last_output}}`. Stop aborts typing and the running command. |
| **User Commentary** | speech bubble | A static review note on the canvas; it never runs or changes the conversation. |
In executable operation text, `{{last_output}}` inserts the previous answer (or your last reply). Inside a static User Commentary, that token stays literal. A Decision only ever compares text — nothing in a flow file is ever run as code.
### Building and running a flow
1. Drag operations from the **Operations bar** onto the canvas, or just click one to drop it in.
2. Double-click an executable figure (or select **Configure**) to set its text and options. Double-click a User Commentary to write directly inside its bubble; select text and format it with the floating mini toolbar.
3. Drag a white **output triangle** on the right of an executable figure onto an **input triangle** on the left of another and release to connect them. Static commentary has no ports. A Decision has two outputs, **Y** and **N**; every output leads to exactly one place, and connecting it again replaces the old line.
4. Choose where the flow begins in the **Start** list, press **Validate**, then **Play**. Validate refuses a flow with a dead end it cannot explain: every executable operation must be reachable from Start, and every Decision needs both branches. Static comments need no connection and cannot be Start.
5. Watch the running figure light up and read everything in **Run output** below the canvas. **Pause** lets the current step finish and holds the next one; **Stop** cancels and waits for the current step to wind down cleanly. Loops are allowed — **Flow settings** caps how many steps one run may take (500 by default, 5,000 at most).
Keep the page — and Tlamatini — open while a flow is playing, especially one with a scheduled prompt. This is not a background scheduler. Closing a question dialog with **Cancel**, **✕** or **Esc** stops the flow; clicking outside it does not.
### Writing static User Commentaries
Add independent **User Commentary** bubbles within the 500-asset limit. Double-click a bubble or press Enter to write directly inside it. Select a word or passage, then use the **floating mini toolbar** to choose its font, text size, text color, bold, italic or underline. One note can mix styles: an Arial italic sentence, a large Verdana heading and a Georgia paragraph. With no text selected, a formatting choice sets the style for the next text you type. When the note is selected outside editing, formatting applies to the entire note. The color swatches change the bubble; the underlined **A** opens text colors. Alignment applies to the whole note.
Drag **any border or corner** to reshape the bubble; the handles also respond to arrow keys (Shift moves farther). Resizing works while writing. **Fit text** removes spare height. There are no numeric dimension fields or commentary configuration dialogs. **Edit comment** in the main toolbar or context menu starts the same inline editor. **Done** or Ctrl+Enter saves the complete edit; **Cancel** or Escape restores its original text, styles, size and position. Ctrl+Z/Y undoes/redoes changes while writing. Completed edits, moving, resizing, copying and deletion also support flow Undo/Redo.
Bubbles grow to contain every wrapped line at the chosen width and mixed fonts, without internal scrollbars. The chosen height is a minimum. Text stays literal, including `<script>` and `{{last_output}}`. Notes have no ports or Start status and never enter model context, history or playback steps. Notes save/open and recover with the diagram. The palettes offer eight bubble colors and eight text colors; fonts are Nunito, Arial, Verdana, Georgia, Times New Roman and Courier New. Text sizes range from 10 to 48, with named choices from Fine through Display. Each note holds up to 100,000 characters and 10,000 formatted runs. Internal geometry limits are width 200–2,400 and minimum height 128–2,400; automatic containment can grow taller. A notes-only diagram can be saved; Play requires an executable operation.
The separate **User Input** operation keeps the former question-and-reply behavior and uses the notched figure. Closing its reply dialog stops the flow.
### Resizing the canvas and Run output
Drag the horizontal divider above **Run output** to give the log **5%–95% of the available canvas/output height**, excluding headers, status and the divider. The starting share is 20%. Both panes scroll independently, keeping text size, figures and canvas zoom unchanged. Tab to the divider and use Up/Down for one percentage point, Shift+Up/Down for five, Home for 5%, or End for 95%. Escape ends a drag. Collapse the output by clicking its heading; reopening restores its height. Window resizing preserves the ratio, and the browser remembers the layout per user. Layout preferences do not change your flow file, its saved draft or Undo/Redo history.
### What a run may and may not touch
Every run is its own little world. It gets its **own conversation** and its **own reference text**, starts empty, and throws both away when it ends. It never edits your chat history, never changes your global embedding settings, and never loads the shared `application` folder into its context. Stopping a flow never cancels an answer she is writing for you in another tab.
### Saving flows: `.fpmt`, not `.pmt`
**File ▸ Save as .fpmt** downloads your diagram; **File ▸ Open .fpmt**, or dropping a file on the canvas, brings it back. Opening a flow never runs it. Since v1.83.0 a `.fpmt` file dropped on any Tlamatini page is recognised by its content and sent here: the panel loads it (asking before it replaces a diagram you have open), and from the chat or the Agentic Control Panel it opens in a new Prompt Flow Panel tab. The October 4 source update also opens prompting flows from the main chat or Windows Explorer and takes you back to the requested editor after login. Both `.flw` and `.fpmt` have per-user Windows association management; see [Windows flow files](docs/windows-flow-files.md). A file may hold up to 500 assets (executable operations and static comments together) and 1,000 connections and be up to 5 MiB. The panel also keeps a draft in your browser, just for you, so a closed tab is not a lost afternoon — but save a file if you want to keep or share the flow.
Why the odd extension? Because **`.pmt` already means something**: it is the extension of Tlamatini's own plain-text system prompt, `prompt.pmt`. Flow diagrams are **`.fpmt`** — *flow prompt* — so the two can never be confused, and the panel refuses to open a `.pmt`. If you made a diagram in the earliest version and saved it as `.pmt`, just rename it to `.fpmt`; nothing inside it needs to change.
Want to see one first? **File ▸ Open example** loads a small branching flow, and the prompting tour — seven prompting operations and a static commentary — ships as `docs/examples/prompting-kickoff.fpmt`. Terminater is the eighth executable operation; configure it separately for a real terminal on your machine.
New saves use document **version 2**. Version 1 files and browser drafts automatically turn their old executable `user_commentary` nodes into **User Input**, keeping IDs, connections and settings. Version 2 `user_commentary` means a static note. Its allowlisted styled `runs` concatenate to the saved literal `text`; older plain version 2 notes normalize to one run. Opening an old file never silently changes a reply step into an annotation. The extension remains `.fpmt`. The commentary/input split and graphical rich-text redesign date from 2026-10-03/04 and shipped in v1.76.0.
### Shortcuts
| Keys | Action |
|---|---|
| Ctrl+S / Ctrl+O | save / open |
| Ctrl+Z / Ctrl+Shift+Z or Ctrl+Y | undo / redo |
| Ctrl+A / Ctrl+D | select all / duplicate the selection with its inner connections |
| Delete or Backspace | delete the selection |
| Arrow keys / Shift+arrows | nudge the selection 10 / 40 units |
| Ctrl + mouse wheel, **Fit** | zoom, fit the whole flow |
| Double-click / Enter on a commentary | edit the static note in its bubble |
| Ctrl+Enter while editing a commentary | save the complete edit |
| Ctrl+B / Ctrl+I / Ctrl+U while editing | bold / italic / underline the selection or next typed text |
| Up/Down on the output divider; Shift for larger steps | resize output by 1 / 5 percentage points |
| Home / End on the output divider | output at 5% / 95% |
| Esc | cancel a commentary edit, close the top dialog, or leave connection mode and clear the selection |
### How it was tested
The file format and the interpreter have their own regression tests, the real WebSocket route is exercised end to end with a stand-in model, and the release build refuses to package itself unless the panel's self-check (`check_prompt_flow_panel`) passes *inside the frozen application*. The `.fpmt` behaviour — default and typed file names, an old `.pmt` name being converted, Unicode and upper-case names, a real reopened flow playing back, the example, and every kind of bad file being refused — was checked in a visible Chrome window with fifteen photographed checkpoints, all passing on 2026-09-25. The 2026-10-03 commentary/input refactor passed 61 backend/packaging tests and nine visible Chrome checkpoints, including 5,000-character containment without scrolling, formatting, resize/Undo/Redo, saved-file/draft recovery, unchanged User Input replies/cancellation and version 1 migration. The October 4 graphical redesign subsequently passed 29 diagram/runner tests, three carriage tests and 23 visible Chrome checkpoints, including three differently styled notes through three save/open/save cycles with equal JSON and rendered line boxes. The October 4 Run output change passed 14 visible Chrome checkpoints for dragging, limits, independent scrolling, unchanged content geometry, keyboard controls, collapse/reopen, window resizing and layout recovery. These are dated implementation checks; regenerating this book does not rerun them. The full maintainer guide is `docs/prompting-flow-designer.md`.
---
## Reading a PDF: open it in the canvas, then hand her the whole document
### The canvas learned to read
For its whole life the canvas on the right of the chat held one kind of thing: text. Code you loaded, code Tlamatini wrote, a file you opened to look at. Anything that was not text had to be *described* to her rather than *shown* to her.
Since **2026-09-16** it holds PDFs as well. Press **Open**, pick a `.pdf`, and the document appears in the canvas — the real document, not an extracted approximation of it. Every page is there. You can select text with the mouse, jump to a page, zoom, fit to width, fit the whole page, rotate, scroll continuously or one page at a time. If the file is password-protected, the viewer asks you for the password itself.
The buttons above the canvas keep the meanings they always had:
| Button | What it does to a PDF |
|---|---|
| **Copy** | Lifts the selectable text out of *every* page. If the file is a scan with no text layer, it says so and explains that OCR is needed — it does not hand you an empty string and let you think the document was blank. |
| **Save As** | Writes the **original PDF bytes** back out, untouched. |
| **Refresh** | Rereads the open document without a chooser. Unchanged text stays in place; changed content replaces it after preparation. Failure or cancellation keeps the previous document/context. |
| **Clear canvas** | Puts the document away and shuts down its background worker. |
### Nothing leaves your machine just because you opened it
This is worth being precise about, because “open a PDF in a web page” normally means “upload a PDF”.
It does not here. The viewer reads the file **in slices, straight off your disk**, as you scroll. The document is never copied whole into memory and never sent anywhere. That has a pleasant consequence: **Tlamatini imposes no file-size limit and no page-count limit.** The test suite opens a sparse 256 MB file and navigates to page 10,001, and both work — past ten thousand pages the viewer switches itself to single-page scrolling, because browsers cannot scroll an arbitrarily tall page, and every page stays reachable. What limits you is your own machine's memory and disk, not a number somebody picked.
And **no AI model is called merely because a PDF is on the screen.** Opening one costs nothing at all.
### “Use as context” — the button that changes what she knows
The canvas has always had a **Use as context** button. On a PDF it means what you would hope: Tlamatini takes in the **whole document** — not the page you are looking at, not a snippet — and can answer questions about it for the rest of the conversation.
Press it and a dialog opens before anything happens. It shows the filename, the size, an elapsed clock, and asks you exactly one question: **Process images?**
**It starts unticked. Every single time.** That is deliberate, and it is the difference between two quite different jobs:
- **Unticked — text only.** She reads the selectable text of every page. No page is rendered, no image is extracted, no vision model is contacted. It is fast, it is private, and it works on a machine with no vision model configured at all. This is the right choice for a report, a manual, a paper, a contract — anything typed.
- **Ticked — text *and* pictures.** She additionally renders every page as an image, pulls out every unique embedded image, and hands each one to her **Image-Interpreter**: the same two-models-in-parallel-plus-a-referee vision pipeline she already uses on your screenshots, running from its own saved configuration, with a prompt written for documents — read the text in the image, interpret the chart, the table, the diagram, and *say plainly when you are unsure*. This is the choice for a scan, a slide deck exported to PDF, an engineering drawing, anything where the meaning lives in the pictures.
**Nothing starts until you press Continue.** Cancel, Escape or closing the dialog before that point does exactly nothing: no upload, no extraction, no analysis, no background job, and no change to whatever context you already had.
### Watching it work, and stopping it
Once you press Continue you get four progress rows, and they are honest ones: **bytes uploaded**, **pages extracted**, **images analysed**, **context loaded**. In text-only mode the image row simply says *Skipped* rather than pretending to do something. The clock starts when you press Continue, not while the dialog sat waiting for you. The final row stays indeterminate until the server confirms it is done — it does not invent a percentage or an estimated time that nobody could actually know.
**Cancel really cancels.** It aborts the browser's requests and signals the background worker, which checks between upload chunks, between pages and between image calls. One honest caveat: a vision request that is already in flight may finish before the cancellation reaches it — the configured engine has its own request timeout. Once the context is actually loading, the button becomes **Close**, and the ordinary chat status keeps you informed.
If the work succeeds the dialog closes itself. If it fails it stays open with its error, so you are never left guessing.
### What she will and will not claim
Three promises are built into this feature, and they are the reason to trust it:
1. **Partial image analysis is reported, never hidden.** Every image keeps its own report. If four figures out of thirty could not be read, the dialog says the analysis was incomplete instead of quietly claiming all thirty succeeded.
2. **A scan with no text is refused, not faked.** Hand her a photographed document with *Process images* unticked and she explains that there is no selectable text and suggests ticking the box — rather than loading an empty document and then answering your questions about it. On a mixed document, the pages that had no text layer are marked in the index.
3. **A failed context load is reported separately, and you can retry.** Extraction succeeding and the context loading are two different events, and they are told apart.
### Where it is kept, and your password
Each prepared document becomes a private package under the runtime agent root, at `context_files/pdf_canvas/<your-user-id>/<document-id>/`: the `source.pdf`, a `document.txt` holding the complete extracted text, and — when you enabled images — the page previews, the extracted images and an `.analysis.txt` report beside each one. Successful packages stay so the context can be restored in a later session. A cancelled or failed preparation **deletes its own half-finished package**, and the upload staging file under Tlamatini's own `Temp` folder is removed when the job ends.
**A PDF password is used in memory to open the document and is written nowhere** — not into browser storage, not into the context text, and not into `tlamatini.log`.
Text-only and image-enabled preparations are cached **separately**, so changing that checkbox can never silently reuse a result prepared the other way.
### What it is made of
The viewer is **Mozilla PDF.js 6.3.289**, vendored into the repository with its unmodified Apache-2.0 licence — the API, the worker, the viewer, the character maps, the standard fonts, the annotation images, the ICC profile and the WebAssembly decoders. That means rendering needs **no npm install and no CDN at runtime**; nothing is fetched from the internet to draw your document. You can reproduce the exact pinned bundle yourself:
```powershell
python scripts/vendor_pdfjs.py
```
It verifies SHA-512 integrity before extracting an explicit subset of files.
Extraction on the server side uses **PyMuPDF**, which Tlamatini already shipped, so this whole feature added **no new Python dependency**. The backend is four small modules — `pdf_context.py` (extraction, private packages, signed context tokens), `pdf_image_analysis.py` (the adapter to the real Image-Interpreter engine), `pdf_context_jobs.py` (a serial background queue with cancellation) and `pdf_context_views.py` (prepare / status / cancel, behind the existing login, CSRF and method checks). The handoff into the retrieval chain is a **user-bound signed token**, so one account's prepared document cannot be handed to another.
Image-Interpreter is **reused, not re-implemented**: its pipeline builder was made import-safe so the PDF backend can call it directly, while running it as a pool agent still does its own logging and process setup exactly as before.
### How it was tested
The browser suite drives **real Chromium** against the application's real markup, real modules and real PDF.js, on an isolated HTTP server, with generated PDF fixtures and genuine extraction and background jobs. It covers the first, middle and last pages; zoom, rotation and resizing; saving the original bytes; copying; handing the document to context; switching between a PDF and a text file; scanned files; password-protected files; a sparse 256 MB file read in ranges; navigating to page 10,001; the progress dialog; cancellation; and retrying. It also **rejects any request for an external asset**, which is how “no CDN” is kept true rather than merely intended. Set `PDF_CANVAS_HEADED=1` to watch it run.
```powershell
python -m pytest Tests/test_pdf_canvas_browser.py Tests/test_pdf_canvas_assets.py -q
python Tlamatini/manage.py test agent.test_pdf_context agent.test_image_interpreter_agent --noinput
```
The complete contract — every guarantee, limit and file path — lives in [docs/pdf-canvas.md](docs/pdf-canvas.md).
---
## Voice commands: your words become the prompt
### Direct microphone beside Send
Click **Mic** next to **Send**, wait for **Listening**, and speak your
prompt. The colored microphone, live input meter, elapsed time and silence
countdown stay in the chat. **Opening microphone** means capture is starting;
**Listening** means actual samples have arrived.
After the configured continuous silence (default **3.5 seconds**) the gate
stops capture. Whisperer immediately begins transcription, then the normal chat
form **sends the recognized text automatically** by default, including the
existing draft. In **Config ▸ Mic**, choose **Keep in the chat input** instead to
review and correct the transcript, then press Send yourself. That mode starts
no task or processing acknowledgment until Send. No read-back dialog is added.
The normal chat/permission behavior then applies to the resulting request.
Successful dispatch triggers the avatar's browser-voice acknowledgment unless
Silent mode suppresses it.
The button does not ask Ollama to choose Whisperer and does not turn on
Multi-Turn, Exec Report, ACPX, Ask Execs, Step-by-Step or Internet. Those current
settings travel with the submitted prompt. Click again or press **Escape** to
cancel; a cancelled or stale transcript cannot be sent. No speech, empty
recognition and errors leave the original draft unchanged. If submission becomes
unavailable after transcription, the recognized text remains in the draft.
Recording uses the **microphone on the Tlamatini host computer**, not a remote
phone or laptop's browser microphone. No extra Whisperer console opens and the
worker does not steal focus. Its diagnostics go to the main Tlamatini console
and log. Config ▸ Models ▸ Speech selects the recognizer. **Mic**, the last
Config entry after Voice, opens the shared system-style settings dialog:
input device/refresh, software gain (0–300%), silence (0.3–20 s), duration cap
(5–600 s), automatic/manual sensitivity, language, English translation, sample
rate, channels, beam size and local VAD. Preferences persist in this browser;
Reset then Save restores inheritance from Whisperer's template. Save affects
the next recording, including when a capture is already underway. Cancel, ✕
and Escape discard edits; outside clicks leave the dialog open. First model
loading or download can delay recognition; the button does not promise zero
latency.
### The VOICE COMMANDS catalog is a different entry point
The catalog's **VOICE COMMANDS** section still contains guided workflows that
ask the model to invoke the wrapped `chat_agent_whisperer` tool. They keep
their own instructions and mode badges; they are not the microphone button.
**#121 — YOUR FIRST VOICE COMMAND** is the Step-by-Step rehearsal: it checks
microphone/recognition setup, shows the transcript and guides a read-only trial.
**#122 — SPEAK YOUR PROMPT** uses Angela's original request:
> *"Tlamatini, using Whisperer record my voice till I finish to tell you a prompt, then use the text extracted as a prompt and invoke it, go!."*
That card selects **Multi-Turn, Exec report and ACPX**. Its transcript read-back
and written-confirmation instructions belong to that catalog workflow. They
must not be described as a confirmation screen implemented by the direct button.
Both routes refuse to invent a command from empty or failed recognition.
### Shared gate, distinct lifecycle
Standalone/workflow Whisperer still supports fixed-duration and file inputs,
saved transcript/audio artifacts, optional Ollama transcript cleanup, structured
`INI_SECTION_WHISPERER` output and downstream targets. The direct button
forces microphone input, silence gating and no cleanup/targets for that job; it
returns text to the existing composer instead of writing workflow output files
or calling an agent pool. It does not create a separate Exec Report tool row.
See [the full microphone contract](docs/chat-microphone-design.md) for the state
machine, configuration precedence, authentication, worker lifecycle, troubleshooting
and dated measurements.
---
## Whisperer listens: the silence gate
### The wrong question
For most of Whisperer's life, asking her to take dictation meant answering a question nobody can actually answer: *how many seconds are you going to speak for?* The agent recorded `record_seconds: 30` and stopped, whether you were mid-sentence or had finished twenty seconds ago. Both failures are bad, and the second one is worse than it looks — a recording full of silence is a recording the speech recogniser has to wade through, and the model will cheerfully hallucinate words into it.
The fix is not a better guess. It is to stop guessing: **keep recording while somebody is talking, and stop once they have finished.** That is the sound gate, and since v1.51.7 it is the default.
### Telling "no duration" from "thirty seconds"
The only genuinely hard part was a small one. `record_seconds` always held a number, so the agent could not tell a user who said nothing about length from a user who deliberately asked for thirty seconds. Inventing a second flag would have meant two knobs that can contradict each other.
The answer was already in the file. A few lines above, `sample_rate: 0` has long meant *"use the microphone's native rate"* — zero as a word, not a quantity. So `record_seconds: 0` now means **"no duration was given: listen until I stop talking"**, and any number you name turns the gate off and is honoured to the sample. *"Record 45 seconds"* still records exactly 45. An old saved flow that stored `30` asked for thirty and still gets thirty; nothing you already built changes behaviour behind your back.
### How it hears you
Whisperer already opened the microphone through a callback that measures every twenty-millisecond block to drive the VU meter, so the gate costs one extra measurement on audio she is holding anyway — no new library, nothing to install, and it works exactly the same with the cloud recognisers.
The rule is deliberately simple. Whisperer tracks the room's own noise, and counts anything about nine decibels above it as a voice. Speech must last two blocks before it counts, so a chair creak is not a word; silence only begins accruing a little below the speech threshold, which is what stops the countdown flickering on the quiet tail of a sentence. Any voice at all resets the clock to zero.
Two decisions inside that are worth knowing, because both protect *you* rather than the machine:
- **She starts sensitive, not cautious.** A gate that listens to the room first, then decides, will take the voice of somebody who begins talking immediately for the room itself — and then gate that person out, cutting them off mid-sentence. So the threshold begins at a fixed, generous floor and is tightened only by evidence.
- **Fifteen seconds of unbroken "voice" re-measures the room.** People breathe; a fan does not. A steady mechanical hum sits above the starting threshold and would otherwise read as somebody talking forever, so a stretch with no gaps at all is treated as a machine and the room is re-baselined around it.
### The countdown you can watch
**Direct chat dictation shows its meter and countdown inside the composer and opens no extra window.** The console example below belongs only to standalone/workflow Whisperer's legacy REC indicator; it is not the microphone-button interface. The sample uses the current 3.5-second silence default:
```
● REC [████████──────] 12.3s ● VOICE [──────────] 0.0s/3.5s
● REC [──────────────] 14.5s ○ silence [██████────] 2.2s/3.5s
■ REC STOPPED ✓ silence 3.5s captured 15.8s
```
The silence bar fills as the room stays quiet and empties the instant you speak, turning amber past halfway and red past eighty percent — so you can see it about to fire and keep talking if you want it. The final line says *why* it stopped, which matters: a recording that ends on its own and does not explain itself reads as a crash.
### The thing that would have made it invisible
The largest risk in this change was never the audio code. It was habit. Whisperer's own tool description used to tell Tlamatini that she *"records record_seconds (default 30)"*, and its worked example opened with *"Transcribe 5 seconds…"*. Had that stayed, Tlamatini would have kept volunteering a duration out of politeness, the gate would never have fired once, and the whole feature would have been invisible — while looking, from the outside, exactly like it worked.
So the description now says plainly: **do not pass `record_seconds` unless the user named a duration**, and a test fails if that sentence ever drifts back out.
Two smaller honesty repairs travelled with it. The reported recording length used to echo what was *requested*; it now reports what was actually *captured*, which under a gate are different numbers by design. And because the gate lives inside the audio callback, a sound driver that refuses that callback cannot be gated at all — standalone Whisperer records a fixed length and **says so**, rather than calling it gated listening. Direct chat dictation instead refuses that fallback and reports an error, because automatic submission requires a working gate and reliable cancellation.
### Knobs
| Setting | Default | What it does |
| --- | --- | --- |
| `record_seconds` | `0` | **The switch.** 0 = listen until the speaker stops; N = record exactly N seconds, gate off. |
| `silence_timeout_seconds` | `3.5` | Silence that ends the recording. The clock starts when the microphone opens, so an empty room still stops. |
| `silence_gate` | `auto` | `auto` follows `record_seconds`; `on` forces the gate (the duration becomes a ceiling); `off` forces a fixed recording. |
| `max_record_seconds` | `300` | Ceiling for a gate that never fires — a radio left playing is never silent. |
| `silence_threshold_db` | `0` | 0 measures the room automatically. A negative value forces a fixed threshold for an unusual room. |
---
## Edit and save a canvas file
Open or drop text, source code, JSON or Markdown into the chat canvas. The filename bar now has **Save**, **Save as**, **Undo**, **Redo**, **Find/replace** and **Wrap**. The dirty marker means the current buffer differs from its saved baseline. Ctrl/Cmd+S saves; Ctrl/Cmd+Shift+S chooses a destination. Copy and Use as context take the edited source. Tab normally moves focus; Ctrl+[ and Ctrl+] indent.
| File kind | Editing and saving |
|---|---|
| Text, code, JSON and configuration | Edit directly with syntax support; Save writes the current buffer. |
| Markdown | Edit Source or inspect sanitized Preview; both share one buffer and undo history. |
| HTML and SVG | Edit source as text; the canvas does not execute the document. |
| PDF | Read-only PDF.js viewer; Save exports the original PDF as a copy. |
| Flow files and images | Their existing designer or attachment route remains in charge. |
On a local connection, **Save as** and the navbar **Save** open a native Windows destination dialog, so you can choose a project folder or another writable location outside Downloads. Toolbar Save reuses an authorized destination; a file without one asks first. Cancelling or failing a save leaves edits in place. A supported browser-granted file handle can also provide write access. Remote browsers need a picker supported by that browser.
Saving compares the disk revision and refuses external modifications instead of overwriting them. Save As can preserve a separate copy. A save captures one buffer revision: if you keep typing, later edits remain dirty. Unedited files retain exact bytes; edited files retain encoding/BOM and existing line terminators. New lines use the original first terminator. UTF-8 and BOM-marked UTF-16 are supported strictly; malformed text and binary files are refused. The text/native-save limit is 64 MiB.
Open, Clear and Refresh ask before discarding edits. Unsaved buffers are held in memory: save before closing the page. Refresh reloads an accessible source; it does not merge changes. Use as context is a snapshot, so use it again after editing when you need to update ordinary chat context.
## Learn in the editor with Interactor
**Interactor is a module and a built-in tool, not a workflow agent.** Enable **Multi-Turn**, **Interactor** in Configure MCPs/Tools and **Live editor** above the canvas. Try: “Help me create a Dockerfile for a Java servlet application using your text editor. Teach one step at a time.”
1. Tlamatini reads the exact unsaved editor buffer and selection. If no path was provided, she creates the requested filename under Temp/interactor in an account-specific unique directory; an existing file is not overwritten.
2. She types a short known segment, for example `FROM `, then hands the next field to you and explains which base image suits your Java runtime. The handoff stops further editor mutations in that chat turn.
3. As you type, optional ghost text matches your prefix against choices prepared for this field. **Tab** accepts the suggestion, **Ctrl+Right** accepts a word and **Escape** dismisses it. You can type a different value.
4. **Continue** sends a new chat turn and reads the live buffer again before reviewing your work. **Another hint** and **Explain selection** also use normal chat while preserving an existing composer draft.
5. Save when you want the buffer persisted. Typing is not saving and does not execute code.
The model prepares field choices once; matching is local, with no model request per keystroke. New questions still have normal chat/model latency. User input, selection, Escape, Stop, cancellation or disconnection interrupts AI typing and keeps the typed prefix editable. A temporarily hidden or occluded tab pauses for up to two extra minutes, then resumes at the next character when visible; a timeout never replays text. Switching to Markdown Preview or hiding the canvas returns control.
Live tutoring accepts at most 262,144 UTF-16 units; a smaller configured tool-result/context budget can refuse a read. Oversized or stale content is refused explicitly, never silently truncated. For permissions and dated source/frozen evidence, see [Canvas editing](docs/canvas-editing.md) and [Interactor](docs/interactor.md).
The [account-creation wizard](docs/user-creation-wizard.md) is separate from editor tutoring: it opens the interactive createsuperuser console and keeps passwords out of chat. It creates an administrative superuser. Saying that you will type a password does not reserve all commands for you, and an optional email field does not authorize an outbound notification.
## Terminals you can watch: Terminater
> **Carried from v1.88.0** (tagged 2026-10-20). The dated evidence below describes that release; current source and frozen-package results are distinguished above.
### The problem it solves
Executer is wonderful when you only want the *answer*: it runs a command in a shell nobody can see and hands back text. But sometimes you want to *watch* — to learn what a command does, to follow a walk-through, to answer the question an installer asks, or to be sure nothing surprising is typed. For those moments Tlamatini now has **Terminater**, the visible sibling of Executer.
### See it in thirty seconds
1. Open the chat and choose **Views ▸ Terminals** (or press **Ctrl+`**). A Terminals area appears under the canvas. Nothing is running yet.
2. Open **Catalog of Prompts**, go to **Run & Execute** and click **TERMINATER FIRST LOOK**. Tick **Multi-Turn** and **Exec report** as the card says.
3. Send it. A real PowerShell appears in the Terminals area, and you can watch Tlamatini *type* a command — a little faster and slower here and there, like a person at about 50 words per minute. When the command finishes she reads the whole output and reports the exit code.
Typing is slow **on purpose**: a 60-character command takes about fifteen seconds. There is no paste and no instant mode, so you always see what is being typed before it runs.
### Take the keyboard — and give it
* **Esc** stops Tlamatini's typing at once and the keyboard is yours. You never have to fight for it.
* **Hand-over.** Try the **TERMINATER HAND-OVER** card. Tlamatini types `ping `, then waits for *you* to type the host and press Enter, then types ` -n 2` and runs the line. Whatever you type is reported separately from what she typed, so the report always says who did what. If you do not finish in five minutes the result is `operator_required` — she never invents a host.
* **Secrets.** When the part you type is a password or a token, the request says `secret: true` and that part is never logged or returned.
### Several terminals at once
The **TERMINATER GUIDED TOUR** card opens a CMD and a PowerShell side by side (`tile-h`), types a version command in each, and then — one step at a time — watches what *you* start to type in one of them and warns you if it looks dangerous or mistyped. The layouts are `tabs`, `tile`, `tile-h`, `tile-v` and `grid`, and the area can be the docked strip or the whole canvas.
### When a command misbehaves
A command that prints nothing for 45 seconds is flagged **hung**; Tlamatini can abort it (Ctrl+C, Ctrl+C again, then ending its processes) without losing the terminal. A command that exits with an error is a red row in the Exec report even though the terminal itself worked, so a flow can branch on it.
### Where else it lives
* On the **Agentic Control Panel** it is workflow agent #86 — draw it, wire it, and a Forker can branch on `{status}` and `{exit_code}`. If a terminal needs your input while a flow runs, the Start dialog gets out of the way so the terminal holds the keyboard.
* In the **Prompt Flow Panel** it is the new parallelogram: the node's text is typed into your terminal and the output becomes `{{last_output}}`.
* It is on the **Ask Execs** list: a visible command can still delete things, so with Ask Execs ticked you see exactly what will be typed before it is.
### Good to know
It needs Windows 10 version 1809 or newer. Closing a page never closes a terminal, but they all end when Tlamatini does. To switch the whole feature off, set `terminater_enabled` to `false` in `config.json`. The complete rules, every action and how it is tested are in `docs/terminater.md`.
---
## Avatar animation repair and reproducible visible tests
### Four expression assets and their animation coordinates
The avatar combines independent eye and mouth states. Each source asset is an opaque **1024 × 1024 RGB JPG**, rendered in the same square viewport without per-state layout changes:
| Application asset | Expression | Converted image deliverable |
| --- | --- | --- |
| `Tlamatini/agent/static/agent/img/avatar/eo_mc.jpg` | Eyes open, mouth closed | `02_eyes_open_neutral.jpg` |
| `Tlamatini/agent/static/agent/img/avatar/ec_mc.jpg` | Eyes closed, mouth closed | `04_eyes_closed_neutral.jpg` |
| `Tlamatini/agent/static/agent/img/avatar/eo_mo.jpg` | Eyes open, mouth open/smiling | `01_eyes_open_smiling.jpg` |
| `Tlamatini/agent/static/agent/img/avatar/ec_mo.jpg` | Eyes closed, mouth open/smiling | `03_eyes_closed_smiling.jpg` |
The [image deliverables](https://github.com/XAIHT/Tlamatini/tree/46a18c8d2965af6e833b1fb3aaa68b3039be4167/output/Gemini_girl_animation/) also preserve the original four 1024-square quadrant crops from the 2048-square source sheet. The animation-oriented PNGs correct small source offsets using translation registration on stable hair/armor, a shared six-pixel inset, and one common Lanczos resampling operation. The image files are not distorted independently or regenerated. `alignment.json` records crop boxes, shifts, dimensions, and hashes. Small non-rigid differences already drawn into the source remain; this is expression switching, not generated video or phoneme-level lip synthesis.
`sprite_2x2.png` contains the aligned frames in original row-major order. `preview.html` preloads/decodes the PNGs and draws them at identical canvas coordinates. The later JPG conversion retained the supplied frames' dimensions and positions, using JPEG quality 98 with no chroma subsampling. Both the [complete image ZIP](https://github.com/XAIHT/Tlamatini/blob/46a18c8d2965af6e833b1fb3aaa68b3039be4167/output/Gemini_girl_animation.zip) and [JPG ZIP](https://github.com/XAIHT/Tlamatini/blob/46a18c8d2965af6e833b1fb3aaa68b3039be4167/output/Gemini_girl_animation/girl_frames_jpg.zip) are committed deliverables.
### Why the entire portrait flashed
The old CSS applied `transition: opacity 90ms linear` to all four full portraits. During a switch, two overlapping frames were each partly transparent. At the midpoint, the combined coverage is `1 - (1 - 0.5) * (1 - 0.5) = 0.75`: **25% of the dark background leaks through**. A real Chromium reproduction measured minimum coverage approximately **0.753**, so the dimming was a rendering problem, not a bad JPG or a `collectstatic` conversion.
Source JPGs, development collected JPGs, frozen installed static/collected/source copies, and the initially served frozen HTTP responses were SHA-256 identical. The real Django collector was independently checked for exact byte copies, then the full project's `collectstatic` was run after migrations. Image dimensions and content did not change.
### The source repair
1. [avatar.css](Tlamatini/agent/static/agent/css/avatar.css) switches visibility with opacity fixed at one and no transition. The neutral frame remains underneath as a fallback. The dock disables selection, preventing a blue highlight when double-clicking to mute.
2. [avatar.js](Tlamatini/agent/static/agent/js/avatar.js) decodes each frame before selecting it. Slow, failed, or pending-decode frames leave the last valid portrait visible; class changes occur together in one JavaScript task. Mouth changes stop when speech is paused, the document is hidden, or reduced motion is requested.
3. [agent_page.html](Tlamatini/agent/templates/agent/agent_page.html) adds a matching avatar cache-busting suffix to CSS and JS. The existing startup-generated `STATIC_VERSION` mechanism remains intact.
Normal behavior is preserved: automatic blinking, mouth movement while native speech runs, click-to-greet or stop, Escape-to-stop, double-click mute, and Ctrl+Shift+M to toggle mute. These repairs do not add a remote video service or change the user's four source JPGs.
### One Python command to run the visible test
From the repository root:
```powershell
.\python\python.exe Tests\run_avatar_tests.py
```
For a different Python environment with the project requirements, use `python Tests/run_avatar_tests.py`. The launcher:
1. Checks Django, Node.js, Playwright, and Chromium. It detects existing project/PATH/Codex-bundled dependencies; it does not silently install packages.
2. Checks that port 8001 is free, without stopping any existing application.
3. Prepares `Temp/avatar_flash_fix/development-test.sqlite3`, using the published session-free fixture when available; applies migrations; and creates/updates only the ordinary test account **user / changeme**.
4. Runs the project's `collectstatic` against the source assets.
5. Starts the real Django development page on **127.0.0.1:8001**, with distinct test cookies so it does not collide with the frozen app's session.
6. Opens a **visible browser**, signs in, runs native Windows speech, checks at least 300 frame changes across three viewports, and exercises pause/resume/stop/mute controls.
7. Saves screenshots and JSON proof under `output/avatar_flash_fix/visible-test/`. Closing the browser stops the server launched by this command. `--auto-close` performs that cleanup after a successful run; `--check` only validates dependencies, and `--prepare-only` stops after database preparation and `collectstatic`.
If dependencies are missing, install Node.js and run `npm install --no-save playwright` followed by `npx playwright install chromium`. `TLAMATINI_TEST_NODE` can select a Node executable; `PLAYWRIGHT_MODULE` can select an installed Playwright package. The test account is deliberately generic and must not be treated as a production credential.
### What was tested, and what was not
The final visible repair run checked **311 transitions and 10,813 browser paints** using **Microsoft Zira — English (United States)**, with all four states exercised, zero transparency/coverage failures, zero within-viewport layout jumps, and no JavaScript errors. Native speech pause/resume, Escape, click-to-greet/stop, double-click mute without blue selection, and keyboard unmute all passed. Earlier focused browser checks also covered delayed image downloads, missing images, held decode promises, and reduced-motion mode. A preceding visible run exposed the selection highlight; it was fixed and the visible test was repeated.
The tests use real Django login, templates, middleware, static HTTP responses, and native speech synthesis. The browser's LLM WebSocket is isolated so these frontend tests do not launch model jobs or modify the frozen session. **LLM answer generation is not claimed as covered.** The developer's normal database and frozen installation were not replaced or restarted during the repair. Frozen-mode rebuild/reinstall remains a separate validation step.
### Repository assets and publication safety
The complete deliverables are now tracked under [output/](output/), rather than hidden by the old blanket `/output/` ignore rule. They include PNG/JPG frames, crops, sprite sheets, ZIPs, preview HTML, scripts, image-alignment metadata, screenshots, visual-test JSON, original code snapshots, and deployment-code backups. The [verification report](https://github.com/XAIHT/Tlamatini/blob/46a18c8d2965af6e833b1fb3aaa68b3039be4167/output/avatar_flash_fix/README.md) explains each evidence file. The test suites are [test_avatar_animation.cjs](Tests/test_avatar_animation.cjs), [test_avatar_collectstatic.py](Tests/test_avatar_collectstatic.py), and [test_avatar_visible.cjs](Tests/test_avatar_visible.cjs), with [run_avatar_tests.py](Tests/run_avatar_tests.py) as the Python entry point.
The committed `output/avatar_flash_fix/development-test.sqlite3` is a **sanitized fixture containing seeded project data and the generic test user**, not a personal/production database. Live login sessions were removed before publication; runtime sessions and server logs are kept under ignored `Temp/avatar_flash_fix/`. Binary attributes prevent Git from changing line endings inside PDFs, PPTX files, image files, ZIP archives, or SQLite fixtures. The original release/file-count snapshots elsewhere in this book retain their stated measurement scope; the newly published avatar assets are documented separately here.
## Demo videos
- [First system-usage walkthrough](https://www.youtube.com/watch?v=CkvDPSd_c-g)
- [Loading a complete project and summarizing its source code](https://www.youtube.com/watch?v=Lrpbt_dPIXw)
- [Installing OpenCV end-to-end in Multi-Turn](https://www.youtube.com/watch?v=bBlqbZVK-Wk)
- [Uninstalling Poco — Exec Report and matching flow](https://www.youtube.com/watch?v=E5vi0q5FxXQ)
- [Designing a flow with FlowCreator's help](https://www.youtube.com/watch?v=Tgoa7Tmoo0o)
---
## Enable Tlamatini as a Blue-hat agent
In this book, **Blue-hat** means a defensive operator posture: Tlamatini helps inspect and respond to security signals on a Windows machine that you own or are explicitly authorised to defend. It does **not** turn her into an unsupervised endpoint-security authority, add a new database-backed workflow Agent, or make every alert a confirmed intrusion. The `security/` directory is an administrator-operated host toolkit. A human chooses when to enable it, reviews the evidence, decides whether response is justified, and owns the resulting Windows policy changes.
This distinction matters because the toolkit has two very different jobs:
1. **Host enablement** changes persistent Windows settings so Tlamatini can run and observe more of the machine.
2. **Defensive monitoring and response** inspects ten signal families and can, in armed modes, add firewall blocks or force-stop selected processes.
The safest operating pattern is always **validate → record a baseline → enable → restart → detect-only → investigate → arm only when justified**.
### The complete `security/` asset map
```text
<Tlamatini-root>/
├── Tlamatini.exe # installed build; absent in a source checkout
├── agents/ # installed agent catalog, including Shoter
├── Tlamatini/agent/agents/ # equivalent source-tree catalog
└── security/
├── README.md # local quick reference
├── enable_tlamatini_v2.bat # self-elevating enablement launcher
├── tlamatini_whitelist_v2.ps1 # verified Windows access setup v2.3
├── run_defender.bat # self-elevating report-only launcher
├── tlamatini_defender.ps1 # report-only monitor, optional response v2.2
├── automated_tests_of_security_assets.py
└── security_logs/ # created at runtime; git-ignored
├── alerts.log # concise alert and response stream
├── monitor.log # complete monitor stream
└── asset_tests/ # visible-test logs, HTML, JSON, and screenshots
```
The two batch files resolve themselves with `%~f0`, carry that path through UAC in an environment variable so spaces are preserved, locate their companion scripts with `%~dp0`, and invoke Windows PowerShell with `-NoProfile -ExecutionPolicy Bypass`. They propagate the PowerShell process's failure code instead of always appearing successful. The PowerShell files derive the repository/install root from `$PSScriptRoot` and `Split-Path -Parent`; there is no fixed drive letter or install-directory name.
### Requirements and authority boundary
- Windows 10 or Windows 11 with Microsoft Defender PowerShell cmdlets and the Windows Firewall/Security-log facilities used by the scripts.
- Administrator approval for the whitelist and defender. The asset regression test itself is non-destructive and does not require elevation.
- A machine you own or have explicit written authority to defend. The toolkit does not authorise scanning, containment, or investigation of third-party systems.
- A human operator who can review Windows events, process paths, firewall rules, and false positives before treating a signal as malicious.
- A recovery plan. There is currently **no bundled undo script** for the whitelist's persistent changes.
The toolkit supplements Microsoft Defender and human incident response. It is not an antivirus engine, EDR, SIEM, sandbox, forensic product, legal conclusion, or certification that a host is clean.
### Validate the assets before changing Windows
Run the persistent test from the repository/install root:
```powershell
python security\automated_tests_of_security_assets.py
```
Open a new foreground PowerShell with `-NoExit`, confirm that it is visible, and run the test there. The harness asks for confirmation before checks and keeps using that console. It parses every PowerShell asset, verifies standalone payload parity, runs mocked permission/defender behavior tests, checks the real BAT parser from a complex path, and captures all screens through **Shoter**. JSON and a viewable HTML summary are saved under `security/security_logs/asset_tests/`. No browser is started automatically.
These checks do not apply Windows policies or run an armed sweep. Screenshots may contain sensitive desktop information. For actual non-admin interpreter, file, shell and loopback checks, run `python security\verify_agent_execution.py` in the same confirmed visible console. Its unique scratch files are removed afterward; it makes no paid model calls and does not certify external dependencies. Use `--visible-console-verified` only after a human confirms that same persistent console. See [security/README.md](security/README.md) for the maintained file inventory and switches.
### Record a pre-enable baseline
The whitelist saves Defender preferences and audit policy before changing them, execution policies before policy changes, and a pre-operation journal for firewall, registry and group membership changes under `security_logs/enablement-<timestamp>/`. These are selective recovery records, not an automatic rollback or a full system restore point. Additional read-only baseline commands are:
```powershell
Get-MpPreference | Select-Object ExclusionPath, ExclusionProcess, `
ControlledFolderAccessAllowedApplications, AttackSurfaceReductionRules_Ids, `
AttackSurfaceReductionRules_Actions
Get-ExecutionPolicy -List
Get-NetFirewallRule -DisplayName "Tlamatini*" -ErrorAction SilentlyContinue
auditpol /get /category:*
(Get-Item "HKLM:\SYSTEM\CurrentControlSet\Services\EventLog\Security").GetValue("CustomSD")
```
Store the output somewhere protected and outside the Tlamatini install tree. Domain policy, Intune, Group Policy, or another security product may own some of these settings; coordinate with that control plane instead of fighting it locally.
### What enablement actually changes
Run `security\enable_tlamatini_v2.bat` once and approve UAC. It launches `tlamatini_whitelist_v2.ps1`; most changes persist after the script exits.
| Area | Actual implementation | Security consequence |
|---|---|---|
| Microsoft Defender | Adds the whole Tlamatini root to `ExclusionPath`, adds `Tlamatini.exe`, and adds bundled Python executables when found to `ExclusionProcess`. | Defender services remain on, but excluded content/processes receive less scanning. Malicious code placed in the excluded tree can inherit that blind spot. |
| Controlled Folder Access | Leaves the global mode unchanged and adds exact discovered Python, application and toolchain paths, including bundled Python in source checkouts. Every entry is read back. | Shared interpreter exceptions also apply to commands run outside Tlamatini. |
| Attack Surface Reduction | Adds verified per-program exceptions. Only explicit `-AuditCompatibility` changes six global rules to **AuditMode (2)**. Action **6 means Warn**, not Audit. | Global changes are opt-in; a rejected or unverifiable setting produces `[ATTENTION]` and exit 2. |
| PowerShell | Sets the current user's execution policy to `RemoteSigned`. | Local scripts can run unsigned; downloaded scripts normally require a trusted signature unless explicitly bypassed. The batch launchers themselves use `Bypass`. |
| Windows Firewall | Creates stable owned outbound/loopback rules, checks ActiveStore, and disables exact local application blocks. Managed/broad blocks are reported and preserved. | Incoming access defaults to loopback; `-AllowLan` explicitly adds LocalSubnet access on Private/Domain profiles. |
| Security event log | Adds and verifies elevated-user Event Log Readers membership by locale-neutral SID, records the change, and leaves CustomSD untouched. | A new logon may be needed; using another administrator account applies membership to that account. |
| Audit policy | Uses stable subcategory GUIDs to enable success/failure Logon, Credential Validation, Sensitive Privilege Use, and User Account Management; success Process Creation. Every `auditpol` exit code is checked. | Produces the events the defender reads, avoids dependence on the Windows display language, and can increase Security-log volume. |
| Process command lines | Sets `ProcessCreationIncludeCmdLine_Enabled=1`. | Event 4688 gains command-line evidence, which is useful but can record sensitive arguments. |
| PowerShell logging | Enables Script Block Logging. | Improves script evidence but may log commands or values that need restricted retention. |
| WMI, tasks, registry, services | Executes `Get-CimInstance`, `Get-ScheduledTask`, Run-key reads, and `Get-Service` probes. | These four steps **verify** the elevated session's existing access; they do not install a WMI provider or create separate task/registry/service permissions. |
The accurate statement is therefore: **Defender, CFA, ASR, and firewall components are not globally disabled, but the whitelist deliberately creates exceptions; selected global ASR rules change only with explicit `-AuditCompatibility`.** Treat the Tlamatini tree and every executable allowed by those exceptions as a privileged trust boundary.
The six optional `-AuditCompatibility` ASR behaviors are: Office applications creating child processes; LSASS credential stealing; WMI event-subscription persistence; executable content from email/webmail; untrusted or unsigned USB processes; and process creation through PSExec/WMI. Their IDs are Microsoft's published identifiers, not locally invented aliases; compare them against the [Microsoft ASR rules reference](https://learn.microsoft.com/en-us/defender-endpoint/attack-surface-reduction-rules-reference) whenever the toolkit is updated.
After enablement completes, restart Tlamatini and open a fresh PowerShell/logon session where practical. Read every `[ATTENTION]` line. Exit 0 means configured checks succeeded, 2 means partial results need attention, and 1 means setup failed. Successful earlier changes are not automatically rolled back.
### Start with detect-only
The batch launcher `run_defender.bat` runs one **report-only** scan. The PS1 also defaults to report-only. To establish a baseline, open a visible persistent PowerShell as Administrator:
```powershell
cd <Tlamatini-root>\security
powershell -NoProfile -ExecutionPolicy Bypass -File .\tlamatini_defender.ps1 -DetectOnly
```
In detect-only mode, a response candidate becomes `WOULD BLOCK` or `WOULD KILL` in the alert log. Let normal developer, automation, firmware, browser, security-testing, backup, and administrative workloads run while you review what the heuristics consider suspicious. Record expected tools, paths, ports, accounts, and tasks before moving to an armed mode.
### Operating modes
```powershell
# One-shot observation; never blocks or kills
.\tlamatini_defender.ps1 -DetectOnly
# Continuous observation every 60 seconds; Ctrl+C stops it
.\tlamatini_defender.ps1 -Watch -DetectOnly
# Continuous observation with a deliberate interval
.\tlamatini_defender.ps1 -Watch -IntervalSeconds 30 -DetectOnly
# One-shot armed response (explicit opt-in)
.\tlamatini_defender.ps1 -Armed
# Armed watch loop
.\tlamatini_defender.ps1 -Armed -Watch -IntervalSeconds 60
# Also kill recognised dual-use tools outside Tlamatini's self roots
.\tlamatini_defender.ps1 -Armed -Aggressive
```
`-Watch` is a foreground loop, not a Windows service or scheduled task. It scans, appends logs, sleeps, and repeats until `Ctrl+C` or process termination. `-IntervalSeconds` is validated from `5` through `86400`, preventing zero/negative busy loops and unbounded accidental values. The whitelist's Windows settings persist; the watch process does not.
### The ten monitor families
| Monitor | Evidence examined | Response behavior |
|---|---|---|
| Defender health | Real-time/antivirus/tamper status, signature age, Defender detections from the last 24 hours. | Alerts and desktop notifications; does not re-enable Defender automatically. |
| Logons | Last 100 Security events 4624/4625, logon types, account and source IP. | Suspicious successful logons alert. Five or more failed events from one non-local IP cause a persistent inbound+outbound firewall block in armed mode. |
| Network | Established TCP connections and listening TCP sockets against a suspicious-port list; optional known-bad IP list (empty by default). | Suspicious ports/listeners alert only. A populated known-bad IP match can create a firewall block. |
| Processes | Process basename, path, known attacker-tool patterns, dual-use names, and execution from Temp/AppData/Public paths. | Known attacker-tool name patterns are force-stopped in armed mode. Dual-use names alert unless `-Aggressive` is set. Suspicious paths alert only. |
| Scheduled tasks | Enabled non-Microsoft/non-Tlamatini tasks, action paths, encoded PowerShell, and download/execute indicators. | Alerts only. |
| Services | Running non-Windows/non-Program-Files services, temporary/user paths, and missing descriptions. | Alerts only. |
| Registry persistence | Run/RunOnce, Winlogon Shell/Userinit, AppInit DLLs, and IFEO Debugger values. | Alerts only. |
| Critical directories | Executable/script/link extensions changed in the last 24 hours under Windows Temp, Public, user Temp, and Startup locations. | Alerts only. |
| Ransomware indicators | Recent event 4688 command lines for shadow-copy/recovery/log destruction; ransom-note names and bursts of five or more encrypted extensions in user data. | Critical alerts and notifications; it does not stop the process or restore files. |
| Account/privilege abuse | Security events 4720/4728/4732/4756 from the last 24 hours plus current local Administrators membership. | Alerts and inventory logging only. |
The monitor is primarily heuristic. A developer tool, legitimate red-team utility, installer, updater, administrative task, backup tool, encrypted archive, unusual service, or sanctioned account change can match these signals. Conversely, a real attacker can evade fixed names, extensions, ports, and event-window limits. Treat every result as a lead that needs corroboration.
### Self-safe classification: availability guard, not trust proof
The defender recognizes the parent of the active `security/` directory, `%LOCALAPPDATA%\Tlamatini` when present, and the registered installation root. Comparisons use literal normalized path boundaries. `Test-IsSelf` compares a process/file path to those roots. A matching process is not auto-killed. Names such as `nmap`, `ncat`, `john`, and `hashcat` are classified as dual-use and only alert by default; `-Aggressive` may stop them when they run outside a self root.
This protects Tlamatini's Nmapper, Kalier, Discoverer, and related work from accidental termination, but **a path match is not a code-signing or provenance check**. If an attacker places a payload inside a recognised/excluded Tlamatini tree, both the defender's self-safe rule and the Defender path exclusion can reduce scrutiny. Protect write access to the install/source tree, review modifications, and never treat "self" as equivalent to "trusted".
### Armed response and rollback
Explicit `-Armed` mode has two containment actions:
1. `Block-SuspiciousIP` creates `Tlamatini Block <IP> Inbound` and `Tlamatini Block <IP> Outbound` Windows Firewall rules.
2. `Stop-SuspiciousProcess` uses `Stop-Process -Force` for a process classified by the known attacker-tool name patterns, after refusing a recognised self path.
The firewall rules have no expiration and remain after the defender exits. List them with:
```powershell
Get-NetFirewallRule -DisplayName "Tlamatini Block *"
```
Before removing a block, correlate its source IP, event times, account, process/network evidence, and organisational policy. Then remove only the validated pair, naming the exact IP-specific rules rather than deleting every Tlamatini rule. Preserve the relevant log lines as the audit record for both containment and release.
There is also no automatic restoration for a killed process, no quarantine copy, no memory capture, and no process-tree containment. During a real incident, preserve evidence and follow your response plan before restarting software or deleting artifacts.
### Read and protect the logs
`Write-Alert` appends every entry to both `security_logs\alerts.log` and `security_logs\monitor.log`; the latter is not currently a separate lower-level stream, so expect substantial duplication and repeated findings in watch mode. There is no built-in rotation, retention limit, deduplication database, or automatic SIEM forwarding.
Severity means **triage priority**, not certainty:
- `INFO` records health, inventory, sweep boundaries, and successful checks.
- `WARNING` records weaker heuristics, stale protection state, access failures, and suspicious paths.
- `ALERT` records stronger leads, dual-use tools, suspicious logons, account events, and detect-only response previews.
- `CRITICAL` records high-risk patterns, armed containment actions, ransomware indicators, Defender disablement, and known attacker-tool names.
The logs may contain usernames, administrator-group membership, IP addresses, executable paths, registry values, task arguments, and command lines. Script-block and process-creation auditing can also place sensitive arguments in Windows event logs. Restrict access, establish retention, and redact before sharing.
### Packaging, updates, and self-modification
`build.py` copies the entire repository `security/` tree beside the installed executable and excludes `security_logs`, `*.log`, and `__pycache__`. On self-update, `security/` is treated as **application code**: a new release replaces the scripts, which is what you want — a fixed defender has to be able to reach a user who already installed a broken one. But `security/security_logs/` is the **operator's evidence** and lives inside that replaced directory, so like the database it needs separate handling: `apply_update.ps1` moves it to `Temp/_security_logs_carryover_<unique-id>` before the delete (step 3c) and moves it back into the new `security/` afterwards (step 5b). A failed stash aborts before application deletion. A failed restore leaves the unique stash intact without overwriting other evidence. `copy_source_assets.py` includes the `.ps1`, `.bat`, `.py`, and Markdown source in self-modify snapshots while pruning every directory named `security_logs`. `.gitignore` likewise excludes `/security/security_logs/`.
These rules keep test screenshots and host telemetry out of Git, the public installer, and self-modify source snapshots. They do not encrypt the logs on the local machine; that remains an operator responsibility.
### Blue-hat deployment checklist
- Read and diff every `security/` asset before elevation.
- Run the non-destructive visible test and require exit code `0`.
- Record Defender, ASR, CFA, firewall, execution-policy, audit-policy, and Security-log baselines.
- Confirm organisational policy permits exclusions, ASR Audit mode, command-line auditing, and Script Block Logging.
- Run the whitelist once, review every warning, and restart the relevant sessions.
- Run detect-only first and document expected false positives.
- Protect the Tlamatini root as a privileged/excluded trust boundary.
- Arm only with an operator present; avoid `-Aggressive` during ordinary development or sanctioned security testing.
- Review persistent firewall blocks and log retention after every armed/watch session.
- Escalate confirmed compromise to a real incident-response process; do not rely on this toolkit alone.
> **Created by Angela López Mendoza (@angelahack1)** — Tlamatini, the one who knows.
---
# Part I — Getting Tlamatini Running
## 1. What is Tlamatini?
**Tlamatini** (Nahuatl for "one who knows") is a local-first AI development and automation environment. It runs in your browser, keeps its control plane on your machine, knows your code, and can actually *do* things — not just describe how to do them. “Local-first” identifies where the application, database, workflows, credentials, agent programs, tools, files, and hardware connections are controlled; it does **not** mean Tlamatini was designed as a local-model-only assistant. Its intended reasoning layer uses Ollama cloud models.
The four things Tlamatini gives you that a plain ChatGPT-style box does not:
1. **A real RAG pipeline** that reads your project files, classifies their architectural roles, and grounds answers in your real source code.
2. **Multi-Turn mode** that turns the chat into a tool operator: the LLM can run shell commands, hit APIs, send emails, take screenshots, type into windows, query SQL — and chain those steps to finish the job.
3. **ACPX** that lets the LLM delegate sub-tasks to external coding-agent CLIs you already have installed (Claude Code, Cursor, Codex, Gemini CLI, Qwen Code, and more).
4. **A visual workflow designer** where you drag 85 different agent types onto a canvas (including the microcontroller-firmware trio **STM32er** / **ESP32er** / **Arduiner**, the **Unrealer** for driving Unreal Engine 5 — see bonus chapter §57 — the **Camcorder** for grabbing photos/video off a webcam, the **Recorder** for capturing audio off a microphone, the matching **AudioPlayer** / **VideoPlayer** for playing sound to the speakers and video on a screen, the **Talker** for synthesizing speech from text, and the **Whisperer** for transcribing speech back into text), wire them up, and run the result as an unattended `.flw` workflow. Save, Validate, and Start all funnel the canvas through a backend **Flow Compiler** (`agent/services/flow_compiler.py`) that consults a single Agent Contract registry — so a flow that runs in source mode runs identically in a frozen `.exe` install.
The **control plane is local**; the intended reasoning plane is cloud-backed through Ollama. The small retrieval embedding model runs locally, while configured `:cloud` models handle demanding chat, planning, coding, tool selection, long contexts, and vision. Prompts and context sent to those models are processed by Ollama's cloud service. The whole application can still be packaged as a standalone Windows `.exe`, but the executable does not remove the Pro-or-higher service requirement for complete functionality.
## 2. What you need before installing
| Requirement | Recommended | Notes |
|---|---|---|
| **Python** | 3.12.10 | **Installer users do NOT need Python** — as of **v1.17.0** the installer carries a self-contained Python 3.12.10 (with all dependencies) into `<install_dir>\python\`, and every pool agent runs on that carried interpreter. Python 3.12.10 is required **only** when running from source (Path A). It is the only version Tlamatini has been fully tested with. |
| **Operating system** | Windows 11 | The visual designer is cross-platform; some Windows-specific helpers (Mouser, Keyboarder, `.flw` / `.fpmt` file associations) are best on Windows. |
| **Disk space** | ~10 GB | Application dependencies, carried runtimes and browsers, plus the local embedding model; optional local LLMs require additional space. |
| **RAM** | 16 GB minimum | 32 GB is comfortable for the bigger embedding models. |
| **Model runtime** | **Ollama** | The local Ollama daemon exposes one API for the local embedding model and the cloud reasoning models Tlamatini uses. |
| **Cloud capacity** | **Ollama Pro or higher** | Mandatory for Tlamatini's complete intended functionality. Free/local-only operation is a limited experimental compatibility path, not the supported full-workload baseline. |
| **An IDE or editor** | Optional | Tlamatini ships with built-in "Open in VS Code / Antigravity / File Explorer" buttons if those are installed. |
You do **not** need administrator rights to install Tlamatini or Ollama if you follow the per-user paths in chapters 3 and 6.
## 3. Installing Ollama (no admin rights needed)
The official Windows PowerShell installer supports a per-user installation. Open PowerShell normally (do *not* right-click → Run as administrator).
### 3.1. Install Ollama into your user profile
```powershell
$env:OLLAMA_INSTALL_DIR = "$env:LOCALAPPDATA\Programs\Ollama"
irm https://ollama.com/install.ps1 | iex
```
This forces the install into a folder you already own (`%LOCALAPPDATA%\Programs\Ollama`) — no machine-wide location, no elevation prompt.
### 3.2. Reopen PowerShell, then verify
Close the PowerShell window. Open a fresh one. Then:
```powershell
ollama --version
```
If you see a version number, Ollama is installed and on `PATH`. If PowerShell complains it does not recognize `ollama`, close it once more, open a fresh window, and try again — the new `PATH` only takes effect in shells launched *after* the install.
### 3.3. Make sure the Ollama service is up
Tlamatini expects Ollama at `http://127.0.0.1:11434`. On most Windows installs Ollama starts on its own. If it is not running, leave a dedicated terminal open with:
```powershell
ollama serve
```
Verify it answers:
```powershell
Invoke-WebRequest http://127.0.0.1:11434/api/tags -UseBasicParsing
```
A normal HTTP response (any non-error code) means Ollama is reachable.
## 4. Pulling the default models
Tlamatini ships with default model names in `Tlamatini/agent/config.json` and several agent `config.yaml` files. Pull them exactly as written:
```powershell
ollama pull Nomic-Embed-Text:latest
ollama pull deepseek-v4.1-flash:cloud
ollama pull jcyhsiao/qwen3.5cloud:latest
ollama pull gemma4:cloud
```
| Model tag | Used by |
|---|---|
| `Nomic-Embed-Text:latest` | RAG embedding model (default — light VRAM footprint, ~600 MB resident) |
| `deepseek-v4.1-flash:cloud` | Default chat model, Multi-Turn unified-agent model, MCP file-search, internet classifier and web summarizer — and the default `model` of most workflow-agent templates (FlowCreator, FlowHypervisor, Crawler, File-Interpreter, Monitor-Log, Monitor-Netstat, Notifier, PDFer, Prompter, PSer, RecMailer, Reviewer, Summarizer, TeleTlamatini, Instant Messaging Doctor, and LaTeXer's repair rung 7) |
| `jcyhsiao/qwen3.5cloud:latest` | Image-Interpreter interpreter 1 (forensic OCR / measurement) and Video-Analyzer interpreter 2 |
| `gemma4:cloud` | Image-Interpreter interpreter 2 and Video-Analyzer interpreter 1; selectable in Config → Models |
Some pulls are large and slow. Start them, walk away, come back.
> **Model names are substitutable; the capacity baseline is not.** You may replace a cloud tag with another compatible Ollama cloud model in `config.json` (Part VII) or an agent's `config.yaml`, provided the chosen name appears in `ollama list`. Replacing every cloud tag with local models is possible as an advanced experiment, but it does not represent the complete, supported Tlamatini configuration and does not remove the documented Pro-or-higher requirement for full functionality.
> **High-detail embedding opt-in.** If your retrieval quality on dense, technical corpora is not good enough with `Nomic-Embed-Text:latest`, you can swap it for `qwen3-embedding:8b` from the **Config → Models** menu inside the app (or by editing the `embeding-model` key in `config.json` and reconnecting). **Use with caution**: `qwen3-embedding:8b` is roughly **10× heavier in VRAM** than the default (~6.24 GB resident vs ~600 MB on a Q4_K_M quant) and will trip the embedding-memory pre-flight guard (Part §34) on 8 GB consumer GPUs. Pull it first with `ollama pull qwen3-embedding:8b`.
## 5. Complete Tlamatini operation requires Ollama Pro or higher
Three of the four default models in chapter §4 are **cloud-backed** — `deepseek-v4.1-flash:cloud`, `jcyhsiao/qwen3.5cloud:latest`, and `gemma4:cloud` (only `Nomic-Embed-Text:latest` runs locally). Most of them carry a literal `:cloud` suffix, but not all: `jcyhsiao/qwen3.5cloud:latest` is that same Qwen3.5 cloud model republished under a community namespace, so the suffix is not the test. The reliable test is `ollama list` — a cloud-backed tag shows a blank **SIZE** column, because nothing was downloaded to your disk. Those models are not actually running on your machine. They live on **Ollama Cloud**, and the `ollama pull <tag>` command registers them with the local daemon, which proxies inference requests to Ollama's service. You must have an Ollama account and be signed in on the host that runs Tlamatini. Ollama documents this local-API/cloud-offload design in its **[official cloud-model documentation](https://docs.ollama.com/cloud)**.
Ollama may make limited cloud use available to free accounts. That does not make Free the operational baseline for this application. **Tlamatini requires an active Ollama Pro plan, or a higher plan such as Max, for its complete intended functionality.** Multi-Turn orchestration, long agent runs, FlowCreator, parallel Image-Interpreter calls, repeated tool decisions, and large project contexts can use cloud quota and concurrency much faster than a single conversational request.
### 5.1. Plans and real account allowances
Pricing and limits change. The following is a dated reference checked on **2026-10-10**,
not a fixed entitlement used by Tlamatini. See [Ollama pricing](https://ollama.com/pricing).
| Plan | Current published allowance | Fit and reporting |
|---|---|---|
| **Free** | Starter usage; additional credits can be purchased. | Light experiments; below Tlamatini’s documented full-workload baseline. Show the account’s actual response. |
| **Pro** | $60 monthly credits; 3 concurrent requests. | The project’s minimum intended tier for full cloud workloads. |
| **Max** | $300 monthly credits; 10 concurrent requests. | Higher capacity for sustained work; use actual returned balances. |
| **Team / Enterprise** | Team advertises $1,000 shared monthly credits; Enterprise is custom. | Shared balances and member usage have different scopes; do not assume organization-wide totals. |
There is no current listed cloud plan named **Unlimited**. Unlimited execution on
your own hardware does not imply unlimited cloud credits. Older Pro/Max accounts
can still report session and weekly limits; their percentages must not be
converted into dollars. [About Usage](docs/usage-dashboard.md) adapts to these
response shapes and explains missing fields.
### 5.2. Why this requirement is not a sponsorship
**XAIHT/Tlamatini is independent from Ollama.** The project is not sponsored, endorsed, affiliated with, funded by, or compensated by Ollama. There is no referral link, commission, revenue share, or other commercial relationship. Users subscribe directly with Ollama, and pricing, availability, quotas, model access, service operation, and privacy terms remain Ollama's responsibility.
The book names Pro because Tlamatini was designed and coded around Ollama's cloud-model interface and practical capacity. This is the same kind of technical disclosure as naming a minimum Python version or a required hardware interface: it tells the reader what the system was built and tested to use. It is not an advertisement and it does not imply that Ollama requires every application to purchase Pro.
### 5.3. What if I do not want to subscribe?
Tlamatini can be launched and experimentally reconfigured without Pro, but that is **a limited compatibility path, not the complete supported product configuration**. A free account may permit light cloud testing. An advanced user with sufficiently powerful hardware may replace cloud tags with local open-weight models. Neither path is represented as equivalent to the cloud-backed system on which Tlamatini's full orchestration behavior was designed and tested.
To experiment with an all-local configuration, use **Config → Models** (or the active source/installed `config.json`) and select locally available models suited to each role. The following older examples illustrate the core/vision subset; they are not a complete migration checklist or a current availability test:
| Config key | Default (cloud) | A reasonable local substitute |
|---|---|---|
| `chained-model` | `deepseek-v4.1-flash:cloud` | `qwen2.5-coder:14b` or `llama3.1:8b` |
| `unified_agent_model` | `deepseek-v4.1-flash:cloud` | same as above |
| `mcp_files_search_model` | `deepseek-v4.1-flash:cloud` | same as above |
| `internet_classifier_model` / `web_summarizer_model` | `deepseek-v4.1-flash:cloud` | same as above |
| `image_interpreter_model` | `jcyhsiao/qwen3.5cloud:latest` | `llama3.2-vision:11b` (a local vision model — pull it first with `ollama pull llama3.2-vision:11b`) |
| `image_interpreter_model_2` | `gemma4:cloud` | same as above |
| `image_merging_model` | `deepseek-v4.1-flash:cloud` | `qwen2.5:32b` or any large local model you can fit in VRAM |
Use **Config → Models** to review all six categories, including Video vision, speech, workflows, documents and monitors; the table above is only a core/vision subset. FlowCreator now uses the global `flowcreator_model` key through `llm.model: "@config"`, and the other model-backed agents inherit their dedicated global keys in the same way. Existing workflows with literal tags remain explicit overrides: set their registered model fields to `"@config"` if they should follow the global choices. Reconnect chat and restart already running agents after saving. Direct cloud speech/Claude paths and external ACPX providers need separate consideration in an all-local setup. [The full reference](docs/model_configuration.md) lists every setting and compatibility constraint. Quality, latency, context capacity, vision support, tool-calling reliability, and concurrency will depend on the exact local models and hardware. Some individual features may work; **complete behavioral equivalence is neither promised nor the supported baseline**.
### 5.4. This subscription is separate from your coding-agent API keys
The Ollama plan only governs `*:cloud` Ollama models. If you plan to use **ACPX** (chapter §46) to delegate sub-tasks to external coding-agent CLIs (`claude`, `cursor-agent`, `codex`, `gemini`, `qwen-code`, …), each of those carries its own credentials: the Anthropic API key for `claude`, OpenAI for `codex`, Google for `gemini`, and so on. Those keys are configured in `Tlamatini/agent/config.json` under the top-level fields *and* the per-agent `acpx.agents.<id>.env` blocks, and they are completely independent of your Ollama subscription. The `setup-new-acpx-key` skill (chapter §15) automates that wiring.
### 5.5. Troubleshooting the cloud path
| Symptom | Likely cause | What to try |
|---|---|---|
| `ollama pull deepseek-v4.1-flash:cloud` succeeds but inference returns "unauthorized" / "401" | Not signed in to Ollama on this host. | Run `ollama signin` (or use the Ollama desktop app) and confirm `ollama whoami` prints your account. |
| Inference returns "rate limit exceeded" / "429" right after a Multi-Turn step | Your plan's concurrent-model or monthly-usage cap is full. | Either upgrade the plan, drop concurrency by running fewer wrapped agents in parallel, or swap one of the cloud tags for a local model in `config.json`. |
| Inference returns "model not available on this plan" | The tag you pulled is gated to a higher tier (Pro/Max only). | Check `ollama.com/pricing` for which models each tier covers, and pick a tag your plan includes — or upgrade. |
| Tlamatini chat says "Ollama backend unreachable" | Local daemon is down, **not** a cloud problem. | `ollama serve` and `Invoke-WebRequest http://127.0.0.1:11434/api/tags -UseBasicParsing` per chapter §3.3. Cloud requests still go through the local daemon. |
## 6. Installing Tlamatini
You have three paths. Pick one.
### Path A — From source (developers, contributors, full control)
```bash
git clone https://github.com/XAIHT/Tlamatini.git
cd Tlamatini
python -m venv venv
# Windows:
venv\Scripts\activate
# Linux/macOS:
source venv/bin/activate
pip install -r requirements.txt
python Tlamatini/manage.py migrate
python Tlamatini/manage.py createsuperuser
python Tlamatini/manage.py collectstatic --noinput
```
When the migrations finish and you have a superuser, run the server (chapter 7).
### Path B — Pre-built one-click installer (end users)
This book covers **v1.92.4**. Torrenter adds account-owned downloads to chat and the Agentic Control Panel. Live Editor and Terminater accept up to 250 WPM. The updater carries the uninstaller and preserves the vault. Release builds lease the Python environment for their requirements fingerprint. Obtain the intended package from [Tlamatini Releases](https://github.com/XAIHT/Tlamatini/releases) and verify its version before installing. The source tag is not itself proof of a published or accepted package.
1. Open the unzipped folder.
2. Double-click **`Installer.exe`**.
3. Pick a destination directory (any folder you can write to — no admin needed).
4. Click **Install**.
5. The installer creates a desktop shortcut, registers `.flw` and `.fpmt` as Tlamatini file types, copies in the bundled Ollama-default `config.json`, and creates a default user (`user` / `changeme`). Since v1.82.0, if anything fails the window stays open with **Retry** and **Change folder**, a verified recovery copy puts back whatever was changed, and the last page shows where its log was written.
That is it. Double-click the desktop shortcut to launch. The carried Python 3.12.10, Java, Git and Playwright browsers all come inside the ZIP — nothing else to install except Ollama and the models from chapters §3–§4.
### Path C — Build the installer yourself (releasers)
See **Part VIII — Deploying & Packaging**. Three scripts, run in order: `build.py` → `build_uninstaller.py` → `build_installer.py`.
## 7. First login
After step 5 (any path), run the server:
```bash
python Tlamatini/manage.py runserver --noreload
```
> **`--noreload` is optional (since 2026-07-11):** plain `python Tlamatini/manage.py runserver` also boots clean and auto-reloads — it used to double-start the MCP ports `:8765` / `:50051` and crash with `WinError 10048`, now fixed in `agent/apps.py`.
(For installer builds, just double-click the desktop shortcut.)
Open `http://127.0.0.1:8000/` in your browser.
| You used... | Log in with |
|---|---|
| Path A (from source) | The superuser you created with `createsuperuser` |
| Path B (installer) | `user` / `changeme` |
| Path C (your own build) | Whatever the build configured |
> **Change the default password.** If you used the installer's default `user / changeme`, the very first thing to do after logging in is open `/admin/`, find your user, and change the password. Especially before exposing the host on a network.
After login you arrive at the welcome page. **Just press Enter** and you are in the chat — that is the default action, so nothing has to be clicked or tabbed first (if you Tab to *Logout* and press Enter you still log out; the shortcut steps aside whenever a button already holds focus). Otherwise click into **`/agent/`** for the chat or **`/agentic_control_panel/`** for the visual designer. The chat is what Part II is about.
---
# Part II — Using the Chat (the dummy walkthrough)
## 8. A tour of the chat page
Open `/agent/agent/` (the page the welcome screen's **Go to Chat** takes you to). Here is what you are looking at:
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ Tlamatini [Open] [Save] [Context ▼] [Panels ▼] [ACPX-Skills ▼] [External ▼] │ ← Top navigation
│ [Config ▼] [DB ▼] [Reconnect] [About ▼] │
├─────────────────────────────────────────────────────────────────────────────┤
│ Multi-Turn ☐ Exec Report ☐ ACPX ☐ Ask Execs ☐ Add internet context ☐ │ ← Toolbar (the five checkboxes!)
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────┐ ┌───────────────────────────────┐ │
│ │ CHAT WINDOW │ │ CODE CANVAS │ │
│ │ (conversation history) │ │ (generated code lives here, │ │
│ │ │ │ syntax-highlighted, with │ │
│ │ │ │ copy / save buttons) │ │
│ │ │ │ │ │
│ └─────────────────────────────────┘ └───────────────────────────────┘ │
│ │
├─────────────────────────────────────────────────────────────────────────────┤
│ Type your prompt here… [ Mic ] [ Send ] │ ← Input bar
└─────────────────────────────────────────────────────────────────────────────┘
```
The five checkboxes in the toolbar are **the** thing to learn. Each one is explained in its own chapter below. They are independent — except **Ask Execs**, which only activates while **Multi-Turn** is ticked — so tick whatever combination fits your task.
The navbar also has a **Config** dropdown now. It exposes two validated dialogs: **Models** for 38 model, engine and voice settings in six searchable categories, and **URLs** for the Ollama / unified-agent / MCP endpoint values. That means the most common runtime settings can now be changed from the chat UI without manually editing `config.json`. The chat/canvas divider was also polished so width changes feel steadier while you work.
**The menu bar was reorganized on 2026-09-25.** Two things you may be looking for moved: **Configure MCPs** and **Configure Agents** now live inside **Config**, right above Models and URLs, and there is a new **Panels** menu that opens the **Agentic Control Panel**, the **Prompt Flow Panel** and — if your account is staff — the Django **Admin Panel**, each in its own tab. The old separate *MCPs*, *Agents* and *Admin* menus are gone. While Tlamatini is busy answering you, most menus grey out so nothing can change under her feet — but **Panels stays open**, because opening another panel in a new tab never disturbs the answer she is writing.
### Speaking instead of typing
The microphone beside Send starts the [direct voice-prompt route](#voice-commands-your-words-become-the-prompt). It listens on the Tlamatini host, shows recording and silence progress inside this composer, then transcribes and sends automatically. Click again or press Escape to cancel. It uses the existing draft and selected modes; no extra terminal opens.
### Pasting a screenshot into the chat (2026-07-14)
You can hand Tlamatini a picture without ever leaving the chat box. Take a screenshot (Print Screen, or a snip), Alt+Tab back to Tlamatini, click where you want it and press **Ctrl+V**. She saves the image into her own `Temp` folder as `image_<timestamp>.jpg`, shows a small **thumbnail chip** just above the input, and writes the image's **full path into your message at the cursor** — so you can keep typing around it: *"Look at C:\Tlamatini\Temp\image_20260714_005340_168.jpg and tell me why the button is cut off."* Then send.
Dropping image files from Explorer onto the chat does exactly the same thing (since v1.83.0 through the page-wide file-drop router described below). Each image gets its own chip; clicking a chip's **×** removes both the thumbnail and its path from your message, so a mis-paste costs nothing.
The reason it hands you a *path* instead of an attachment is that a path is what her vision agent actually eats: **Image-Interpreter** reads the file from disk and answers in text. Tick **Multi-Turn** and she will do it herself — read the screenshot, then act on what she saw.
### Dropping a message (2026-09-30)
Every message card has two small buttons in its header: **Copy** and **Drop**. Drop works on your own messages and on Tlamatini's answers alike.
Press **Drop** and a confirmation explains exactly what happens next: she forgets that message, it leaves the chat, and from your next message on she answers as if it had never been written. Everything before and after it stays exactly as it was, and nothing reconnects. **Cancel** is selected when the dialog opens, so a stray Enter never deletes anything; the red **Drop** confirms, and Escape backs out.
Why it needs no reconnect: Tlamatini re-reads your conversation from her database before every answer (the newest eight saved messages). Nothing else keeps a copy — no cached summary, nothing inside the chain — so removing one saved message is all it takes. Because the window is "the newest eight", an older message may slide back into view, exactly as if the dropped one had never existed.
A few details worth knowing:
- Dropping only her answer leaves your question behind, so she may treat it as not answered yet. Drop both to erase the whole exchange.
- When you drop your own message, the hidden "Referenced Rephrase" rows she saved for it go too, so a page reload cannot bring the dropped words back.
- Status lines such as "I'm working on your request…" were never saved; dropping one only tidies your screen.
- You cannot drop while she is still answering — that answer already read its history.
- If her **memory tool** (External ▸ MCPs ▸ memory) is switched on, she may have copied a fact into that long-term memory. That copy is separate from the chat and stays; ask her to forget it too. The dialog reminds you of this.
## Canvas, Views and avatar in v1.85.0
**Open** chooses a document. **Refresh** reads that same document again without a
file picker. It reports unchanged content, reloads changed content and retains
the previous document when a read fails. Markdown keeps Preview/Source and its
resources; PDF keeps page, zoom, rotation and scroll mode. Snapshot-only sources
explain why Refresh is unavailable. See [Canvas Refresh](docs/canvas-refresh.md).
**Views → Canvas** hides or shows the canvas. Hiding makes chat full width while
keeping the document, refresh handle, PDF viewer, draft, chat history and context.
The setting is saved per account in this browser and synchronized between tabs.
Open, file drops and generated-document links reveal the canvas when needed.
While the canvas is hidden, the avatar is docked and its floating resize handle
is disabled. Showing it again enables the handle and keeps the avatar docked.
**Views → Reset layout** shows both panes, restores the initial divider positions
and docks the avatar, without clearing content. Shrinking the message box keeps
the portrait and its frame horizontally centered between Mic's left edge and
Send's right edge. The portrait remains square with a 120 px minimum.
### Dropping a file anywhere (v1.83.0, 2026-10-14)
Drop a file on any Tlamatini page and she works out what it is — from its **bytes**, not just its name — and sends it where it belongs. One small script, `file_drop_router.js`, listens on every page (the chat, the Agentic Control Panel, the Prompt Flow Panel, the welcome and login pages, the Django admin, and the PDF viewer inside the chat canvas). It only ever claims file drags: dragging text or a palette item is left alone, and the browser no longer wanders off to open the file by itself. Before it, an MCP `.json` dropped on the chat reported an error, on the canvas was opened as text (which killed the import) and on every other page navigated the browser away to the file.
| You drop | What happens |
|---|---|
| An MCP server `.json` (any shape people paste into a desktop MCP client, even a UTF-16 file saved by PowerShell) | A confirmation first — it lists the servers and warns they can launch programs — then the servers join the **External ▸ MCPs** catalog, on any page; an open External ▸ MCPs dialog refreshes. Never silent |
| A `.flw` agent flow | Loads in the Agentic Control Panel; from another page it opens the panel in a new tab |
| A `.fpmt` prompting flow | Loads in the Prompt Flow Panel (asking before it replaces your diagram); from another page it opens in a new tab |
| A picture | Attached to your message, exactly like Ctrl+V (saved in `Temp`) |
| A PDF, a Markdown file, source code or plain JSON | Opens in the chat canvas (on the chat page) |
| A folder, an archive or any other binary | A plain-words note saying where that kind of file can be used |
While you drag, an overlay tells you what will happen on that particular page. A JSON file counts as an MCP config only if every entry in it is a server, so ordinary JSON is never hijacked, and the login page never imports or navigates anything. Each page tells the router what it can do with `TlamatiniFileDrop.register(kinds, fn, {label})`, and the label is also the text of the drag overlay. Checked by 138 router cases and a headed Chrome run of 58 of 58 checks with 24 Shoter photos; the contract is in `docs/claude/frontend.md`.
### The message box and her avatar (v1.83.0, 2026-10-13)
With her avatar docked above the buttons, the vertical divider between the chat history and the message box now shrinks the portrait as well as the box. It used to reserve the portrait's current width as the box's minimum height, so a wide portrait locked the divider in place. Now the docked portrait's side is a quarter of the form's width, but never more than the room above the Send and Mic row and never less than 120 px (a 106 px face with its 22 px resize handle). The first, quarter-width portrait is kept until you first move the divider, and undocking her keeps the floating size you chose. At 1440 CSS px the portrait went from 282 px to 120 px, returning about 162 px to the chat history. Verified in headed Chrome (19 docked-avatar and 18 existing composer checkpoints); a change credited to Tlamatini's-AutoBot ([record](docs/changes/2026-10-13-docked-avatar-divider.md)).
### About → Usage: read credits while she works
Open **About → Usage** at any time, including during a long answer. **Cloud credits**
shows the configured Ollama account’s reported monthly and purchased balances,
monthly usage and refill weeks/date. Its 7/30-day spend and token charts are a
separate rolling history. **Chat activity** and **Models** show measured calls for
your Tlamatini user since recording began, with peak UTC dates and model graphs.
They do not reconstruct older history or count other applications and child agents.
The **ABOUT USAGE** window occupies at most about 60% of the client area, follows
the normal dialog typography and theme, and displays at most two decimal places.
Refresh follows the context gauge; account requests are cached for one minute.
If a field is unavailable, the dialog explains it and offers **Ollama Usage ↗**.
Legacy session/weekly limits are percentages, not invented dollar allowances.
This feature was verified from source on 2026-10-10 and awaits a rebuilt release.
For plan behavior, data boundaries, refresh timing and dated tests, read the
[complete Usage guide](docs/usage-dashboard.md).
## 9. Asking your first question (no toggles)
Leave every checkbox unticked. This is the **simplest** possible chat: one question in, one answer out.
**Try this:**
> "Write a Python function that validates an email address using a regular expression. Just the function, no main."
The bot answers in the chat panel. If it generates code, you will see it appear in the code canvas on the right with a filename header and a copy button.
What is happening under the hood:
1. Your message goes to the server over WebSocket.
2. Tlamatini decides it is a programming question (not a system / files question), runs prompt-shape validation, and sends it to the LLM.
3. The LLM responds in one shot. Done.
This is the **legacy one-shot chat path**. It is fast, deterministic, and uses no tools. It is also intentionally limited: the LLM cannot run anything on your machine, cannot search the web, cannot read files. For that, you tick checkboxes.
## 10. Setting code as context
Most of the time, you want the bot to answer questions about *your* code, not generic Python. Click **Context** in the top nav:
| Menu entry | What it does |
|---|---|
| **Set directory as context** | Loads a folder. Tlamatini reads every text file in it, splits them into chunks, embeds them, builds a hybrid FAISS+BM25 index, and grounds every answer in those files. |
| **Set file as context** | Loads a single file (smaller scope, faster). |
| **Set canvas as context** | Uses the code currently shown in the right-hand canvas as context (handy for iterative editing). |
| **Clear context** | Removes the current context and rebuilds the chain empty. |
> **PDFs take a different door.** A PDF is not loaded from this menu: you open it in the canvas with **Open** and then press the canvas's own **Use as context** button, which reads the whole document (and, if you ask, looks at its pictures). See *Reading a PDF: open it in the canvas, then hand her the whole document* near the front of this book.
After you set a context, the top of the page shows a green banner with the path. Now ask:
> "How does authentication work in this project?"
The bot will quote real files, reference real classes, and stay grounded. If it cannot find the answer in the loaded files, it will say so rather than hallucinate.
> **What if my model runs out of memory?** If Ollama returns "model requires more system memory" while building the embedding index, Tlamatini does **not** wipe the loaded files. It packs them into a fallback context and keeps answering from the raw source until embeddings can be built again. Retrieval quality drops; access to your code does not.
One subtle but important reliability fix landed here: when a browser refresh restores a saved context, the chat input is now disabled immediately and remains disabled until the contextual RAG chain has really finished rebuilding. In older builds, the user could briefly type into a half-restored session because the welcome-back banner arrived before the context-loading lifecycle had finished.
## 10.5. Binary files are skipped automatically (and the log tells you which)
Section 10 showed you how to point Tlamatini at a folder. Real folders are messy: alongside your source code there are compiled binaries, images, ZIPs, PDFs, model weights, databases and build leftovers. None of that is readable text, and feeding it into the embedding index actively hurts you — it wastes GPU memory and time, and it buries your actual code under noise so the answers get worse.
Since 2026-07-26 Tlamatini handles this for you. **Every file is screened by its real content before it is loaded**, and the binary ones are quietly left out. You do not have to configure anything.
**This is not the same thing as "Set file type omissions".** Both exist and both run:
| | What it looks at | Who decides | Example it catches |
|---|---|---|---|
| **Set file type omissions** (Context menu) | the file's **name** | you | `*.doc`, `package-lock.json` |
| **Binary guard** (automatic) | the file's **bytes** | Tlamatini | a `.pyc`, a `.so`, a PNG someone renamed `notes.md` |
**It is fast.** Known binary types (`.png`, `.exe`, `.zip`, …) are recognised from the extension and never even opened. Everything else is judged from a single 8 KiB peek at the start of the file — so checking a 4 GB video costs the same as checking a README.
**It tells you what it did.** Open `tlamatini.log` and search for `BINARY-GUARD`:
```
--- [BINARY-GUARD] 3 binary file(s) OMITTED from the context / embedding chain
--- [BINARY-GUARD] Detected by: extension=2, signature=1
--- [BINARY-GUARD] ✗ OMITTED C:\proj\assets\logo.png [extension: known binary extension .png]
--- [BINARY-GUARD] ✗ OMITTED C:\proj\notes.md [signature: PNG image]
```
So if you ever wonder *"why isn't that file in my context?"*, the log answers it directly — file by file, with the reason.
**It errs on the side of keeping your files.** If a file cannot be read, or the evidence is ambiguous, Tlamatini loads it as text rather than dropping it. Text in Spanish, French or any legacy encoding is always kept. The guard only removes what it is confident about.
**If you want to change it**, in `config.json`:
- `"binary_context_detection": false` — turn the whole thing off (back to the old behaviour).
- `"binary_detection_force_text_extensions": [".dat"]` — force an extension to be treated as text; this always wins.
- `"binary_detection_extra_binary_extensions": [".myblob"]` — add your own binary types.
## 11. The "Add internet context" toggle
Tick **Add internet context** when the question genuinely needs information from the web. Examples:
- "What is the latest stable version of FastAPI as of right now?"
- "Show me a current example of using OpenAI's responses API."
- "What does the `2024-11-30` row in the React 19 changelog say?"
Tlamatini classifies your question with a small LLM call ("does this need the web?"), then runs a DuckDuckGo search, fetches and summarizes the top results, and inlines the summary into the LLM's context.
Leave it **unticked** for everything that does not need fresh web data. The classifier is fast but a web round-trip still adds latency.
## 12. The "Multi-Turn" toggle (turning Tlamatini into a *doer*)
This is the big one. Until you tick **Multi-Turn**, Tlamatini only *describes* things. With Multi-Turn ticked, Tlamatini can *do* them.
### What it does
Multi-Turn flips Tlamatini from "answerer" to **operator**:
- The chat skips its prompt-shape validator (you no longer have to phrase requests as questions).
- A request-scoped **planner** picks the relevant tools from **111 built-ins**: 21 direct/module tools, 68 wrapped chat-agents, 12 ACPX/Skill tools, and 10 External-MCP supervisors. Healthy active servers can add lazily discovered `ext__<server>__<tool>` remotes on top without inflating every prompt permanently.
- The unified-agent loop runs **up to 4096 iterations** (`unified_agent_max_iterations`): the LLM calls a tool, sees the result, decides what to call next, and chains its way to the goal.
- Wrapped sub-agents launch **silently** in the background (no console window pop-ups).
### When to use it
- "Run my project's test suite and tell me what failed."
- "Crawl this URL, summarize the result, and email it to me."
- "Open Notepad, type 'Hello world', save it as `out.txt`, then close it."
- "Install OpenCV in the `venv` here and verify it works." (See the demo video.)
- Any task where you would otherwise paste commands one at a time.
### Step-by-step: your first Multi-Turn run
1. Tick **Multi-Turn**. Leave the others unticked for now.
2. Type:
> "Take a screenshot of my desktop and save it to `C:\Tlamatini-test\shot.png`."
3. Hit **Send**.
4. Watch the chat. You will see:
- A short "Working on it…" sort of message.
- A final answer like "Done — saved to C:\Tlamatini-test\shot.png."
5. Open the file. The screenshot is there.
What just happened: the planner picked `chat_agent_shoter` (the wrapped Shoter agent), the LLM called it with the right `output_dir`, the agent took a screenshot, the LLM read the JSON result, and replied with the path.
### What you'll see if it goes well
- The LLM does **not** ask you for permission to take the screenshot — Multi-Turn means "you're an operator, not an advisor."
- A **Create Flow** button appears in the message header (chapter 16 explains it).
- The answer ends in a period, with no `END-RESPONSE` sentinel leaking through.
### Common pitfalls
| Symptom | Fix |
|---|---|
| LLM says "Tool X is not available" | The planner did not bind that tool. Check the `[Planner._select]` log lines in the console — adjust your prompt so the relevant keywords are present, or relax the threshold by raising `max_selected_tools` in `config.json`. |
| LLM tries to do everything in plain text | You forgot to tick Multi-Turn. Tick it and resend. |
| Tool runs but the LLM calls the same tool twice with identical args | This is suppressed by the dedup guard. The second call returns "skipped — duplicate" and the LLM moves on. |
| 4096 iterations exhausted | Your task probably hit a polling loop (e.g. waiting for an external service). Use `chat_agent_sleeper` instead of busy-polling, or break the task into two prompts. |
### Combine Multi-Turn with context
Multi-Turn and **Set context** stack. If your project is loaded as context, the LLM can ask the codebase about itself *and* run tools on the result. Example:
> "Find the file in this project that defines the `User` model, then run `python -c \"...\"` to dump its `__dict__` schema."
The planner pulls `chat_agent_executer` for the shell call; the loaded context tells the LLM where the model lives.
## 13. The "Exec Report" toggle (seeing every step)
### What it does
Below the LLM's prose answer, Tlamatini appends **per-agent execution tables** — one HTML table per wrapped Multi-Turn agent that actually ran during the request, each row a real tool call with a deterministic SUCCESS/FAILURE verdict. Observational and read-only agents are included too; the report records execution truth, not only filesystem mutation.
This is the "show your work" view. It is the ground-truth counterpart to the prose summary. The prose can be ambiguous; the tables are not.
### When to use it
- Always when you are debugging a Multi-Turn run.
- Always when you want to convert a chat into a flow (chapter 16).
- Whenever you want to be sure what *actually* happened on disk / on the network.
### Step-by-step
1. Tick **Multi-Turn** AND **Exec Report**. Both must be on.
2. Run a Multi-Turn task that touches multiple state-changing tools, e.g.:
> "Create the file `C:\test\hello.txt` with the content `Hi from Tlamatini`, then read it back and tell me its size."
3. Send.
4. After the prose answer, scroll down. You will see a styled table:
```
┌──────────────────────────────── List of File-Creator Operations ─┐
│ # │ Command │ ✓/✗ │
│ 1 │ filepath='C:\test\hello.txt' content='Hi from Tla…' │ ✓ │
└──────────────────────────────────────────────────────────────────┘
┌──────────────────────────────── List of Executer Operations ─────┐
│ # │ Command │ ✓/✗ │
│ 1 │ type C:\test\hello.txt │ ✓ │
└──────────────────────────────────────────────────────────────────┘
```
### What gets a table
Capture is automatic for every wrapped `chat_agent_*` tool that actually executes. That includes state-changing agents, diagnostics, readers, media output/capture, and observational agents such as Shoter, Camcorder, Recorder, Talker, AudioPlayer, VideoPlayer, Whisperer, Grepper, File-Interpreter, and Image-Interpreter. Direct execution tools, ACPX actions, and `invoke_skill` are also captured; management/polling helpers that do not perform the requested operation remain excluded. `_EXEC_REPORT_TOOLS` in `agent/mcp_agent.py` is now an optional styling/grouping map, not a capture whitelist.
The verdict engine reads the agent's own `INI_SECTION` self-report before the process exit code. Its vocabulary is closed and split into five disjoint classes: completed diagnostics, intact completed work, degraded work, work not done, and agent errors. A diagnostic that successfully finds a problem is green because the finding is its deliverable. A degraded or missing deliverable is red even when some output exists. Unknown status tokens remain fail-open for runtime continuity but are named by rule `R8b`, while `agent/test_status_vocabulary.py` scans every pool agent and prevents undeclared tokens from shipping. Kuberneter follows this contract by keeping its numeric result in `returncode`, publishing `success`, and using `status: ok|failed`.
### A persistence detail worth knowing
The Exec Report tables are **persisted into the chat history**, not just broadcast live. Reload the page — the tables are still there. This is intentional, and the order in `process_llm_response()` is strict: classify success → append exec-report HTML → save → broadcast. Do not reorder.
## 14. The "ACPX" toggle (delegating to external coding-agent CLIs)
### What it does
ACPX is the most ambitious feature. It lets Tlamatini **spawn external coding-agent CLIs** (Anthropic's `claude` CLI, Cursor's `cursor-agent`, OpenAI's `codex`, Google's `gemini`, Alibaba's `qwen-code`, plus eight more) **as child processes**, talk to them over stdin/stdout, and broker their output back to the LLM as if they were native tools.
Picture it like this:
```
┌──────────────────────────────────────┐
│ You (in the chat) │
└──────────────┬───────────────────────┘
│ "Use claude to refactor X, then have gemini critique it"
▼
┌──────────────────────────────────────┐
│ Tlamatini chat (the Multi-Turn LLM) │
│ acp_doctor → acp_spawn(claude) → │
│ acp_send_and_wait → acp_relay → │
│ acp_spawn(gemini) → acp_kill │
└──────────────┬───────────────────────┘
│ subprocess.Popen
▼
┌──────────────────────────────────────┐
│ External CLIs running on your box │
│ claude, gemini, cursor, codex, … │
└──────────────────────────────────────┘
```
The full ACPX deep-dive is in **Part VI §46**. This chapter is just the toolbar walkthrough.
### When to use it
Tick **ACPX** when:
- You want a sub-task delegated to a *different* LLM than the one driving Tlamatini's chat. (Example: your chat runs deepseek-v4.1-flash:cloud, but you want Claude Code to do the actual refactor because it is better at long-context Python.)
- You want a **multi-CLI relay** — Claude does the first pass, Gemini critiques, Cursor applies the fix.
- You need a coding agent that can edit files in the *current working directory* without the wrapping ceremony of Multi-Turn.
Untick **ACPX** when you don't have any external CLIs installed, or when you want the legacy pre-ACPX Multi-Turn flow. The 12-tool ACPX surface is filtered out of the planner — the LLM never sees it.
### Step-by-step: your first ACPX run
**Prereq:** at least one external CLI installed and on PATH. The simplest is Anthropic's:
```bash
npm install -g @anthropic-ai/claude-code
claude --version
```
Then put your API key into `Tlamatini/agent/config.json`:
```json
{
"ANTHROPIC_API_KEY": "sk-ant-api03-...",
"acpx": {
"agents": {
"claude": { "env": { "ANTHROPIC_API_KEY": "sk-ant-api03-..." } }
}
}
}
```
(Two layers — top-level for Tlamatini's own Anthropic calls, and `acpx.agents.claude.env` for the spawned `claude` CLI. The skill `setup-new-acpx-key` automates this; see chapter 15.)
Now in the chat, tick **Multi-Turn** AND **ACPX** AND **Exec Report**, and type:
> "Use ACPX to spawn the claude CLI in `C:/Development/Tlamatini`, ask it to summarize CLAUDE.md in 5 bullet points, harvest the answer, and kill the session."
You will see:
1. The LLM calls `acp_doctor` first (always).
2. Then `acp_spawn(agent_id="claude", task="...summarize CLAUDE.md...")`.
3. A `session_id` comes back; the LLM uses it.
4. `acp_send_and_wait` to ensure the answer is fully drained.
5. `acp_kill` to shut the child cleanly.
6. The 5-bullet summary appears in the prose answer.
7. The Exec Report shows a "List of ACPx Operations" table with all four tool calls.
### Common pitfalls
| Symptom | Fix |
|---|---|
| `acp_doctor` says `claude` is not resolvable | The CLI is not on `PATH`. Either fix `PATH`, or set `acpx.agents.claude.command` to the absolute path of `claude.cmd`. |
| Transcript only shows the outbound prompt, no answer | This used to be the bug for TUI-style CLIs (claude/gemini/cursor/qwen on Windows). It is fixed in current builds via the `oneshot-prompt` transport. If you see it, your build is older than May 2026 — update. |
| Session left running | Always end with `acp_kill`. The LLM is rule-instructed to do so, but if a request times out, manually call `acp_list_sessions` and `acp_kill`. |
| API key not picked up | Order matters: per-agent `acpx.agents.<id>.env` wins over a shell-exported variable. Double-check both layers. |
## 14.5. The "Ask Execs" toggle (approve every step before it runs)
**Ask Execs** is the human-in-the-loop safety belt. With it on, Tlamatini asks your permission *before* it runs each action in a Multi-Turn chain — and a single **Deny** stops the whole run.
It is a **Multi-Turn-only modifier**: the checkbox is disabled and greyed until you tick **Multi-Turn**, because the permission prompt lives inside the Multi-Turn tool loop. Leave it unticked and Tlamatini behaves exactly like it always has.
### What you see
When **Ask Execs** is on, before each state-changing **Tool / MCP / Agent** the chain wants to run, it *pauses* and a modal dialog (same look-and-feel as every other Tlamatini dialog) shows you:
- **Tlamatini Tool / MCP / Agent** — what is about to execute, e.g. `Tool: Executer`, `Agent: SSHer`, `MCP / ACPX agent: ACPx`, `Skill: summarize`.
- **Underlying tool** — the raw tool name (`execute_command`, `chat_agent_ssher`, `acp_spawn`, …).
- **Parameters of execution** — the full argument set (read-only).
- **Program to be executed** — the command / script / intent text (read-only).
- **Shell to be executed** — `cmd.exe / PowerShell (Windows)`, `Python interpreter`, `Remote SSH shell @ <host>`, `Kali Linux (MCP-Kali-Server)`, … (read-only).
Two buttons:
- **Proceed** (green) — run this step, then continue to the next one (which prompts again).
- **Deny** (red) — **halt the entire chain immediately**. Nothing else runs.
### What a Deny gives you back
1. The prose answer up to that point.
2. The **Exec Report** tables of the steps that *did* run — only if **Exec Report** is also ticked.
3. **Always**, a big red **⛔ "Execution interrupted"** banner that names exactly the Tool/MCP/Agent you denied, plus its program/command, shell, and parameters — so you have an auditable record of where and why the run stopped.
### What is NOT prompted
Read-only / polling tools (`chat_agent_run_status`, `chat_agent_run_log`, `get_current_time`, `window_present`, …) only *observe*; they are not "executions", so they never trigger a prompt.
### Step-by-step: your first Ask-Execs run
1. Tick **Multi-Turn**, then **Ask Execs** (it only becomes clickable once Multi-Turn is on). Tick **Exec Report** too if you want the run table.
2. Send: *"Delete every `*.tmp` file under `C:/Temp`, then list what's left."*
3. A permission dialog appears for the deletion step, showing the exact command and shell.
4. Click **Deny** → the run halts with the red banner naming the denied deletion. Click **Proceed** → it deletes and carries on to the listing step (which prompts again).
This is the toggle to reach for whenever a request touches destructive or sensitive operations and you want to eyeball each action before it runs.
> **Under the hood.** The Multi-Turn executor runs in a worker thread and *blocks* on a browser round-trip via `agent/exec_permission.py` (`ExecPermissionBroker`): it emits a permission request onto the WebSocket and waits on a `threading.Event` until your Proceed/Deny reply resolves it. The round-trip is **fail-safe** — if the browser disconnects, the request is cancelled, or the emit fails, the decision defaults to *Deny* so an unconfirmed action never runs. See chapter §35 (Multi-Turn pipeline).
## 15. Combining the five toggles — worked examples
### Example A — "Set up a new ACPX agent_id from scratch"
**Tick:** Multi-Turn + ACPX + Exec Report.
> "Use the `setup-new-acpx-key` skill to register my Gemini API key (paste your key) for the `gemini` agent_id. Then run `acp_doctor` to verify it works."
The LLM picks `invoke_skill`, the skill walks itself through writing `data.keys`, patching `config.json` at both layers, and finishes with `acp_doctor` confirming `gemini` is resolvable.
### Example B — "Build a feature with three different LLMs in sequence"
**Tick:** all four (Multi-Turn + Exec Report + ACPX + internet).
> "Crawl https://docs.python.org/3/library/asyncio-task.html, summarize it. Then have claude propose a refactor of `myapp/worker.py` using those new patterns. Hand the proposal to gemini and ask it to critique. Apply the cleaned-up critique to the file with cursor. Use `acp_relay` between legs."
This is the multi-CLI relay flow. The Exec Report at the end shows every spawn, send, relay and kill — auditable.
### Example C — "Just answer this code question, please"
**Tick:** nothing. Set context to your project. Ask:
> "Where do we cap the connection-pool size in the database client?"
You get a quoted-and-cited answer in <2 seconds. No tools, no wait, no overhead.
### Example D — "Run my tests, but don't crash my desktop with popups"
**Tick:** Multi-Turn + Exec Report.
> "Run `pytest -x -k auth` in this project and summarize the output. If anything fails, show me the failing assertion."
Multi-Turn suppresses console pop-ups for wrapped runtimes; Exec Report gives you an audit table after.
## 16. From chat to flow — the Create Flow button
When a Multi-Turn run **succeeds** and used at least one state-changing tool, Tlamatini renders a **Create Flow** button in the message header. Click it to download a `.flw` JSON file that mirrors the exact tool sequence the LLM ran, laid out left-to-right, ready to load in the visual designer.
How the gating works (all four conditions must be true):
| # | Condition | Why |
|---|---|---|
| 1 | Multi-Turn was on | One-shot chats never use tools. |
| 2 | At least one tool that maps to an ACP agent succeeded | Read-only tools are excluded from flow generation. |
| 3 | Answer was classified `SUCCESS` by an LLM-based classifier | Hides the button on failures. |
| 4 | The user is logged in (not anonymous) | Sanity. |
The classifier is a tiny `chained-model` call with a strict binary prompt. It fails *open* (returns SUCCESS on internal error) so the button is never hidden by a flake.
What the generated `.flw` looks like (3-tool example):
```
Starter ──► Crawler ──► File-Creator ──► Ender
```
You can immediately re-open it in `/agentic_control_panel/` and run it as an unattended workflow — the LLM is no longer involved.
**Behind the scenes** (since commit `0bea21d`, May 2026), clicking Create Flow does NOT just dump the legacy draft. The browser POSTs the draft + `tool_calls_log` to `/agent/flow_from_tool_calls/`, which runs it through `agent/services/flow_spec.py::normalize_flow_payload()` and returns `flow_spec_to_legacy_json(spec, redact=True)` — a registry-canonical `.flw` whose agent / pool names match the backend Agent Contract registry and whose known secret fields (e.g. `tlamatini.password`) are stripped before the file ever touches your disk. If the round-trip fails (offline frozen install, backend down), the browser gracefully falls back to the un-normalized legacy draft so you still get a usable file.
---
## 17. The DB menu — backups and the next-session swap-in
Tlamatini's whole world — chat history, agent definitions, Tool/MCP toggle rows, session state, deployed-pool metadata, the per-user account that gates the chat — lives inside a single SQLite file: `db.sqlite3`. The top-of-page **DB** dropdown gives you a safe, GUI-first way to handle that file without ever stopping Tlamatini, opening Explorer, or remembering where the database actually sits in frozen vs. source mode.
The dropdown has two entries:
| Entry | What it does | When you use it |
|---|---|---|
| **Backup database** | Copies the live `db.sqlite3` to a directory you pick — keeps the live database in place. | Before risky changes (mass-deploying agents, migrating, experimenting), or as a routine snapshot. |
| **Set DB** | Stages a `db.sqlite3` file of *your* choice so Tlamatini swaps it in on the **next start-up**. The current database is moved into a timestamped archive folder. | Restoring a backup, importing a friend's database, switching between project-specific databases. |
### 17.1. Backup database — the read-only copy
`DB → Backup database` opens a dialog with one input — the **target directory** you want the copy written into. The input is **live-validated** (350 ms debounce): as you type, the page asks the server `GET /agent/check_backup_directory/?path=…` and colors the status line green / amber / red:
| State | Status line | Meaning |
|---|---|---|
| Green | `Directory exists. db.sqlite3 will be saved here.` | Ready to backup. |
| Amber | `A filename was specified — please specify the directory only.` | You typed a file path instead of a directory; Tlamatini always names the output `db.sqlite3` so it can be loaded back later. |
| Red | `Directory does not exist.` | Path is missing on disk. |
Click **Backup** and Tlamatini calls `POST /agent/backup_db/`, which resolves the live database path via `settings.DATABASES['default']['NAME']` (so source-mode and frozen-mode work identically) and writes a consistent SQLite snapshot into `<your-dir>/db.sqlite3`. Because the live database uses WAL, SQLite's online backup API includes committed pages still resident in `db.sqlite3-wal`. The result must pass `PRAGMA quick_check` before the UI reports success. **The live database stays open and unchanged** — Backup is purely additive.
### 17.2. Set DB — staging a database for the next session
`DB → Set DB` is the harder direction: it replaces the database the next time Tlamatini boots. The dialog has one input — the **full path to a `db.sqlite3` file** you want loaded — and the same live-validation behavior as Backup, but with stricter rules:
| State | Status line | Meaning |
|---|---|---|
| Green | `File exists. It will be loaded on the next start-up.` | Real `db.sqlite3` with a valid SQLite header. |
| Amber | `File found, but its name is not "db.sqlite3". Tlamatini will still stage it as db.sqlite3.` | Some users keep snapshot files named like `db_2026-05-14.sqlite3` — they still work because Tlamatini renames on stage. |
| Amber | `Specify the full path to a db.sqlite3 file, not a directory.` | You typed a directory; Set DB needs a file path. |
| Red | `The selected file does not look like a SQLite database.` | The first 16 bytes don't match the `SQLite format 3\x00` magic header. |
| Red | `File does not exist.` | Path is missing on disk. |
The SQLite header check is a cheap sanity guard — it catches the common "I picked the wrong file" mistake (a `.csv`, an `.flw`, a screenshot, an empty file) before Tlamatini commits to swapping it in.
When you click **Set** and the file passes validation, the page POSTs `POST /agent/set_db/`. The view computes the deployment-specific **staging directory** — `<exe_dir>/DB/ToLoad/` in frozen mode, `<repo>/Tlamatini/DB/ToLoad/` in source mode — creates it if needed, and writes a consistent, self-contained snapshot there as `DB/ToLoad/db.sqlite3`. If the selected database has committed pages in a neighboring WAL, those pages are included. **The live database is NOT touched.** SQLite is open in-process while Tlamatini runs; replacing it mid-flight would corrupt the live connection pool, so Set DB *only stages*.
Immediately after staging succeeds, the dialog is replaced by a second one — a yellow ⚠ warning panel — telling you in two sentences:
> **The selected database will be loaded the next time Tlamatini starts.**
> If you want it loaded immediately, you must restart Tlamatini completely so the swap-in can run BEFORE Django opens the live database.
Click **OK** and the dialog closes. There is no Cancel — by the time you see this dialog the file is already staged.
### 17.3. The start-up swap-in (what "next session" actually means)
The third leg of the DB mechanic — and the only one without a UI surface — is the start-up swap-in itself. It lives at the very top of `Tlamatini/manage.py` and runs in this exact order before *anything Django* is imported:
```
_apply_pending_db_swap() ← runs directly BEFORE Django
↓
[ os.path detection: frozen or source? ]
↓
[ DB/ToLoad/db.sqlite3 exists? ]
│
├─ NO ──► return (no-op, normal start-up continues)
│
└─ YES ──► [1] mkdir DB/Older/<YYYY-MM-DD_HHMMSS>/
[2] shutil.move(live db.sqlite3 + sidecars → Older/<timestamp>/)
[3] ensure no stale live WAL/SHM remains
[4] shutil.move(DB/ToLoad/db.sqlite3 → live db.sqlite3 path)
[5] return (Django opens the freshly-swapped file)
```
The three guarantees this gives you:
1. **Pre-Django timing.** Because the swap-in runs before the `from django.core.management import execute_from_command_line` line, Django's SQLite connection pool is never holding a stale file descriptor at the moment of the swap. A simple **Reconnect** from the navbar is NOT enough to trigger the swap-in — you must restart the entire process (close the console, launch Tlamatini again).
2. **Atomic moves, no stale WAL.** Both legs use `shutil.move` (filesystem rename when possible, copy+delete across mounts) so the source files are consumed. The outgoing SQLite sidecars are archived with their database, and none may remain beside the promoted database. A second launch with `DB/ToLoad/` empty is automatically a no-op — there's no "stuck flag" to clear.
3. **Mode-correct path resolution.** Frozen mode looks at `<exe_dir>/DB/ToLoad/db.sqlite3` (where you can browse to in Explorer); source mode looks at `<repo>/Tlamatini/DB/ToLoad/db.sqlite3` (where `manage.py` lives). The live `db.sqlite3` path is computed the same way Django does — `_MEIPASS/db.sqlite3` under PyInstaller, `<manage.py dir>/db.sqlite3` in source mode — so the swap-in writes to exactly the location Django will open.
If anything fails inside the swap-in (locked file on Windows, corrupt source, permission error), the function catches the exception, prints `--- [DB SWAP] Skipped due to error: <reason>` to `tlamatini.log`, and lets Tlamatini start normally with the previous database. **A bad ToLoad file must never block start-up** — that would lock you out of your own database.
### 17.4. The Older audit trail
Every successful swap-in leaves a complete record under `<base>/DB/Older/<YYYY-MM-DD_HHMMSS>/`. The timestamp is the local time at the moment the swap happened, filesystem-safe on Windows / Linux / macOS. If the outgoing database has WAL/SHM sidecars, they remain beside its `db.sqlite3` in the same archive:
```
DB/
├─ ToLoad/
│ └─ (empty most of the time; momentary home of the next-session pick)
└─ Older/
├─ 2026-05-14_153022/
│ ├─ db.sqlite3 ← database that was live before swap #1
│ └─ db.sqlite3-wal ← committed pages, when present
├─ 2026-05-14_164410/
│ └─ db.sqlite3 ← database that was live before swap #2
└─ 2026-05-14_172908/
└─ db.sqlite3 ← and so on
```
Because Set DB **moves** (not copies) the prior live database into Older, this archive is the only built-in recovery path. To roll back to a previous database, you do exactly the same thing you did to load a new one: copy the archived `db.sqlite3` back into `DB/ToLoad/` and restart. The swap-in archives the *current* live one under a fresh timestamp and promotes your roll-back pick.
Tlamatini **never** deletes anything from `DB/Older/`. If swaps become routine, you may want to prune the oldest folders by hand — but read each carefully first, because each `db.sqlite3` contains chat history, agent definitions, session state, and your user account, so think twice before deleting any of them.
### 17.5. The DB tree ships in both modes
The `DB/ToLoad/` and `DB/Older/` directories must exist on day one — the swap-in opens them with `os.makedirs(exist_ok=True)`, but having them pre-seeded with documentation prevents user confusion. So:
- **Source / dev mode**: `Tlamatini/Tlamatini/DB/ToLoad/README.md` and `Tlamatini/Tlamatini/DB/Older/README.md` are checked into the repo. Each is a short standalone guide describing the contract, how to use the directory, and the rollback recipe. They survive in git only because git ignores empty directories; the README is the trick that keeps the tree.
- **Frozen mode**: `build.py` extends its `empty_dirs` tuple to include `"DB/ToLoad"` and `"DB/Older"`. The PyInstaller post-build step creates both under `dist/manage/` (which becomes the install root next to `Tlamatini.exe`), and the `pkg.zip` packager preserves empty directories via explicit zip entries. End-users get the same `DB/{ToLoad,Older}/` tree from the very first launch, ready to receive a drop or archive a swap.
### 17.6. Mental model — three questions to ask before each operation
When you reach for `DB → Backup database`, ask:
> *"Do I want a copy I can come back to, while continuing to use the live database?"*
When you reach for `DB → Set DB`, ask:
> *"Am I prepared to restart Tlamatini? Do I want the current database moved out of the way and replaced?"*
When you reach into `DB/Older/` by hand, ask:
> *"Which timestamp do I trust? Am I about to overwrite the only running database with one that may be older / smaller / from a different machine?"*
The DB menu is intentionally small — three primitives (backup, stage, archive), one swap window (process restart), one audit trail (timestamps). Everything else is just discipline.
---
## 17.5. The ACPX-Skills menu — admining the skill catalog
Tlamatini ships with 27 markdown skill packages under `Tlamatini/agent/skills_pkg/`. Each one is a small playbook the LLM can run: `acp-router` picks the right external coding-agent CLI for an intent, `summarize` compresses text to a target word count, `setup-new-acpx-key` walks the user through plugging in a new API key, `skill-creator` bootstraps a brand-new SKILL.md, `flow-making` turns a plain-language objective into a canvas-loadable `.flw` by driving the FlowCreator engine (it supersedes the older `tlamatini-flow-from-objective`, which now delegates to it), `code-review` runs a senior-engineer pass over a git diff (returning an APPROVE / REQUEST_CHANGES verdict) and `security-audit` sweeps a path with the installed SAST / secret / dependency scanners, the `tlamatini_*` family audits and refactors Tlamatini's own codebase, and a handful of integration stubs (`github`, `gmail`, `notion`, `slack`, `jira`, `todoist`, `trello`, `weather`) reach out to third-party services through their REST APIs. They run inside a sandbox called the **SkillHarness**: budgets cap iterations, wall-clock, and tokens; declared filesystem / shell / network / db permissions gate side effects; declared inputs and outputs are validated on the way in and the way out.
Before May 2026, the only way to interact with this catalog was through the LLM itself. You'd type "list the skills you have" and the chat would call `list_skills` and read out the rows; you'd type "use the summarize skill to compress this" and the chat would call `invoke_skill('summarize', '{...}')` and surface the result. Catalog hygiene — knowing exactly what was installed, picking which skills the planner was allowed to surface, reloading the registry after editing a SKILL.md on disk — all of it had to be routed through the model. That worked but it felt wrong: a piece of catalog admin that belongs to the person at the keyboard kept asking permission from the model.
The **ACPX-Skills** dropdown closes that gap. It lives between **Panels** and **External** in the chat navbar (before the 2026-09-25 menu reorganization it sat between *Agents* and *Config*) and has four entries:
### `ACPX-Skills → Browse Skills`
Opens a two-pane modal: a search-filterable list on the left with one row per skill (a small green dot for enabled, a small grey dot for disabled, the skill's name, and a runtime tag — `IN-PROCESS` or `ACPX`), and a detail pane on the right that fills in when you click a row. The detail pane is read-only and pulled fresh from the registry on every click — it shows the description, the runtime (and the `acpx_agent` if `runtime: acpx`), the budget triple (`12 iter · 180 s · 30000 tokens` style), the trigger keywords the planner uses to score the skill against a prompt, the `requires_tools` and `requires_mcps` arrays (so you can immediately see if a skill depends on something you've disabled), the inputs and outputs (with required-field markers and per-field types), the on-disk path to the SKILL.md, the first sixteen characters of its body SHA-256, and finally the full markdown body of the skill in a scrollable pre block. A search box at the top of the dialog filters the list as you type — matching against name + description + keywords. The list shows `N / 27` so you always know how much was hidden by your search. There's also a small note at the right when the registry has orphan DB rows — skills the database thinks exist but whose SKILL.md was deleted on disk; Diagnostics gives you the full list.
Use Browse when (a) you've just authored a new SKILL.md and you want to confirm it parsed correctly, (b) you want to know exactly what permissions a third-party integration skill is asking for before you enable it, (c) you're debugging a failing `invoke_skill` and you want to look at the body the LLM was supposed to follow.
### `ACPX-Skills → Configure Skills`
A checkbox grid, one row per skill, that mirrors the **Configure MCPs** and **Configure Agents** dialogs you already know (both under **Config** since 2026-09-25). Each row is `[ ] skill-name — description`. Toggle a checkbox off, click **Continue**, and a couple of things happen simultaneously: the `Skill.enabled` column flips to `false` in the database, and the change broadcasts over the same WebSocket channel the existing MCPs/Tools/Agents toggles ride — the payload is encoded as `name=description=true|false,...` exactly the way `set-mcps` / `set-tools` / `set-agents` encode their payloads. The backend's new `set-skills` handler in `consumers.AgentConsumer.receive` parses the payload and calls `save_skill(name, enabled)` for each row.
After the toggle lands, two consequences arrive immediately for the next request:
- The LLM's `list_skills` tool filters disabled rows out of its returned array — disabled skills become invisible to enumeration, so the planner won't surface them.
- The LLM's `invoke_skill('<disabled-skill>', '{...}')` returns `{"ok": false, "code": "SKILL_DISABLED"}` instead of running.
Toggling the row back to enabled restores both behaviors. This is the right knob when you want to hide an unfinished skill from the planner, when you don't have the API key for an integration skill (no point letting the LLM keep trying `notion` without `NOTION_TOKEN`), or when you're running a demo and you want a minimal tool surface so the model doesn't waste turns picking obscure options.
### `ACPX-Skills → Diagnostics`
A cross-check report that catches the kind of drift you'd otherwise only discover at runtime. It has four sections; each is collapsed when clean (green ✓) and expanded with a red ⚠ count when something's wrong:
- **Missing tool dependencies** — for each skill whose `requires_tools` lists a Tool you've currently disabled in the **Tools** dialog. A disabled tool means the skill would fail at runtime; Diagnostics surfaces it before the LLM tries.
- **Missing MCP dependencies** — same idea against disabled `Mcp` rows.
- **Unknown ACPX agents** — for `runtime: acpx` skills, flags any `acpx_agent` value that isn't in the live `AcpAgent` table (typo, removed CLI, etc).
- **Orphan DB rows** — `Skill` rows whose SKILL.md file no longer exists on disk; typically a sign that someone deleted a skill directory without running Reload.
A header strip at the top gives you the counts at a glance: how many skills are on disk, how many DB rows exist, how many Tools/MCPs/ACPX agents are tracked. The endpoint is `GET /agent/skills/_/diagnostics/` — pure read, no writes.
### `ACPX-Skills → Reload Registry`
A single-click action that re-runs the registry boot pipeline: rescan `agent/skills_pkg/`, refresh every `Skill` DB row's metadata cache (description, runtime, acpx_agent, frontmatter_json, body_sha256), prune any DB row whose SKILL.md is gone. The user-toggled `enabled` field is preserved. A toast confirms the new count.
Use Reload after authoring a new SKILL.md or editing an existing one. No server restart needed.
### The line the database deliberately does not cross
The `Skill` model itself was older than this dropdown — it was added back in migration `0071_acpx_skills.py` with a richer shape than the `Tool` / `Mcp` / `Agent` toggle rows: it has `name`, `description`, `runtime`, `acpx_agent`, `enabled`, `frontmatter_json`, `body_sha256`, `last_loaded_at`. Building this admin UI on top of it required a small but deliberate decision: the dropdown **only** ever writes the `enabled` boolean. Every other column is owned by `agent/acpx/service.py::boot_skills()` and is overwritten on every reload from the SKILL.md on disk. The cached `frontmatter_json` and `body_sha256` are present because earlier work needed them for fast lookups; the admin UI ignores them and reads fresh from the registry instead, so the disk stays the only source of truth for permissions, budgets, and body.
This matters because the obvious alternative — "let the user override a skill's `max_seconds` from the browser" — is a trap. The next backup would silently archive that override, the next `git pull` would not show it in `git diff`, and a user on a different machine would have no way to know why their identical SKILL.md behaved differently. Editing the SKILL.md and clicking Reload keeps every behavioural change visible in a file and a commit. The DB stays at "enumeration plus enable/disable", exactly the way the Mcps and Agents dialogs constrain themselves.
### Where to look in the code
- `agent/views.py` — `list_skills_view`, `skill_detail_view`, `reload_skills_view`, `skills_diagnostics_view`.
- `agent/urls.py` — `/agent/skills/`, `/agent/skills/<name>/`, `/agent/skills/_/reload/`, `/agent/skills/_/diagnostics/`.
- `agent/consumers.py` — `skill_establishment`, `get_all_skills`, `save_skill`, the new `set-skills` branch, the establishment loops in both the session-restore and rebuild paths.
- `agent/acpx/tools.py` — `_disabled_skill_names()` plus the gating clauses in `list_skills` and `invoke_skill`. Fails OPEN on DB exception by design.
- `agent/templates/agent/agent_page.html` — the dropdown, the three dialog containers, the asset includes.
- `agent/static/agent/js/skills_dialog.js` — all four dialogs (Configure / Browse / Diagnostics / Reload) in one module.
- `agent/static/agent/css/skills_dialog.css` — styling.
- Coverage: 14 tests across `SkillsAdminEndpointTests`, `SkillsToolSurfaceGatingTests`, `SkillsNavbarTemplateContractTests`.
---
# Part III — The Visual Workflow Designer
## 18. Why drag-and-drop flows
The chat is amazing for one-off tasks. But some jobs you want to:
- Run on a schedule (every hour, every Monday at 9 a.m.)
- Run unattended on a remote server
- Run identically every time, with no LLM creativity in the loop
- Compose at design-time so the steps are auditable before any LLM is involved
Those are flows. You drag agents from a sidebar onto a canvas, draw lines between them, configure their parameters, save the result as a `.flw` file, and run it.
## 19. Anatomy of the canvas
Open it from the chat's **Panels ▸ Agentic Control Panel** (it lives at `/agent/agentic_control_panel/` and opens in its own tab):
```
┌─────────────────────────────────────────────────────────────────────────┐
│ ▶ Start ⏸ Pause ⏹ Stop ⚠ Hypervisor 💾 Save 📂 Load ✓ Validate │ ← Toolbar
├──────────────────┬──────────────────────────────────────────────────────┤
│ Sidebar │ │
│ ─ Control │ │
│ Starter │ │
│ Ender │ │
│ ─ Routing │ CANVAS │
│ Forker │ (draggable agents, │
│ Asker │ connections, LEDs) │
│ ─ Logic Gates │ │
│ AND OR … │ │
│ ─ Action │ │
│ Executer │ │
│ … │ │
│ … │ │
└──────────────────┴──────────────────────────────────────────────────────┘
```
A few facts about the canvas you need to internalize:
- **It scrolls.** The visible area is a viewport (`#submonitor-container`). The actual canvas (`#canvas-content`) grows to the right and bottom as you drop more agents. Use the themed scrollbars.
- **Each agent is a draggable item.** Double-click to open its config. Right-click for the context menu (description, log, explore directory, open cmd, restart).
- **Connections are typed.** A green line means "start the target after this finishes" (`target_agents`). A blue line means "monitor this source's log" (`source_agents`). The direction matters.
- **LEDs show state.** Green = running, red = down while the flow is active, yellow blinking = paused, gray = stopped/idle.
- **It has an editor toolbar (2026-09-25)**, in a strip right above the canvas: **↶ Undo / ↷ Redo** (Ctrl+Z / Ctrl+Y), **⚙ Configure** (opens the one selected agent's settings — the same thing a double-click does), **▣ Duplicate** (copies the selected agents *together with their settings* and the connections between them; FlowCreator and FlowHypervisor are one-per-flow, so they are never duplicated), **⌫ Delete**, a **Starters** list that scrolls to any Starter on a big canvas (Start still runs all of them), **Flow settings** (show or hide the dot grid, set the zoom, quick-configure Starter/FlowCreator/FlowHypervisor), and **− 100% + Fit** to zoom from 25 % to 200 % or fit the whole flow on screen. Zooming never changes the positions saved in your `.flw`.
- **You can search the agents.** The box above the sidebar filters the 85 agents by name as you type ("shot" finds Shoter), even while the list is still loading.
- **An empty canvas helps you start.** It says *Design your agent flow* and offers **Try an example**, which lays down a Starter → Sleeper → Ender flow you can undo in one step. **Help** in the top bar lists every gesture.
- **Editing locks while a flow runs or is paused.** Stop the flow to change it. A leading **•** in the browser tab title means you have unsaved changes.
Another recent change matters if you use the config dialogs heavily: dialog-edited wiring fields now survive the compile pass. In practice that means a user-edited `source_agents`, `target_agents`, or Ender kill list is preserved, while the canvas still contributes its live connections where appropriate. Validate and Start no longer flatten those deliberate edits back into stale pool defaults.
## 20. Your first flow (3-agent example)
Goal: run a shell command, take a screenshot, end.
**Step-by-step:**
1. Drag **Starter** onto the canvas (top-left).
2. Drag **Executer** to its right.
3. Drag **Shoter** further right.
4. Drag **Ender** at the far right.
5. Connect: Starter → Executer → Shoter → Ender (click the right edge of one, drag to the left edge of the next).
6. Double-click **Executer** and set its `command` to `dir C:\` (or `ls /tmp` on Linux).
7. Double-click **Shoter** and set `output_dir` to a folder you can write to.
8. Double-click **Ender** and confirm its `target_agents` lists every other agent (Starter, Executer, Shoter) — Ender's job is to terminate them on completion.
9. Click **✓ Validate** — Tlamatini runs structural checks (no orphans, no self-connections, terminal agents reachable).
10. Click **▶ Start**.
You'll see LEDs go green, then sequential outputs in the log viewer, then everything turns gray. Open your `output_dir` — there is a screenshot.
## 21. Saving and loading `.flw` files
Use **File ▸ Save as**, pick a name. You get a JSON file with all node positions, configs, and connections. Distribute it; somebody else loads it via **File ▸ Open**, gets the same flow. Positions are always saved at 100 % zoom, so a flow you arranged while zoomed out reopens exactly where you put it. Loading a flow clears the undo history — the loaded diagram is the new starting point. A `.flw` file dropped on the canvas — or on any other Tlamatini page — is recognised by its content and loaded here (or opened in a new Agentic Control Panel tab) instead of being opened as plain text.
`.flw` files are also what the chat's **Create Flow** button (chapter 16) emits.
## 22. Pause, Resume, and Stop
The three buttons each do something different:
| Button | What happens |
|---|---|
| **⏸ Pause** | Stores the currently-running agents into `paused_agents.reanim`, kills their processes, **leaves logs and `reanim*` state files intact**. The ACP enters paused state (yellow LEDs). |
| **▶ Resume** (after pause) | Reanimates each saved agent with `AGENT_REANIMATED=1`. Each agent reads its `reanim*` files and continues from exactly where it stopped. |
| **⏹ Stop** | Hard stop. Ender runs its termination logic; reanimation files are cleared. |
This is why long-running workflows (Crawler scraping 10,000 URLs, Parametrizer iterating through segments) survive pauses without data loss.
The stop path also got harder to break in mixed flows. Current builds are better at killing lingering session processes during cleanup, so a half-manual / half-compiled run is less likely to leave zombie agents behind before the next start.
## 23. FlowHypervisor (your watchdog)
Click **⚠ Hypervisor** and a system-managed FlowHypervisor agent starts watching every other running agent. It is an LLM that:
- Reads each agent's log incrementally.
- Builds an execution matrix from current output-slot/lifecycle contracts, including Counter branches and Parametrizer targets; kill lists and observation dependencies remain separate.
- Looks for: stuck agents (started >5 min ago, no output, downstream never fired); broken chains (agent finished, but its `target_agents` never started); fatal/critical errors; user-imposed time constraints.
- Outputs exactly **`OK`** when healthy or **`ATTENTION NEEDED { explanation }`** when not.
If a problem fires, the browser shows an alert dialog. You can append your own rules to the watchdog through the FlowHypervisor agent's `user_instructions` config field — useful for "don't flag this known false-positive" or "wake me if X is silent for >10 min."
## 24. FlowCreator — let an LLM design the flow for you
Drag a **FlowCreator** node onto the canvas, double-click it, and type a natural-language objective:
> "Every hour, crawl our status page; if it shows ERROR, email the on-call engineer; otherwise, do nothing."
Click **Generate**. FlowCreator selects capabilities from all 85 installed agents, then designs with the relevant `agentic_skill.md` sections and current schemas/contracts. It validates references, connection slots and Parametrizer mappings, with bounded repair, before writing a `.flw` and publishing the canvas result. You can review, edit parameters, and run. GUI-Manager remains design only.
This is the highest-leverage feature for non-technical users: you describe what you want, the system *draws* the flow.
**You can also do this straight from chat — no canvas required.** With Multi-Turn on, just ask: *"create a flow that watches my GlassFish log and Telegrams me the summary when it sees an error, call it glassfish_alert.flw"*. The wrapped **`chat_agent_flowcreator`** tool drives the same FlowCreator engine and writes the same canvas-loadable **`.flw` file** to disk (into Tlamatini's Temp folder by default, or wherever you pass `output_dir`); Tlamatini gives you back the file path to open in the Control Panel. It writes the file — it does not run the flow, so you stay in control of when it goes live.
## 25. Parametrizer (chaining outputs into the next agent's config)
This is the agent that makes multi-stage pipelines work without manual `config.yaml` editing.
### The problem it solves
Tlamatini agents communicate through **log files** and **`config.yaml` files** — they have no shared memory. So if you want Apirer's response to become Kyber-Cipher's input, somebody has to copy data between agents.
Parametrizer is that somebody. It reads structured output segments from one source agent's log, injects mapped values into one target agent's `config.yaml`, runs that target, waits for it to finish, restores the target config, advances its source cursor, and only then moves to the next source segment.
### The unified section format
Every "Parametrizer-friendly" agent emits its results in **one** format:
```
INI_SECTION_<AGENT_TYPE><<<
key1: value1
key2: value2
multi-line body content (becomes 'response_body')
>>>END_SECTION_<AGENT_TYPE>
```
Rules:
- `<AGENT_TYPE>` is the agent's UPPERCASE name (`APIRER`, `CRAWLER`, `KYBER_KEYGEN`, …).
- The first blank line separates the KV header from the body.
- The whole section must be emitted in a **single `logging.info()` call** — concurrent log writes from other threads could otherwise interleave and corrupt the block.
- One section per output unit. If the agent produces N results, emit N sections.
### Supported source agents
46 agents emit Parametrizer-compatible sections:
Apirer, Gitter, Kuberneter, Crawler, Summarizer, File-Interpreter, Image-Interpreter, File-Extractor, Prompter, FlowCreator, Kyber-KeyGen, Kyber-Cipher, Kyber-DeCipher, Googler, **Playwrighter**, **ACPXer**, Shoter, **Camcorder**, **Recorder**, **AudioPlayer**, **VideoPlayer**, Mouser, **Windower**, **Unrealer**, **Blenderer**, **Reviewer**, **Analyzer**, **Kalier**, **STM32er**, **ESP32er**, **ESPHomer**, **Arduiner**, **De-Compresser**, **Talker**, **Whisperer**, **Editor**, **Grepper**, **Globber**, **Telegrammer**, **Whatsapper**, **Instant Messaging Doctor**, **MCP Doctor**, **Discoverer**, **Video-Analyzer**.
### How the visual mapping works
On the canvas, double-click a Parametrizer node to open its custom mapping dialog (not the standard config editor):
1. Left column shows the source agent's available output fields (cyan).
2. Right column shows the target agent's `config.yaml` parameters, flattened to dot-notation if nested (orange).
3. Click a source field, then a target parameter, to draw a curved Bezier line connecting them.
4. Click any line to remove it.
5. Save. Tlamatini writes the mapping to `interconnection-scheme.csv` in the deployed Parametrizer pool directory.
### Iterative execution
If the source produces 5 segments (e.g. Apirer hits 5 endpoints), Parametrizer processes them **one at a time**, in order, with a full `config.yaml.bck` backup-and-restore around each one. The target agent gets a clean config every time, and its log is archived as `<target>_segment_1.log`, `<target>_segment_2.log`, … so each segment outcome stays inspectable.
### A canonical example
```
Apirer ──▶ Parametrizer ──▶ Kyber-Cipher
```
- Apirer calls 3 endpoints, emits 3 `INI_SECTION_APIRER<<<` blocks.
- Parametrizer maps `response_body → buffer` (the value to encrypt).
- Kyber-Cipher runs 3 times, once per response body, encrypting each.
No manual config editing. No race conditions. Pause-safe.
# Part IV — The Tlamatini Bestiary
A compact reference for all 85 workflow-agent types. The **Parametrizer** chapter (§25) explains output mapping; **Unrealer** gets a full bonus chapter at §57, **Blenderer** at §59, and **ESPHomer** at §60.
> **Naming reminder.** The `agentDescription` (set by each migration) is the single source of truth. CSS classmap key, sidebar visual, and connection-handler name all derive from it.
> **Description-tooltip source.** The hover tooltip and right-click "Description" dialog both pull their text from `agents_descriptions.md` at the repo root (the Django view parses its `## Workflow Agents` tables and injects them as `agent_purpose_map`). `README.md` is kept as a legacy fallback only. Editing a row in `agents_descriptions.md` changes both human docs AND the live UI text.
## Control
| Agent | Purpose |
|---|---|
| **Starter** | Entry point. Launches first agents. |
| **Ender** | Terminates all agents in `target_agents`, then launches `output_agents` (typically Cleaners, FlowBackers). |
| **Stopper** | Single-threaded pattern-matcher that kills agents when patterns appear in source logs. |
| **Cleaner** | Post-Ender cleanup of logs / PIDs. |
| **Sleeper** | Wait `duration_ms` and trigger downstream. |
| **Croner** | Scheduled trigger (`HH:MM`). |
## Routing
| Agent | Purpose |
|---|---|
| **Raiser** | Watches a source log for a pattern; starts downstream when found. The "event-driven launcher". |
| **Forker** | Auto-routes to Path A or Path B based on two pattern sets. |
| **Asker** | Interactive A/B chooser; pauses the flow until the user picks in the browser. 5-min timeout. |
| **Counter** | Persistent counter; routes to Path L (`<` threshold) or Path G (`>=` threshold). |
## Logic gates
| Agent | Purpose |
|---|---|
| **OR** | Fires when EITHER of two sources completes. |
| **AND** | Fires when BOTH of two sources complete. |
| **Barrier** | Generalized AND: fires when ALL N sources complete. |
## Action
| Agent | Purpose |
|---|---|
| **Executer** | Shell command. |
| **Pythonxer** | Inline Python behind a strict `compile()` + blocking-Ruff gate (`ruff_blocking` default true); ALWAYS triggers downstream regardless of outcome (exit code drives only the LED + Multi-Turn retry loop). |
| **Prompter** | LLM prompt → log. |
| **Summarizer** | LLM polls source logs for events; one-shot mode also accepts `input_text`. |
| **Crawler** | Reads a web page (by default exactly the one given; range modes also walk its links) with raw-content or text capture and LLM analysis. Block- and hang-proof: every page and the whole run are time-bounded, and a bot wall, 401/403/429 or timeout is reported as `status: blocked` / `timeout`, never as page content. |
| **Googler** | Block- and hang-proof indexed-web search with readable-text/raw extraction or URL-only delivery. One wall-clock deadline (`deadline_seconds`, default 120) bounds the whole run. Tier 0 asks six server-rendered routes over plain HTTP in parallel (DuckDuckGo HTML/Lite, Bing, Mojeek, Brave, Yahoo); Tier 1 opens real installed Chrome with a persistent profile (`window_mode` visible/offscreen/headless, bundled Chromium fallback) across eight routes; Tier 2 answers from open knowledge sources (Wikipedia, arXiv, OpenAlex, Hacker News, GitHub, Internet Archive, Project Gutenberg) and says so. A relevance guard rejects poisoned answers, a refusal classifier names CAPTCHA/anomaly/rate-limit walls, a shared health ledger cools refusing engines instead of hammering them, and `search_status` reports ok, no_matches, blocked, unreachable or timeout truthfully; the log names the route that answered. Its visual/pool configuration includes a structured dork builder: presets for books, lawful public/open books, papers, manuals, docs, slides, sheets, and directories; aliases such as `ebook`, `docs`, `code`, and `data`; exact/exclusion/date/range/proximity and title/URL/text/anchor/discovery operators; and mechanically correct parenthesized uppercase-`OR` groups for multiple sites or file types. Explicit fields override preset defaults. Advanced date/proximity/range semantics are Google-specific and may broaden on fallback engines; pin `engines: [google]` for exact full-vocabulary behavior and to skip Tier 0. Use `content_mode: links_only` for PDF/EPUB/file hunts, then feed the URLs to Apirer/File-Extractor/File-Interpreter. The direct Multi-Turn tool accepts equivalent operators in `query`; structured fields are canvas/pool-only. It finds public indexed URLs, does not bypass access controls, and leaves copyright/licensing/authorization decisions to the user. |
| **Playwrighter** | Scripted interactive browser automation via Playwright (Chromium/Firefox/WebKit). Drives a REAL browser through an ordered list of declarative steps — `goto` / `click` / `fill` / `press` / `wait_for` / `extract_text` / `extract_attr` / `screenshot` / `assert_visible` / `assert_text` / `download` — for authenticated, JS-rendered, multi-step flows (logins, forms, wizards, SPA scraping, end-to-end UI checks) that Crawler (static fetch) and Googler (search) cannot do. Deterministic (no LLM). Set `headless: false` to watch it drive and `hold_open_seconds: N` (alias `hold_open_ms`) to keep the browser visible N seconds after the last step before it closes. Emits an `INI_SECTION_PLAYWRIGHTER<<<` block (`start_url`, `final_url`, `status`, `steps_run`, `assert_result`, `response_body`) and always triggers `target_agents`. Canvas counterpart of the `chat_agent_playwrighter` Multi-Turn tool. |
| **Apirer** | HTTP REST request with structured logging. |
| **Gitter** | Git operations on a local repo. |
| **SSHer / SCPer** | SSH command execution / SCP file transfer. |
| **Dockerer / Kuberneter** | Docker / Kubernetes commands. |
| **PSer** | LLM-powered semantic process finder. |
| **Jenkinser** | Jenkins pipeline trigger with CSRF crumb support. |
| **SQLer / Mongoxer** | SQL Server / MongoDB scripting (external windows). |
| **Mover / Deleter** | File move/copy / deletion (glob, recursive, `filetype_exclusions`). |
| **Shoter** | Screenshot capture (read-only). |
| **Camcorder** | Physical-camera (webcam) capture via OpenCV (`cv2`) — Shoter's hardware-camera sibling (Shoter = screen, Camcorder = camera). `capture_mode` ∈ `photo` (default, one `.jpg`) / `video` (a `.mp4` segment of `video_duration_seconds`, no audio); `camera_index` selects the device; `resolution_width`/`resolution_height` default `0×0` = camera-native (set `W×H` to request a mode — requested + read-back logged); `warmup_seconds` lets the sensor settle. Saves to `Pictures/TlamatiniCamcorder` with a collision-proof timestamped filename (override via `output_dir`). Observational (read-only, but STILL in the Exec Report); emits `INI_SECTION_CAMCORDER<<<` and always triggers `target_agents`. Needs `opencv-python`. Canvas counterpart of `chat_agent_camcorder`. |
| **Recorder** | Microphone / audio-input capture via `sounddevice`, saved as a WAV (stdlib `wave`) — the audio sibling of the capture trio (Shoter = screen, Camcorder = camera, Recorder = sound). Records from the system DEFAULT input device for `record_seconds` (pick another mic with `device_index` — the agent logs the numbered device list at startup — or by case-insensitive name substring with `device_name`); `sample_rate` defaults `0` = the device's NATIVE rate (forcing an unsupported rate raises a PortAudio error, so the safe default lets the device choose — the value used is read back + logged); `channels` defaults mono (`1`, clamped down to the device max); `input_gain_percent` is POST-capture digital gain (`100` = unity; `200`/`50`/`0` = louder/quieter/silence — amplifying may CLIP, so `clipped_samples` is reported). Saves to `Music/TlamatiniRecords` with a collision-proof timestamped filename (override via `output_dir`). Observational (read-only, but STILL in the Exec Report); emits `INI_SECTION_RECORDER<<<` and always triggers `target_agents`. Needs `sounddevice`. Canvas counterpart of `chat_agent_recorder`. |
| **AudioPlayer** | Audio-file PLAYBACK to a system OUTPUT device (speakers) via `soundfile` (decode) + `sounddevice` (stream) — the playback counterpart of Recorder (mic-IN → speakers-OUT). `audio_file` (required) is the path (WAV/FLAC/OGG/AIFF, MP3 with a recent libsndfile); plays to the system DEFAULT output by default (`device_index`/`device_name` to pick another). `volume_percent` is a software gain (`100` = unity; clip count reported). **`time_played`**: `0` = the whole file once; `N>0` = exactly N s — a longer file is TRUNCATED, a shorter one is LOOPED (whole repeats + a final partial segment) via a streaming wrap-around callback (no giant buffer). `sample_rate` defaults `0` = the file's own native rate (correct pitch; read from the file). Does NOT change the OS default output device. Observational/output (STILL in the Exec Report); emits `INI_SECTION_AUDIOPLAYER<<<` and always triggers `target_agents`. Needs `sounddevice` + `soundfile`. Canvas counterpart of `chat_agent_audioplayer`. |
| **VideoPlayer** | Video-file PLAYBACK (WITH audio) on a chosen DISPLAY via `ffpyplayer` (decode + synced audio + volume; its pip wheel BUNDLES ffmpeg + SDL — no external ffmpeg, no runtime download — collected into the frozen build by `build.py --collect-all ffpyplayer`) + OpenCV (`cv2`) for the window; degrades to SILENT cv2 video if ffpyplayer is absent. `video_file` (required) is the path (.mp4/.mov/.mkv/.avi/.webm). `display_index` picks the monitor (`-1` = primary; enumerated via Win32 `EnumDisplayMonitors`, logged at startup). `volume_percent` = audio level (capped at 100). **`time_played`** TRUNCATES a longer video or LOOPS a shorter one (whole repeats + final partial). `window_width`/`window_height` size the window (`0` = native, centered on the chosen display); `fullscreen` fills the monitor; `keep_aspect` letterboxes (cv2 `WINDOW_KEEPRATIO`) instead of stretching. Observational/output (STILL in the Exec Report); emits `INI_SECTION_VIDEOPLAYER<<<` and always triggers `target_agents`. Needs `ffpyplayer` + `opencv-python`. Canvas counterpart of `chat_agent_videoplayer`. |
| **Talker** | TEXT-TO-SPEECH (TTS): speaks `input_text` aloud through the speakers by driving an OLLAMA connection running a neural TTS model (selected by `talker_model` in Config > Models > Speech; initial default `legraphista/Orpheus:3b-ft-q8`) — streams the model's audio tokens, decodes them to a 24 kHz WAV with the **SNAC** codec, saves the file, and plays it. The voice-synthesis sibling of the media family (AudioPlayer plays an existing file; Talker GENERATES speech from text). **FEMALE-VOICE-ONLY by design** (Tlamatini is female; a male voice is FORBIDDEN — asking for one makes Talker close its execution with "male voice is forbidden by design — NOW CLOSING.. BYE", never substituting): permitted voices `tara` (default) / `leah` / `jess` / `mia` / `zoe`, and `gender` accepts only `female`. `emotion` weaves a paralinguistic tag (`<laugh>` / `<sigh>` / 8 total) into the speech; `language` is a hint; generation knobs `temperature` / `top_p` / `top_k` / `min_p` / `repetition_penalty` / `max_tokens` / `seed`; playback `device_index` / `volume_percent` / `sample_rate`. Observational/output (STILL in the Exec Report); emits `INI_SECTION_TALKER<<<` and always triggers `target_agents`. Rendering audible audio needs `snac` + `torch` (CPU is fine); without them it degrades to `status: tokens_only` (saves tokens, no sound — not a crash). Canvas counterpart of the `chat_agent_talker` Multi-Turn tool. |
| **Whisperer** | **Direct chat microphone:** Mic beside Send starts host capture without an LLM decision and stops on the configured silence gate. Config → Mic (last entry) selects automatic submission with the current draft/modes or an editable transcript for manual Send, and configures device, gain, gate and recognition preferences per browser. Each recording snapshots its settings. It has no extra console, workflow artifact output or wrapped-tool Exec Report row. **Standalone/workflow surface:** SPEECH-TO-TEXT (STT / voice recognition), the sibling of Talker — turns spoken audio into a text string. 100% self-sufficient for the microphone: it opens, configures (channels / sample-rate / gain) and records the mic ITSELF (no Recorder dependency; `record_seconds: 0` uses the sound gate by default), or transcribes a given audio FILE (`input_source` ∈ mic / file / auto, `audio_file`). Transcription engine: **faster-whisper LOCALLY by default** — it auto-detects an NVIDIA GPU via CTranslate2 and ALWAYS falls back to CPU (int8) on a machine without one (and auto-retries on CPU if the GPU path fails); `model` ∈ tiny / base / small / medium / large-v3 / large-v3-turbo (default `base`). Cloud engines `cloud-groq` / `cloud-openai` are also supported. **NOTE — Ollama CANNOT do speech-to-text** (no audio input): recognition is always done by the ASR engine; an optional Ollama pass only tidies the FINISHED transcript's punctuation. Observational (but STILL in the Exec Report); emits `INI_SECTION_WHISPERER<<<` (body = the transcript text) and always triggers `target_agents`. Needs `faster-whisper` for local transcription (absent + no cloud key → `status: engine_unavailable`, not a crash). Canvas counterpart of the `chat_agent_whisperer` Multi-Turn tool. |
| **Mouser** | Pointer movement, click, drag, scroll, click-at-window, locate-image. |
| **Keyboarder** | Keyboard typing / hotkey chords (PyAutoGUI). |
| **Terminater** | Visible, user-supervised terminals (CMD / PowerShell / PowerShell 7) in the Terminals area. Types every key like a person (`typing_wpm` 50 by default, 20-250; no paste, no instant mode), waits for the command, reads the output back in full, detects a hung command (`hang_seconds` 45), hands the keyboard to the user at `<<USER:hint>>`, and lets the user take over with Esc. One `action` per run (`run`, `open`, `type`, `send_key`, `wait`, `read`, `abort`, `kill`, `force_kill`, `layout`, `watch_user`, `warn_user`, `handover`, …); terminals persist in the Tlamatini hub between runs. Emits `INI_SECTION_TERMINATER` (`status`, `exit_code`, `typed`, `user_part`, …) and always starts its `target_agents`. Windows 10 1809+ (ConPTY). Agent #86; see [docs/terminater.md](docs/terminater.md). |
| **Windower** | Deterministic Win32 window manager (pywin32 + ctypes). Locates an application window by title (`match_mode` ∈ substring/exact/regex, plus `match_index` to disambiguate same-titled windows) and runs ONE window-lifecycle operation: `focus`, `minimize`, `maximize`, `restore`, `move`, `resize`, `move_resize`, `close`, `topmost` / `untopmost` (always-on-top), or `arrange` (snap/tile to left/right/top/bottom halves, four quadrants, center, or full) — or `list` every open window with its position, size, and state. The window member of the desktop-UI trio (Windower = the window, Mouser = clicks, Keyboarder = typing). Ports the window-management subset of Microsoft's Windows-MCP (incl. the AttachThreadInput cross-process focus-transfer dance). Emits an `INI_SECTION_WINDOWER<<<` block (`action`, `window_title`, `matched`, `match_count`, `state`, `left`, `top`, `width`, `height`, `response_body`) and always triggers `target_agents`. Canvas counterpart of the `chat_agent_windower` Multi-Turn tool. |
| **Kalier** | Kali Linux offensive-security bridge to the **MCP-Kali-Server** (`server.py` Flask API; default `http://127.0.0.1:5000`; stdlib `urllib`, self-contained — no `requests`/`mcp` deps in the pool). `action` ∈ `command` / `nmap` / `gobuster` / `dirb` / `nikto` / `sqlmap` / `metasploit` / `hydra` / `john` / `wpscan` / `enum4linux` / `health`. Emits an `INI_SECTION_KALIER<<<` block (`action`, `endpoint`, `method`, `subject`, `return_code`, `success`, `timed_out`, `server_url`, `response_body`) and always triggers `target_agents` so a Forker can branch on `{success}`/`{return_code}`. Canvas counterpart of the `chat_agent_kalier` Multi-Turn tool. **Authorized targets only.** |
| **Nmapper** | LOCAL, **use-only** nmap bridge for pentesters and CTF recon — the instant, zero-install sibling of Kalier (a REMOTE Kali box) and Discoverer (the ProjectDiscovery suite). It RUNS an nmap the user installed and **never bundles or redistributes nmap** (the NPSL forbids embedding it), resolving PATH → `C:\Program Files (x86)\Nmap` / `C:\Program Files\Nmap` → a `%LOCALAPPDATA%\Tlamatini\nmap` copy; with none found it REFUSES gracefully with install guidance, and `action: install` downloads + launches the OFFICIAL FREE nmap installer (which also brings Npcap). The DEFAULT is an **unprivileged TCP connect scan** (`-sT`) needing no Npcap and no admin, so a fresh install scans immediately; on Windows without Npcap a SYN scan auto-downgrades to `-sT`, `-O` is dropped (both warned) and a raw `udp` scan is refused — never a crash. `action` ∈ quick (`-sT -sV -sC -Pn -T4 --top-ports 1000`, the CTF opener) / full / top_ports / version / scripts (NSE) / host_discovery / udp / custom (shell metacharacters rejected) / validate / install. Runs `-oX` (parsed for `hosts_up` / `open_ports`) + `-oN` (the human report). Emits `INI_SECTION_NMAPPER<<<` (`action`, `target`, `scan_technique`, `ports`, `return_code`, `success`, `hosts_up`, `open_ports`, `npcap_present`, `xml_path`, `output_path`, `stage`) and always triggers `target_agents`. Canvas counterpart of the `chat_agent_nmapper` Multi-Turn tool. **Authorized targets only** — treat scan output as untrusted data. |
| **File-Creator** | Write a file. |
| **File-Interpreter / File-Extractor** | Document parsing (DOCX, PPTX, XLSX, PDF, …); raw text extraction with strings-fallback for unknowns. |
| **Image-Interpreter** | LLM vision analysis of images. |
| **J-Decompiler** | JAR/WAR/CLASS decompilation via bundled `jd-cli`. |
| **De-Compresser** | Deterministic archive worker (compress OR decompress). Inferred direction: `input` ext or `output` ext picks the operation. Supports `.gz`, `.zip`, `.7z`, `.tar.gz`/`.gz.tar`. Password from `DE_COMPRESSER_PWD` env var when `passwordless=false`. |
| **Telegrammer** | Single-run Telegram agent on **official Telegram surfaces only**. Bot mode uses the official Bot API (`telegram.bot_token` from @BotFather); optional user-session mode uses Telegram API credentials (`telegram.api_id`, `telegram.api_hash`, `telegram.session_name` / `session_string`) when private `@username` sends need a logged-in Telegram account. `mode` ∈ `auto` / `send` / `receive`: in `send` it posts `message` (or resolves a `contact_name`) to a human-readable `telegram.chat_id` / `@username`, then starts `target_agents`; in `receive` it waits up to `rx_max_seconds` for an inbound Bot API update (optionally filtered by `rx_from_chat_id` / `rx_match`), then starts `target_agents`. Contacts should keep people as `telegram: "@username"`; any Bot API numeric routing id belongs only in Telegrammer's private local username cache. Parametrizer SOURCE — emits `INI_SECTION_TELEGRAMMER<<<` (`mode`, `direction`, `chat_id`, `status`, `message_id`, body = `response_body`) with the visible recipient preserved. |
| **Whatsapper** | Single-run WhatsApp send/receive agent with an explicit identity switch. `provider=cloud` (default) uses the **official Meta WhatsApp Cloud API** from the business number: `mode` ∈ `auto` / `send` / `receive`, templates and the 24-hour free-text window apply, and receive runs the verified Graph-API webhook. `provider=web` / `me` sends from the operator's own personal number through a persistent Playwright WhatsApp Web profile after a one-time headed QR login; it is UNOFFICIAL, ignores templates, and carries account-ban risk. Contacts can resolve through `contact_name`; providers are never switched silently. Either route starts `target_agents`, exits, and emits `INI_SECTION_WHATSAPPER<<<` (`mode`, `direction`, `recipient`, `status`, `message_id`, body = `response_body`). No Twilio/TextMeBot route exists. |
| **Instant Messaging Doctor** | Diagnostic/repair companion for **Telegrammer** and **Whatsapper**. It validates official Telegram token/user-session readiness, contact-book resolution, readable Telegram `@username` reachability, official Meta WhatsApp Cloud API phone-number/token/template/webhook readiness, WhatsApp 24-hour/template policy blockers, and failure-log excerpts. It can run directly as `chat_agent_instant_messaging_doctor` or automatically after Telegrammer/Whatsapper failures. Parametrizer SOURCE — emits `INI_SECTION_INSTANT_MESSAGING_DOCTOR<<<` (`platform`, `status`, `telegram_status`, `whatsapp_status`, `contact_status`, `repair_status`, `retry_status`, `actions_required`, body = `response_body`) so Forker can branch on ready vs operator-required repairs. It never substitutes Twilio/TextMeBot or silently switches an explicitly selected WhatsApp Web session. |
| **TeleTlamatini** | Long-running Telegram bridge that exposes the full Multi-Turn + Exec Report Tlamatini chat to authorized Telegram users. |
| **ACPXer** | Visual canvas counterpart of the 12 LLM-facing ACPX tools. One node = one external-CLI session lifecycle. |
| **Unrealer** | Drives Unreal Engine 5 via the Unreal MCP plugin's TCP socket protocol (`127.0.0.1:55557` by default — plugin must already be running inside an UE5 editor instance). One node sends one JSON command (`{"type": <verb>, "params": {...}}`) and captures the engine's response into an `INI_SECTION_UNREALER<<<` block. Up to a 53-command surface across nine categories — editor / blueprint / node / project / umg plus system (in-editor `execute_python` + console), level, asset, and material. (See bonus chapter §57.) |
| **Blenderer** | Drives a running Blender through the OFFICIAL Blender MCP add-on's TCP socket (`localhost:9876`). Unlike Unreal's verb-dispatch protocol the Blender wire format is a **code-execution** protocol — each run sends `{"type":"execute","code":<python>}` (NUL-framed) and Blender runs that Python — so instead of forcing you to hand-write `bpy`, Blenderer exposes a rich `command` catalog: passthrough `execute_code`; read-only `ping` / `scene_info` / `get_objects` / `get_object_detail` / `blendfile_summary`; and mutating/output `create_object` (cube/sphere/cylinder/cone/plane/monkey/torus) / `delete_object` / `set_material` / `screenshot` / `render` (file outputs default under `<app>/Temp`). Self-contained: Tlamatini speaks the add-on socket DIRECTLY with a stdlib NUL-byte socket, skipping blender.org's `blmcp` bridge and its bare chat client, so the whole canvas / Multi-Turn / Exec-Report / Parametrizer stack sits on top. Requires Blender running with the add-on enabled, "Online access" on and the server started — the agent does not launch Blender. Captures the reply into an `INI_SECTION_BLENDERER<<<` block and always triggers `target_agents` on success or failure. Canvas counterpart of the `chat_agent_blenderer` Multi-Turn tool. |
| **Reviewer** | LLM-powered code reviewer. Resolves a `git diff` for `repo_path` (`diff_ref` like `HEAD~1` / `origin/main`, or empty = uncommitted working-tree + staged changes), sends it to an Ollama model with a senior-engineer prompt, and emits an `INI_SECTION_REVIEWER<<<` block whose first field is a `verdict` (`APPROVE` / `REQUEST_CHANGES` / `COMMENT`). Always triggers `target_agents`, so a downstream Forker can branch on `{verdict}`. Canvas counterpart of the `code-review` skill. |
| **Analyzer** | Deterministic static-analysis / security scanner (no LLM). Runs whichever of `bandit` / `semgrep` / `ruff` / `eslint` / `gitleaks` / `pip-audit` are on PATH over `target_path`, aggregates findings, and emits an `INI_SECTION_ANALYZER<<<` block whose `status` is `clean` / `findings` / `error` and whose `total_findings` is routable. Always triggers `target_agents`, so a downstream Forker can gate on `{status}` / `{total_findings}`. Canvas counterpart of the `security-audit` skill. |
| **STM32er** | STM32 firmware bridge to the **STM32 Template Project MCP** (`https://github.com/XAIHT/STM32TemplateProjectMCP`), driven through a self-contained inline MCP stdio JSON-RPC client (no `mcp` dependency in the pool). It scaffolds, builds, flashes, and observes STM32F407VG firmware. `action` ∈ the **23 MCP tools** + 2 composites (`serial_session`, `live_monitor`) + 2 meta (`bootstrap`, `validate`). **Zero-config auto-bootstrap**: with no on-disk `server_script` (the default is now empty), STM32er DOWNLOADS the MCP itself — a shallow `git clone`, falling back to the GitHub zip when git is absent — into `%LOCALAPPDATA%/Tlamatini/STM32TemplateProjectMCP`, pip-installs `mcp` + `pyserial` if missing, and validates the install, so the user only ever installs **STM32CubeIDE + Tlamatini** (`action: bootstrap`; config keys `auto_bootstrap` / `mcp_repo_url` / `mcp_ref` / `mcp_install_dir` / `auto_update` / `pip_install`). **Safety preflight** (critical-mission fail-safe): it validates the compiler / CubeIDE / make / programmer / ST-LINK driver + probe / device family before any compile or flash and REFUSES rather than mis-build or mis-flash — a compile needs no board, while flash / erase / reset / serial / SWD / `live_*` require a connected ST-LINK, and a cross-STM32F-family device is refused (`action: validate`; config keys `preflight`, `device`). The MCP template is still STM32F407VG-specific, so STM32er safely refuses other families (a multi-family fork is future work). Emits an `INI_SECTION_STM32ER<<<` block and always triggers `target_agents`. Canvas counterpart of the `chat_agent_stm32er` Multi-Turn tool. The first of the microcontroller-firmware trio (STM32er drives an MCP **server**; ESP32er and Arduiner drive a CLI **directly**). **Serial caveat:** on the STM32F4-Discovery family (incl. the STM32F407G-DISC1) the on-board ST-LINK does **not** bridge its USB Virtual COM Port to the MCU's USART pins (unlike *Nucleo* boards), so a `serial_session` read of the VCP returns nothing even while the firmware runs — to capture USART2 output (PA2 = TX / PA3 = RX) you must wire an external USB-TTL adapter; the SWD `live_monitor` proof needs no wiring. (See the demo nuance in the changelog.) |
| **ESP32er** | ESP32 firmware bridge built on **PlatformIO Core** (`https://platformio.org`). Unlike STM32er — which drives a separate MCP server because STM32CubeIDE has no unified CLI — PlatformIO already ships a complete `pio` CLI, so ESP32er invokes `pio` subcommands **directly** (the Kalier / Executer pattern; **no MCP server**) over a stdlib-only agent (`subprocess` + `urllib`, no `pio` dependency in the pool). `action` selects ONE capability per run across environment/meta (`bootstrap`, `validate`, `system_info`, `boards`), project lifecycle (`create_project`, `write_source`, `read_source`, `list_sources`, `clean`), build & flash (`build`, `upload`, `build_and_upload`, `list_artifacts`, and the one-call composite **`scaffold_build_upload`** = create→write→build→upload→optional-monitor in a single run — fail-safe: it skips only the upload leg with a "built OK" result when no board is connected), serial HIL (`device_list`, `monitor`, `monitor_session` = upload→monitor), and packages/QA (`pkg_install`, `pkg_list`, `pkg_update`, `check`, `test`). **Zero-config auto-bootstrap**: with no on-disk `pio_executable` and `auto_bootstrap: true`, ESP32er downloads PlatformIO Core itself (the official `get-platformio.py` installer, with a `pip install platformio` fallback) into `%LOCALAPPDATA%/Tlamatini/platformio`, so the user installs **only the board USB driver + Tlamatini**. **Safety preflight** (fail-safe): before any build/upload it validates `pio` is resolvable + a `platformio.ini` exists, and for an upload/monitor that a serial port is connected (`pio device list`; ESP32 flashes over its onboard USB-serial bootloader so no external JTAG probe is needed for upload) — and REFUSES rather than run a build/upload that cannot succeed; a non-espressif32 platform is a warning, not a refusal. Emits an `INI_SECTION_ESP32ER<<<` block (`action`, `tool`, `ok`, `returncode`, `success`, `project_dir`, `port`, `environment`, `stage`, `response_body`) and always triggers `target_agents` so a Forker can branch on `{success}` / `{returncode}`. Note: the FIRST build downloads the espressif32 platform + toolchain (hundreds of MB). Canvas counterpart of the `chat_agent_esp32er` Multi-Turn tool. (See the ESP32 Template Project in bonus chapter §58.) |
| **Arduiner** | Arduino firmware bridge built on the **Arduino CLI** (`https://arduino.github.io/arduino-cli/`). The third microcontroller agent and the direct-CLI sibling of ESP32er: like PlatformIO's `pio`, `arduino-cli` is itself a complete CLI, so Arduiner invokes `arduino-cli` subcommands **directly** (**no MCP server**) over a stdlib-only agent (`subprocess` + `urllib` + `zipfile`/`tarfile`). **The microcontroller is selected by `fqbn`** (Fully Qualified Board Name, e.g. `arduino:avr:uno`, `arduino:avr:mega2560`, `esp32:esp32:esp32`); `port` + `baud` set the upload/monitor link. `action` selects ONE capability per run across environment/meta (`bootstrap`, `validate`, `system_info`, `boards`, `device_list`), cores & libraries (`core_update_index`, `core_search`, `core_list`, `core_install`, `core_uninstall`, `lib_update_index`, `lib_search`, `lib_list`, `lib_install`), project lifecycle (`create_project`, `write_source`, `read_source`, `list_sources`), build & flash (`build`, `upload`, `build_and_upload`, `clean`, `list_artifacts`), and serial HIL (`monitor`, `monitor_session` = upload→monitor). **Zero-config auto-bootstrap**: with no on-disk `arduino_cli_executable` and `auto_bootstrap: true`, Arduiner downloads the arduino-cli binary itself (the platform release archive from `downloads.arduino.cc`, unzipped into `%LOCALAPPDATA%/Tlamatini/arduino-cli`) then runs `config init` + `core update-index` — a binary download, not a pip install — so the user installs **only the board USB driver + Tlamatini**. **Auto-core-install**: arduino-cli does NOT auto-install platforms on compile, so before a build/upload Arduiner derives the FQBN's platform and, when missing and `auto_core_install: true`, runs `core update-index` + `core install` (honoring `additional_urls` for third-party ESP32/STM32/RP2040 cores); when off it REFUSES with the exact `core install` to run. **Safety preflight** (fail-safe): validates `arduino-cli` is resolvable + a sketch (`.ino`) + an FQBN exist, and for an upload/monitor that a serial port is connected (`arduino-cli board list`); a malformed FQBN is a warning, not a refusal. **Uniform template project**: `create_project` scaffolds from the bundled `ArduinoTemplateProject/` (the Arduino analog of the STM32 / ESP32 templates), renames the `.ino`, and stamps the FQBN/port into the template's `sketch.yaml` profile. Emits an `INI_SECTION_ARDUINER<<<` block (`action`, `tool`, `ok`, `returncode`, `success`, `fqbn`, `port`, `sketch_path`, `stage`, `response_body`) and always triggers `target_agents`. Note: the FIRST core install + compile downloads the board toolchain so it is slow. Canvas counterpart of the `chat_agent_arduiner` Multi-Turn tool. |
| **ESPHomer** | ESPHome smart-home device-firmware bridge for ESP32 / ESP8266 / RP2040 / BK72xx, built on the **`esphome` CLI** (`https://esphome.io`) — the fourth microcontroller-firmware agent and the odd one out: where STM32er / ESP32er / Arduiner author source code, ESPHomer authors **configuration** (a device is a `*.yaml` file, **NO C++**). Like ESP32er and Arduiner it drives the CLI **directly** (**no MCP server**) over a stdlib-only agent (`subprocess` + `glob`). `action` selects ONE capability per run across environment/meta (`bootstrap`, `validate`, `version`), device-YAML lifecycle (**`new_config`** — a built-in headless generator that writes a minimal valid device YAML from `name` / `platform` / `board` / `wifi_ssid` / `wifi_password` / `led_pin`, the unattended replacement for `esphome wizard` — `write_config`, `read_config`, `config`, `clean`), build & flash (`compile`, `upload`, `run`, `list_artifacts`, and the one-call composite **`scaffold_compile_upload`** = author→config→compile→upload→optional-logs in a single run — fail-safe: it skips only the upload leg with a "compiled OK" result when no board is connected), and serial/OTA HIL (`logs`). **Zero-config auto-bootstrap**: with no on-disk `esphome_executable` and `auto_bootstrap: true`, ESPHomer `pip install esphome` into Tlamatini's own carried Python — no IDE, no Go binary, no installer script — so the user installs **only the board USB driver + Tlamatini**. **Safety preflight** (fail-safe): before any compile/upload it validates `esphome` is resolvable + a device YAML exists, and for an upload/logs/run that a serial port is connected OR an OTA host is supplied in `port` (ESPHome's first flash is USB-serial; after that, every generated device's `ota:` block lets you re-flash over WiFi by passing the device IP as `port`) — and REFUSES rather than run a build that cannot succeed. Emits an `INI_SECTION_ESPHOMER<<<` block (`action`, `tool`, `ok`, `returncode`, `success`, `config_path`, `name`, `port`, `stage`, `response_body`) and always triggers `target_agents`. Note: the FIRST compile downloads the platform + toolchain (via PlatformIO under the hood) so it is slow. Canvas counterpart of the `chat_agent_esphomer` Multi-Turn tool. (See bonus chapter §60.) |
## Cryptography (post-quantum)
| Agent | Purpose |
|---|---|
| **Kyber-KeyGen** | CRYSTALS-Kyber public/private key pair (Kyber-512/768/1024). |
| **Kyber-Cipher** | Kyber encapsulation + AES-256-CTR encryption. |
| **Kyber-DeCipher** | Kyber decapsulation + AES-256-CTR decryption. |
## Utility
| Agent | Purpose |
|---|---|
| **Parametrizer** | Strict single-lane queue mapping source-agent log segments into target-agent config. (See §25.) |
| **FlowBacker** | Post-Ender backup of session logs/configs. |
| **MCP Doctor** | Diagnoses a catalogued **external MCP server WITHOUT connecting to it** — the triage nurse you consult before wiring a new MCP up. Reads the External-MCPs catalog and reports the truth about an entry: which transport it uses (stdio / streamable-http / sse / websocket), what runtime prerequisite it needs (Docker / Node-NPX / UVX / Python / Java / .NET / …), whether that command is actually on PATH, whether its secrets are still placeholders, what is blocking it, and the single next step to take. Emits an `INI_SECTION_MCP_DOCTOR<<<` block so a downstream Forker can branch on `{status}`, and always triggers `target_agents`. The canvas counterpart of the `external_mcp_doctor` tool and of the wrapped `chat_agent_mcp_doctor` Multi-Turn tool; part of the External MCPs feature (a config-driven universal MCP client). |
| **Globber** | Read-only FILE discovery by glob pattern (the Claude-Glob equivalent): finds files matching `pattern` (`*.py`, `**/*.md` — `**` recursive) under `path`, newest-first by default (`sort_by` ∈ mtime / name / none, `max_results` default 500). Files only; changes nothing. The enumeration step ahead of Grepper / File-Interpreter / Editor — prefer it over an Executer `dir`/`ls`. Emits `INI_SECTION_GLOBBER<<<` (`pattern`, `path`, `matches`, `truncated`, `status`, body = the file list) and always triggers `target_agents`. Canvas counterpart of the `chat_agent_globber` Multi-Turn tool. |
| **Grepper** | Read-only regex CONTENT search across one file or a whole tree (the Claude-Grep equivalent): returns `file:line:match` lines for a Python-regex `pattern`, with a `glob` basename filter, `case_insensitive`, `output_mode` ∈ content / files / count, `max_results` default 200; prunes noise dirs (.git / node_modules / venv / __pycache__ / dist / build). Its BOM-first reader searches UTF-8, UTF-16/32, cp1252, and Latin-1 text, testing BOMs before NUL bytes so PowerShell logs and accented Windows/Spanish sources are not mistaken for binaries; genuinely binary or unreadable files are skipped. **Since 2026-09-14 it also READS**: `output_mode: lines` returns a chosen range of ONE file verbatim with no pattern (`start_line`/`end_line`, 1-based inclusive, `0` = first/last), refusing a directory or a binary rather than guessing; with `line_numbers: false` it additionally emits `content_b64`, a base64 copy of the exact bytes — needed because the agent log is written in text mode and rewrites line endings on Windows, so only the encoded copy is safe to build an Editor `old_string` from. That closes the last gap in the toolset: find → search → read → edit now happens entirely inside Tlamatini, with no shell step. Emits `INI_SECTION_GREPPER<<<` (`pattern`, `path`, `glob`, `matches`, `files_searched`, `truncated`, `start_line`, `end_line`, `lines_returned`, `total_lines`, `content_b64`, `status`, body = the matches or the verbatim slice) and always triggers `target_agents`. Canvas counterpart of the `chat_agent_grepper` Multi-Turn tool. |
| **Editor** | Surgical in-place find-and-replace on ONE existing text file (the Claude-Edit equivalent): replaces an EXACT `old_string` with `new_string`, refusing a non-unique match unless `replace_all: true`; `old_string_b64` / `new_string_b64` give a byte-exact channel for code with backslashes/quotes; byte-exact, line endings preserved. State-changing → it IS in the Exec Report. Prefer it over File-Creator when only PART of a file should change. Emits `INI_SECTION_EDITOR<<<` (`file_path`, `status` ∈ edited / not_found / not_unique / noop / error, `occurrences`, `replacements`, body = `response_body`) and always triggers `target_agents` so a Forker can branch on `{status}`. Canvas counterpart of the `chat_agent_editor` Multi-Turn tool. |
| **PDFer** | DOCUMENT COMPOSER: authors a PDF from an answer, Markdown, HTML, plain text, images or existing PDFs, with a cover, tables, page numbers and optional TOC. Modes: `auto`, `markdown`, `html`, `text`, `images`, `mixed`, `merge`, `info`, `validate`, `styles`. `mode: styles` lists **24 visual identities across five families** without writing a PDF (`status: inspected`). Choose cute/nursery, cyberpunk, cosmic, electronics or Tlamatini artwork with `style`; keep document purpose in `nuance`, which retains **20 content themes**. `style: auto` preserves automatic styling. A single `predominant_color` recolors the palette; per-role, font and page overrides remain available. Vector artwork respects the content's decoration ceiling, including no ornament for legal, clinical and financial documents. The default atelier path measures cover text and table widths, continues long cover text onto later pages, and audits the finished PDF for overlap, off-sheet text, blank pages and contrast. Read `layout_clean` and diagnostics before claiming verification. Styles apply to atelier composition; legacy CSS, image-only layouts and existing merged pages retain their own designs. No new runtime dependencies; optional Ollama polish/design remain off. Saves to the Documents known-folder under `TlamatiniPDF`, or a chosen `output_dir`, with collision-safe naming. Emits `INI_SECTION_PDFER`: `mode`, `source_type`, `output_path`, `output_dir`, `filename`, `page_count`, `bytes`, `images_used`, `engine`, `nuance`, `nuance_confidence`, `nuance_source`, `style`, `style_family`, `palette`, `predominant_color`, `background_mode`, `font_pairing`, `decorations`, `overlaps`, `layout_clean`, `repairs`, `status`, plus body=`response_body`. Always triggers downstream agents. Available as a canvas node and `chat_agent_pdfer`; Exec Report and Ask-Execs tier A integration remain. [Styles, examples and verification](Tlamatini/agent/agents/pdfer/STYLES.md). |
| **PPTXer** | Deck composer: builds editable PowerPoint from text, Markdown, outlines, or JSON, with embedded media, diagrams, charts, tables, and generated artwork. **36 named treatments** (24 automatic content treatments + 12 explicit styles) and **17 font pairings**. Set `nuance` to a style key or leave it empty for content classification; color, font, background, and density overrides remain available. Font measurement and continuation slides preserve complete long text. Saved geometry, native text bounds, and rendered contrast are audited; inspect `layout_clean`, `ground_truth`, and confidence, and review `created_with_findings`. Atomic saves protect existing files; native rendering uses a bounded worker for installed PowerPoint. [Style and verification guide](Tlamatini/agent/agents/pptxer/STYLES.md). |
| **LaTeXer** | LaTeX TYPESETTING — the typesetting sibling of PDFer (PDFer *composes* a PDF from Markdown/HTML/images; LaTeXer *typesets* one from real `.tex` source: proper mathematics, bibliographies, numbered cross-references, an index). Embeds the whole `mcp-latex-server` tool surface NATIVELY — no MCP server, no sidecar, no new dependency (stdlib only) — and adds whole-PROJECT compilation (master `.tex` auto-detected, every `\input` followed), a real BibTeX/biber + makeindex + makeglossaries convergence loop, latexmk pass-through, and LaTeX-log diagnostics a human can read. `action` ∈ compile / compile_project / scaffold_compile / create_file / create_from_template (article, report, book, beamer, letter, cv, homework, spanish-article) / edit_file / read_file / list_files / validate_tex (a STATIC lint needing no LaTeX) / structure / clean / validate / install / list_styles; `auto_preamble` wraps a bare fragment like `$E = mc^2$`. **Compilation requires a TeX distribution; MiKTeX is recommended** — Tlamatini bundles no TeX distribution (several GB; the release stays under 2 GB), and MiKTeX is preferred because it installs a missing `.sty` on demand mid-compile, when the package is available; with no engine, compilation REFUSES cleanly and `action: install` launches the official installer. ⚠️ latexmk is probed for USABILITY not presence (it is a Perl script most Windows boxes cannot run → automatic fallback to the built-in loop). `shell_escape` OFF by default (`\write18` = arbitrary command execution). PDFs land in `Documents/TlamatiniLaTeX`, projects in `Templates/LaTeXer`; `compiled_with_errors` means a PDF exists but LaTeX complained and is reported as exactly that. State-changing → in the Exec Report and on the Ask-Execs tier-A allowlist. Emits `INI_SECTION_LATEXER<<<` (`action`, `engine`, `distribution`, `tex_path`, `project_dir`, `output_path`, `output_dir`, `filename`, `page_count`, `bytes`, `passes`, `bibliography`, `errors`, `warnings`, `success`, `status`, `style`, `style_family`, `style_mode`, `style_count`; body = `response_body`) and always triggers `target_agents`. Canvas counterpart of the `chat_agent_latexer` Multi-Turn tool. **30 opt-in styles across six families**: editorial, cute/nursery, cyberpunk, cosmic, electronics and extraterrestrial Tlamatini. `template` selects the document structure; `style` selects its visual identity. The catalogue action needs no TeX; see the [signature collection](Tlamatini/agent/agents/latexer/STYLES.md). |
| **Discoverer** | **ProjectDiscovery** recon-suite bridge — ONE tool per run: `subfinder` (passive subdomain enum) / `httpx` (HTTP probe + fingerprint) / `naabu` (port scan; Windows-safe CONNECT default) / `katana` (crawler) / `nuclei` (template vuln scan) / `cvemap`→`vulnx` (CVE search; cvemap's own API was retired Aug 2025, so the tool runs `vulnx`), plus meta `bootstrap` / `validate` / `update_templates` / `list_tools`. Direct CLIs — no MCP server, stdlib-only, the sibling of Kalier / ESP32er / Arduiner. **Zero-config PRIVATE Go toolchain**: on first use it downloads the Go compiler into `<install_dir>/Go` and `go install`s the tools into `<install_dir>/Go/bin-tools` (no system Go, no PATH change; slow once, then cached). PDCP key OPTIONAL — set once in **Config ▸ Access Keys Wizard ▸ "Security Recon (ProjectDiscovery)"** (auto-injected on every run; redacted from `.flw` exports and by `regen_secrets.py`). Fail-safe preflight REFUSES rather than mis-scan. Emits `INI_SECTION_DISCOVERER<<<` (`tool`, `target`, `returncode`, `success`, `findings_count`, `json_path`, `pdcp_used`, `stage`, body = output) and always triggers `target_agents`. **Authorized targets only.** Canvas counterpart of the `chat_agent_discoverer` Multi-Turn tool. |
| **Video-Analyzer** | Video-Analyzer supports `analysis_type: robotics` (default), `transcription`, and `summary`. Robotics preserves the deterministic motion gate and dual-vision/merge verdict, with `PASS_OK` only on two explicit independent passes. Transcription reads selected video audio tracks (`audio_tracks: all` or `0,1`) using Whisperer's local faster-whisper backend, GPU auto/CPU fallback and timestamped segments; it never opens a microphone. Summary combines speech with two independent visual observers over frame batches spanning the whole clip, then synthesizes an overview, chronology, readable screen text, facts, steps, decisions, action items and limitations. Content modes accept static scenes, bypass the motion gate, and emit `TLM_ANALYSIS::` tokens rather than robotics verdicts. Missing audio/speech and partial failures are explicit. Each content run saves transcript, segments, report and full analysis artifacts. Parametrizer and wrapped chat results expose `analysis_type`, `analysis_token`, `transcript`, `summary`, `audio_status`, timestamped `segments_json` and artifact paths; the body remains `response_body`. Input remains a file, wildcard, newest video in a folder or Camcorder pool name, or a web link (YouTube and other sites): Video-Analyzer downloads it itself (one item; sound only for transcription), joins picture and sound with PyAV when ffmpeg is absent, keeps a self-updating private yt-dlp, names every download failure, and adds `source_url`/`source_title` to its result. Always starts downstream agents; sampled perception is not exhaustive. |
| **NetSpeed-Calculator** | Measures THIS machine's Internet connection and reports it **with an error bar** — download, upload, latency, jitter, packet loss and **bufferbloat** — following RFC 6349 (TCP throughput) and RFC 3550 §6.4.1 (jitter) against SEVERAL keyless public providers at once (Cloudflare, Ookla, Fast.com/Netflix, LibreSpeed, Hetzner, CacheFly — no API key, no login, no account). Per provider and direction it opens `parallel_streams` concurrent TCP flows, **DISCARDS the slow-start ramp** (`warmup_seconds`), and samples **d(bytes)/dt** every `sample_interval_seconds` rather than the naive total÷elapsed — which is what makes dropping the ramp possible and stops a constant socket-buffer occupancy inflating the upload figure. Samples are Tukey-IQR outlier-rejected, trimmed-mean averaged and given a Student-t interval; the providers are then FUSED by a **DerSimonian-Laird random-effects meta-analysis** that publishes a 95% confidence interval and the **I²** heterogeneity figure — so when CDNs genuinely disagree the interval widens on purpose instead of the flattering one winning. Bufferbloat is the RTT increase *under load*, graded **A+ → F**, and is usually the real reason a "fast" link has choppy calls. Endpoint discovery is live and self-healing: Ookla and LibreSpeed servers are chosen by **measured RTT** (a published list is geography, and geography is only a proxy for network distance), Cloudflare's object is clamped below the size it rejects with HTTP 403, LibreSpeed's retired backend is replaced by its public server list, `speed.hetzner.de` (now NXDOMAIN) by the `.com` mirror mesh, and Hetzner's reset-on-query-string is handled by a per-provider `cache_bust` flag plus a runtime self-heal. **A transfer that moves zero bytes must NAME its cause** — in the log and in the saved result — because a silent `0.00 Mbps` is indistinguishable from a slow link. `action` ∈ full / download / upload / latency / **validate** / providers; `validate` probes reachability without running throughput measurements, and `providers` prints the read-only catalog. Saves a full JSON artifact under `<app>/Temp/NetSpeedCalculator`; a fail-safe preflight REFUSES rather than publish a number it cannot trust. Emits `INI_SECTION_NETSPEED_CALCULATOR<<<` and always triggers `target_agents`. **⚠️ Consumes real, possibly metered bandwidth (~100-200 MB per full run)** and deliberately saturates the link — Ask-Execs tier D. Canvas counterpart of the `chat_agent_netspeed_calculator` Multi-Turn tool. |
### PPTXer: choose a style and keep complete text visible
Use `nuance` for the visual identity. The twelve additions are `swiss_editorial`, `warm_editorial`, `midnight_luxe`, `botanical`, `oceanic`, `blueprint`, `terracotta`, `nordic_frost`, `bauhaus`, `lavender_studio`, `monochrome_ink`, `sunset_coral`. Together with the original 24 content treatments, they make 36 named treatments. Empty `nuance` keeps automatic classification; a style key or display name selects a preset. Existing color, font, background, and density overrides take priority.
```yaml
action: create
nuance: blueprint
slide_size: "16:9"
generate_art: true
input_text: |
# Architecture review
## Delivery plan
- Explain the system and its next milestone.
```
Long content is measured and can continue across slides; 256 characters is a tested field length, not a maximum. Inspect `status`, `layout_clean`, `ground_truth`, and the report's confidence before delivering a deck. `created_with_findings` means a deck exists and needs review. The [canonical guide](Tlamatini/agent/agents/pptxer/STYLES.md) records the 126-test snapshot, the 151-slide native new-style run, exact scope, and commands to regenerate the disposable evidence.
### LaTeXer: choose a structure, then a style
Use `template` for article, report, book, beamer, letter, cv, homework or spanish-article. Independently choose one of the 30 `style` IDs: for example `baby_blush`, `cyberpunk`, `cosmic_nebula`, `circuit_board` or `quetzal_supernova`. `action: list_styles` returns all IDs, aliases, families and motifs without an installed TeX engine.
```yaml
action: scaffold_compile
template: report
style: tlamatini_celestial
title: Signals from the stars
subtitle: A notebook of discovery
style_mode: print
style_decoration: rich
style_cover: true
predominant_color: "#238D87"
content: |
\section{First observations}
Every signal is a question waiting for a measurement.
```
`screen` uses the selected style's light or dark palette; `print` uses white paper. `style_decoration` accepts none/restrained/rich, and a false `style_cover` gives a compact title (Beamer retains its title frame without illustration). Report/book include contents; CV/homework use compact headings; letters use a conventional opening and signature. Beamer has a separate rendering recipe and accepts a frame body. Metadata and content remain LaTeX: escape literal special characters and use `content_b64` for complex chat payloads.
Styles apply to `create_file`, `create_from_template`, `scaffold_compile`, and fragment `compile` with `auto_preamble: true`. Omit `style` for complete `.tex` sources and projects; those already own their preamble. Empty/none/plain/default preserves legacy rendering. Unknown styles are refused. The generated source is editable and uses original TikZ geometry and portable Latin Modern fonts. The palette maintains a 4.5:1 text contrast floor against assigned grounds; custom LaTeX still needs normal layout review.
The [complete style guide](Tlamatini/agent/agents/latexer/STYLES.md) includes all IDs, table/callout helpers, result fields and the real-engine verification matrix. In flows, inspect `status`/`success` before forwarding `output_path`; a catalogue response is `listed`, with `style_count: 30`, and creates no PDF.
## Terminal / monitoring (do NOT start downstream)
| Agent | Purpose |
|---|---|
| **Monitor-Log** | LLM-powered log file monitor. |
| **Monitor-Netstat** | LLM-powered network port monitor. |
| **Emailer** | SMTP email on pattern detection. |
| **RecMailer** | IMAP receiver with LLM keyword analysis. |
| **Notifier** | Browser popup + optional sound on pattern detection (LangGraph). |
| **FlowHypervisor** | LLM watchdog over running agents. (See §23.) |
## AI / design
| Agent | Purpose |
|---|---|
| **FlowCreator** | LLM that designs flows from natural-language objectives. (See §24.) |
---
# Part V — The Tool Surface
Every tool the chat LLM can call in Multi-Turn mode. Tools can be individually enabled/disabled via the **Tools Dialog** in the chat.
## 27. Core tools
| Tool | What it does |
|---|---|
| `get_current_time` | Returns current datetime. |
| `execute_command` | Shell command. |
| `execute_file` | Run a Python script in a new terminal. |
| `execute_netstat` | Network diagnostics. |
| `launch_view_image` | Open an image in a viewer. |
| `unzip_file` | Extract ZIP archives. |
| `decompile_java` | Decompile JAR/WAR via bundled `jd-cli`. |
| `opus_analyze_image` | Image analysis with Claude Opus. |
| `qwen_analyze_image` | Image analysis with Qwen via Ollama. |
| `googler` | Direct Playwright web search + extraction; put manual Google operators in `query`. Structured dork fields/presets are available on the visual/pool Googler agent. |
| `agent_parametrizer` | Configure a template workflow agent from chat. |
| `agent_starter` | Start a template workflow agent from chat. |
| `agent_stopper` | Stop a template workflow agent. |
| `agent_stat_getter` | Check template-agent runtime status. |
| `interactor` | Read and edit the live unsaved canvas; teach one segment, guide a learner field, save with authorization. |
## 28. Wrapped chat-agent tools (68)
Use `chat_agent_torrenter` to inspect a source, submit selected files and read a verified manifest. Jobs keeps the transfer running when the chat closes. See the [Torrenter guide](docs/torrenter.md).
Each wrapped tool launches an isolated, sequenced runtime copy of a workflow agent template under `agent/agents/pools/_chat_runs_/{agent}_{seq:03d}_{short_id}/`. Failed runs are preserved.
| Family | Tool names |
|---|---|
| **Execution & files** | `chat_agent_executer`, `chat_agent_pythonxer`, `chat_agent_terminater`, `chat_agent_pser`, `chat_agent_move_file`, `chat_agent_deleter`, `chat_agent_sleeper` |
| **DevOps & infra** | `chat_agent_gitter`, `chat_agent_dockerer`, `chat_agent_kuberneter`, `chat_agent_jenkinser`, `chat_agent_ssher`, `chat_agent_scper` |
| **Data & interpretation** | `chat_agent_sqler`, `chat_agent_mongoxer`, `chat_agent_file_creator`, `chat_agent_file_extractor`, `chat_agent_file_interpreter`, `chat_agent_image_interpreter`, `chat_agent_summarize_text` |
| **Notifications & comms** | `chat_agent_send_email`, `chat_agent_notifier`, `chat_agent_telegrammer`, `chat_agent_whatsapper`, `chat_agent_recmailer` |
| **Desktop UI automation** | `chat_agent_shoter` (read-only), `chat_agent_camcorder` (read-only — webcam photo/video via OpenCV; canvas counterpart is the Camcorder workflow agent), `chat_agent_recorder` (read-only — microphone audio → WAV via `sounddevice`; canvas counterpart is the Recorder workflow agent), `chat_agent_audioplayer` (observational/output — plays an audio file to the speakers via `soundfile` + `sounddevice`, with `volume_percent` and a `time_played` truncate/loop; canvas counterpart is the AudioPlayer workflow agent), `chat_agent_videoplayer` (observational/output — plays a video file with audio on a chosen display via `ffpyplayer` + OpenCV, with `display_index` / `volume_percent` / `time_played` truncate-loop / window-size / `fullscreen`; canvas counterpart is the VideoPlayer workflow agent), `chat_agent_keyboarder`, `chat_agent_mouser`, `chat_agent_windower` |
| **Routing** | `chat_agent_asker` |
| **Archives & decompilation** | `chat_agent_j_decompiler`, `chat_agent_de_compresser` |
| **Game engines** | `chat_agent_unrealer` (drives an Unreal Engine 5 editor via the Unreal MCP plugin's TCP socket; canvas counterpart is the Unrealer workflow agent — see §57) |
| **3D / DCC** | `chat_agent_blenderer` (drives Blender via the official Blender MCP add-on's TCP socket — a code-execution protocol with a rich action catalog; canvas counterpart is the Blenderer workflow agent — see §59) |
| **Embedded / firmware** | `chat_agent_stm32er` (scaffolds / builds / flashes / observes STM32F407VG firmware through the STM32 Template Project MCP — 23 MCP tools + `serial_session` / `live_monitor` composites + `bootstrap` / `validate` meta-actions; zero-config auto-bootstrap downloads the MCP itself, and a safety preflight refuses to compile or flash on a bad toolchain / missing ST-LINK / wrong device family; canvas counterpart is the STM32er workflow agent), `chat_agent_esp32er` (ESP32 firmware via PlatformIO `pio` CLI directly — **no MCP server**; `scaffold_build_upload` collapses create→write→build→upload into one run; zero-config `get-platformio.py` bootstrap + serial-port preflight; canvas counterpart is the ESP32er workflow agent), `chat_agent_arduiner` (Arduino firmware via `arduino-cli` directly — **no MCP server**; MCU chosen by `fqbn`; zero-config arduino-cli binary bootstrap + auto-core-install; ships an ArduinoTemplateProject scaffold; canvas counterpart is the Arduiner workflow agent) |
| **Web & browser** | `chat_agent_playwrighter` (drives a real browser through a scripted step list — login, forms, clicks, waits, extraction, screenshots, asserts, downloads; canvas counterpart is the Playwrighter workflow agent) |
| **Crawling, monitoring, APIs, prompts, crypto** | `chat_agent_crawler`, `chat_agent_monitor_log`, `chat_agent_monitor_netstat`, `chat_agent_apirer`, `chat_agent_prompter`, `chat_agent_kyber_keygen`, `chat_agent_kyber_cipher`, `chat_agent_kyber_deciph` |
## 29. Wrapped runtime lifecycle tools (6)
After launching a wrapped agent, you can monitor and control it:
| Tool | What it does |
|---|---|
| `chat_agent_run_list` | List recent runs (capped by `chat_agent_limit_runs`). |
| `chat_agent_run_status` | Inspect status of one run. |
| `chat_agent_run_log` | Read the latest log excerpt. |
| `chat_agent_run_stop` | Stop a run by `run_id`. |
| `chat_agent_run_wait` | **Block** until a run reaches a terminal status (or `max_seconds` fires). Replaces busy-poll loops. |
| `window_present(title)` | Fast (<100 ms) yes/no helper for "is this window open?" — use this instead of `chat_agent_image_interpreter` for window-presence gates. |
## 30. ACPX & Skills tools (12)
The ACPX/Skill surface. Every tool returns a JSON envelope. Failure envelopes are always `{ ok: false, reason: "...", code: "..." }`.
| Tool | What it does |
|---|---|
| `acp_doctor` | Health-probe the runtime + enumerate every registered ACP agent with on-PATH `resolvable` and `cli_version`. **Always call first** when starting an ACPX flow. |
| `list_acp_agents` | Cheap enumeration without the probe. |
| `acp_spawn(agent_id, task, …)` | Spawn an external CLI as a child process. Returns `session_id`, `transport`, `transcript_path`, `events`. **TUI agents return sub-second**; pass `timeout_seconds>0` to force a drain on spawn. |
| `acp_send(session_id, text, …)` | Send a follow-up turn. |
| `acp_send_and_wait(session_id, text, until_idle_seconds=10, max_wait_seconds=180)` | Send and **block until the child settles**. Prefer this for "wait for the full answer" prompts. |
| `acp_kill(session_id)` | Terminate a session. Returns `transcript_path` so the row in Exec Report can cite it. |
| `acp_transcript(session_id, max_chars, direction)` | Read the on-disk NDJSON transcript. Use for harvest / cite-evidence prompts. |
| `acp_session_status(session_id)` | `{alive, pid, transcript_size, last_event_at, closed}`. |
| `acp_list_sessions` | Enumerate live sessions. |
| `acp_relay(session_id_src, session_id_dst, transform, …)` | **Single-call hand-off** between sessions. Replaces transcript→manipulate→send. |
| `invoke_skill(skill_name, args_json)` | Run a registered SKILL.md package inside the `SkillHarness`. |
| `list_skills(filter_keywords)` | List every registered skill. |
The 27 seed skills (`agent/skills_pkg/<name>/SKILL.md`) cover: `hello-world`, `skill-creator`, `acp-router`, `setup-new-acpx-key`, `summarize`, `weather`, `flow-making` (objective → canvas-loadable `.flw` by driving the FlowCreator engine), `create-new-agent` / `create-new-mcp` (authoring runbooks for Tlamatini's own surfaces), `code-review` (senior-engineer git-diff review with a verdict), `security-audit` (multi-scanner SAST/secret/dependency sweep), `kali-pentest` (authorized Kali Linux / MCP-Kali-Server assessment runbook driving the Kalier agent), `tlamatini-*` (8 maintenance skills: csrf-exempt-audit, exec-report-row-adder, allowed-hosts-tighten, planner-trace-replay, flow-from-objective — now delegates to flow-making, flw-doctor, new-acp-agent, static-version-bumper), and OpenClaw-format ports for `github`, `notion`, `jira`, `slack`, `gmail`, `todoist`, `trello`.
---
# Part VI — Inside Tlamatini
This is the deep-dive section. Skip if you only want to use Tlamatini.
## 31. The big picture
```
┌─────────────────────────────────────────────────────────────────────┐
│ Browser (Chat UI / ACP Workflow Designer) │
└──────────────────────────────────┬──────────────────────────────────┘
│ WebSocket (ws://)
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Django Channels (Daphne ASGI) │
│ AgentConsumer → routing, session, heartbeat │
└──────────────────────────────────┬──────────────────────────────────┘
│
┌───────────────────────────┼───────────────────────────┐
▼ ▼ ▼
┌──────────────────┐ ┌────────────────────┐ ┌─────────────────────┐
│ RAG Pipeline │ │ Unified Agent │ │ MCP Services │
│ Document loader │ │ Multi-Turn loop │ │ System metrics │
│ FAISS + BM25 │ │ Tool execution │ │ (WebSocket) │
│ Context budget │ │ Planner / DAG │ │ File search (gRPC) │
│ Fallback mode │ │ Wrapped runtimes │ │ │
└──────────────────┘ └────────────────────┘ └─────────────────────┘
│ │ │
└─────────────┬─────────────┴─────────────┬─────────────┘
▼ ▼
┌─────────────────────────┐ ┌──────────────────────────┐
│ LLM Backends │ │ ACPX Runtime │
│ Ollama Claude Qwen │ │ External CLIs as kids │
└─────────────────────────┘ └──────────────────────────┘
```
## 32. The Five Layers
The system is organized in five conceptual layers. Each layer has a single responsibility.
| Layer | Responsibility | Where it lives |
|---|---|---|
| **1. Persisted toggles** | Database rows for `Mcp`, `Tool`, and `Agent` — the UI's enable/disable state. | `agent/models.py` |
| **2. Runtime MCP services** | System-Metrics (WebSocket) and Files-Search (gRPC) running as daemon threads. | `agent/mcp_*` |
| **3. Context fetcher chains** | LCEL sidecars that fetch system / files context and inject it into the answer chain. | `agent/chain_*_lcel.py` |
| **4. Main answer chains** | Basic / History-aware / Unified chains. `factory.py` monkey-patches `invoke()` to wire context from sidecars. | `agent/rag/chains/` |
| **5. Unified-agent tools** | Synchronous LangChain `@tool` functions returned by `get_mcp_tools()`. Only active when the unified-agent chain is selected. | `agent/tools.py` |
## 33. RAG pipeline
When you set a directory as context:
1. **Load** every text file under the path.
2. **Chunk** into 3000-character windows with 800-char overlap.
3. **Extract metadata** — class names, function names, imports, file roles (`controller`, `data_model`, `service_layer`, …).
4. **Embed** each chunk using the model in `config.embeding-model`.
5. **Build** the FAISS index.
6. **Build** the BM25 index in parallel.
7. At query time, both indexes return their top-K hits; **Reciprocal Rank Fusion** combines them; **context budgeting** picks chunks within token limits, allocating 60% to high-relevance, 20% to architecture, 15% to related, 5% to documentation.
If embedding fails (out-of-memory), **memory-insufficient fallback** kicks in: the loaded source files are packed into a raw context block and injected directly into the prompt-only / unified-agent path. You get reduced retrieval quality, not a wiped chat.
## 34. Embedding-memory pre-flight guard (GPU hosts)
When you click **Set directory as context** in the Context menu, Tlamatini is about to do something dangerously bursty: walk every text file under the path, split each into chunks, push every chunk through Ollama's embedding API to build a FAISS index, and only then return control to the chat. On a PC or consumer GPU this is the moment of truth — if the embedding model needs more VRAM than the GPU can spare, Ollama starts evicting and re-loading on every batch, and what should be a thirty-second operation turns into a multi-hour stall while RAM and VRAM swap back and forth across the PCIe bus. The dev box this codebase is calibrated on — an RTX 4070 Laptop with 8 188 MiB of VRAM — sees exactly this with the default embedding model `qwen3-embedding:8b`, which sits at ~6.24 GB resident, 77.9 % of total. Add a chat model on top, and the GPU is over capacity.
The **embedding-memory pre-flight guard** (`Tlamatini/agent/embedding_memory_guard.py`, introduced 2026-05-12) catches the problem before the embed burst starts. It runs only when an NVIDIA GPU is detected. On CPU-only, AMD, and Apple Silicon hosts it is a silent no-op, and the legacy load path is unchanged. The guard is informational and non-blocking — it raises a chat-bubble warning, then lets the load proceed. The choice of whether to wait it out, hit Cancel, or change models in `config.json` belongs to the user.
### Where the guard fires
There is exactly one hook in the codebase, and it lives in `agent/consumers.py::setup_contextual_rag_chain`. After the consumer broadcasts the "Loading context…" banner to the chat (`MSG_AGENT_LOADING_CONTEXT`), and before it schedules the heavy `asyncio.to_thread(setup_llm_with_context, …)` call that drives `FAISS.from_documents(...)`, the guard runs inside its own `asyncio.to_thread` — both so a slow `nvidia-smi` probe never blocks the Channels event loop, and so the whole step can be wrapped in `try/except Exception` to enforce a strict fail-open contract.
```
WebSocket "set-directory-as-context"
↓
consumers.py:setup_contextual_rag_chain(path_only)
↓
broadcast MSG_AGENT_LOADING_CONTEXT ──→ user sees "Loading context…"
↓
► embedding_memory_guard.check_embedding_memory_for_directory(...)
│
├── returns None → proceed silently
│ (no GPU, under threshold, or any probe failed)
│
└── returns warning → broadcast HTML warning chat bubble
→ proceed anyway (informational, non-blocking)
↓
asyncio.to_thread(setup_llm_with_context, ...)
└─ OllamaEmbeddings + FAISS.from_documents(...) ← the VRAM burst
```
If anything goes wrong inside the guard — `nvidia-smi` missing, Ollama unreachable, a JSON-shape change in a future Ollama release — the `try/except` swallows the exception, prints a one-line `--- [EMBED-MEM] Pre-flight check skipped (fail-open): …` to `tlamatini.log`, and the load continues. A diagnostic must never block the user. This is the same fail-open philosophy `agent/gpu_perf.py` uses for its model-pinning hook, and it pairs naturally with the existing `_has_nvidia_gpu()` cache that the guard re-uses for its first gate.
### Detection — who pays for the check
The guard's very first action is to ask the cached `gpu_perf._has_nvidia_gpu()` probe whether this host has an NVIDIA GPU at all. That helper runs `nvidia-smi -L` exactly once per process and caches the result in a module-level boolean. On a CPU-only Linux or Windows machine, an AMD GPU, or Apple Silicon, the probe returns `False` once at server start, and **every subsequent call to the guard returns `None` immediately** — no subprocesses spawned, no HTTP calls made, no overhead beyond a single boolean check.
This is the portability guarantee. A fresh `git pull` on a no-GPU box behaves exactly as before the guard existed. The 28 dedicated no-GPU compatibility tests in `agent/test_embedding_memory_guard.py::NoGpuCompatibilityTests` pin the contract in place; the most consequential of them, `test_real_entry_point_call_never_raises`, calls the real entry point with the real subprocess + real urllib code paths and asserts the return is **either** `None` **or** a well-formed warning dict — never an exception. The same test passes on this RTX 4070 dev box and on a CPU-only CI image.
### Three-tier VRAM prediction
When the GPU gate passes, the guard predicts the embedding model's resident VRAM in three tiers, in priority order — each more accurate than the next is convenient:
**Tier A — model already resident.** A `GET /api/ps` against the Ollama daemon returns a list of currently-loaded models, each with a `size_vram` field that is the exact bytes Ollama allocated for that model. If the configured `embeding-model` shows up in the list, the guard uses `size_vram` verbatim. There is no estimation here — this is daemon ground truth.
**Tier B — model on disk but not loaded.** A `POST /api/show` returns the model's metadata: parameter count, quantization level, embedding dimension, layer count, attention shape. The guard computes weights bytes as `parameter_count × bits_per_weight(quant) / 8`, then multiplies by an overhead factor that accounts for KV cache, activation buffers, and the GGML allocator's slack. The bits-per-weight table follows the standard llama.cpp / GGUF averages — `F16` is 16 bits, `Q8_0` is 8.5, `Q4_K_M` is 4.83, `Q2_K` is 2.96. Unknown quants fall back to a conservative `5.0`. The overhead is **×1.40** for models with at least one billion parameters and **×2.20** for sub-1 B models, where the proportional cost of fixed KV/buffer overhead is larger.
**Tier C — anything else.** If Ollama is down, the model has not been pulled, or it is a cloud model (`:cloud` suffix), the guard returns `None` and lets the load proceed. This is fail-open by design.
The overhead numbers are not guesses; they were calibrated on this dev box against two real models:
| Model | params × bits/8 (raw) | × overhead = predicted | Ollama-reported resident | error |
|---|---|---|---|---|
| `qwen3-embedding:8b` (Q4_K_M) | 4.54 GB | **6.36 GB** (× 1.40) | 6.24 GB | +1.9 % |
| `Nomic-Embed-Text:latest` (F16) | 274 MB | **603 MB** (× 2.20) | 600 MB | +0.5 % |
Both predictions land within 2 % of the measured value. The two-tier overhead split (large vs small) gives a tighter fit than any single multiplier would have. When a future model family proves the calibration off by more than 10 %, the two `_OVERHEAD_*` constants in the module are the only knobs to recalibrate.
The same `/api/show` payload also tells the guard the model's embedding dimension. The key is architecture-prefixed — `qwen3.embedding_length=4096`, `nomic-bert.embedding_length=768` — and the guard finds it by suffix match, so a new architecture (whatever Ollama adopts next) does not need code changes.
### What VRAM is the model competing against?
The guard reads total VRAM via `nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits`, parses one integer per line, and **picks the smallest of the values**. Why the smallest? Because Ollama loads each model into a **single device** by default. Using the sum of all GPUs would silently under-report the constraint on a heterogeneous rig (e.g. an 8 GiB consumer card paired with a 4 GiB eGPU), and using the largest would mis-predict on the bad cases the warning is meant to catch. The smallest single-GPU value is the conservative constraint.
The 80 % threshold then compares: when **predicted_vram ≥ 0.80 × smallest_gpu_total**, the warning fires. The threshold is a function argument (default `0.80`) so the trigger can be tuned per installation. The number was picked to leave room for chat-model overhead, OS-reserved memory, and the activation buffers Ollama needs on every embed batch — anything tighter than 80 % is already in the "expect slow embed throughput or RAM↔VRAM swap" zone.
### The chat-bubble warning
When the threshold trips, the guard returns a structured dict and the consumer renders it as an HTML chat bubble — the same `agent_message` surface the chat already uses for every Tlamatini-side message. A real example from this dev box, with the threshold artificially lowered to 70 % so `qwen3-embedding:8b` trips:
> ⚠️ **Embedding-memory warning**
>
> Embedding model `qwen3-embedding:8b` needs ~6,378 MiB of VRAM (currently resident in VRAM), which is **77.9 %** of the smallest GPU's total (8,188 MiB) — above the safety threshold of 70 %.
>
> Projected FAISS vector store (RAM, not VRAM): ~28 MiB across 1,847 chunks at dim 4096.
>
> Context loading will continue, but expect slow embedding throughput or RAM↔VRAM swap. To eliminate the pressure, switch `embeding-model` in `config.json` to a smaller model (e.g. `nomic-embed-text:v1.5`) and restart.
Notice the source qualifier: *currently resident in VRAM* means the prediction came from Tier A — exact `/api/ps`. *Estimated from model parameters* means Tier B. The user always knows whether they are looking at ground truth or an extrapolation.
The FAISS line is an informational projection. The guard pre-scans the chosen directory the same way `CustomTextLoader` will, applying the same exclusions (`package-lock.json`, `yarn.lock`, `*.<user-omitted>`, …), and counts projected chunks per file using `ceil(file_size / (chunk_size − chunk_overlap))`, capped by `max_chunks_per_file`. Multiplied by `embedding_dim × 4` bytes (float32 vector storage), that gives the FAISS index's RAM footprint. This is **not** VRAM; FAISS lives on CPU. It is shown for one reason: very large directories produce very large indexes that can matter for the host's RAM budget independently of the GPU question.
### Tuning, overrides, and explicit non-goals
The guard exposes four knobs, all in the module:
| Knob | Default | When to change |
|---|---|---|
| `threshold` (function argument) | `0.80` | Pass `0.70` on small GPUs (6 GB cards) where 80 % is already too tight. |
| `_OVERHEAD_LARGE` constant | `1.40` | Recalibrate against `/api/ps` if a new model family proves it off by > 10 %. |
| `_OVERHEAD_SMALL` constant | `2.20` | Same calibration story for sub-1 B models. |
| `_QUANT_BITS` dict | standard table | Add an entry when a new GGUF quant ships. |
And four things the guard deliberately does **not** do:
It does not abort context loading. The warning is informational. If you want abort-on-warning behavior, you can wire a confirm/cancel WebSocket round-trip — the surface is described in `agent_page_init.js` near the `set-dir-context` listener.
It does not estimate the chat model's VRAM. Only the embedding model is checked, because that is the model the directory-load path forces into VRAM at this exact moment. The chat model is handled separately by `gpu_perf.pin_ollama_model`.
It does not persist warnings. Each context-load runs an independent check; toggling `config.json` and reloading will surface a fresh prediction next time.
It does not call `nvidia-smi` on CPU-only hosts. Both gates — the cached `_has_nvidia_gpu_cached()` and the `_gpu_total_memory_bytes()` query — short-circuit before any subprocess spawn. `top` on a CPU-only box will show zero new processes when a user clicks **Set directory as context**.
### Test coverage
The guard ships with **49 automated tests** in `agent/test_embedding_memory_guard.py`, organized into seven `SimpleTestCase` classes. Two of them deserve special mention.
`PredictFromShowTests` (3 tests) pins the Tier-B math against the two reference models the calibration was derived from: `qwen3-embedding:8b` must predict within ±5 % of the 6.24 GB measured, and `Nomic-Embed-Text:latest` within reasonable bounds of the 600 MB measured. If a future change to the overhead constants or the bits-per-weight table breaks these, the suite fails loudly.
`NoGpuCompatibilityTests` (28 tests) is the portability proof. Its coverage matrix walks every failure mode a non-NVIDIA / no-driver / no-Ollama machine can hit: `nvidia-smi` binary missing entirely; `nvidia-smi` present but driver unloaded; the binary hangs and `subprocess.run` raises `TimeoutExpired`; permission denied; generic `OSError`; empty or unparseable output; heterogeneous multi-GPU rig; `gpu_perf` module missing or its probe raising; Ollama daemon offline; closed port; malformed URLs; model not in `/api/ps`; entry on a CPU-only host; GPU detected but `--query-gpu` fails; GPU detected but Ollama offline; pathological 0 MiB GPU readings; empty `ollama_base_url`; deleted or non-existent directory paths; unreadable files inside the walked tree; partial warning dicts with missing optional keys. The capstone test, `test_real_entry_point_call_never_raises`, makes the *actual* `subprocess.run(["nvidia-smi", …])` and `urlopen("http://127.0.0.1:11434/…")` calls against whatever the runner offers, and asserts the return is **either** `None` **or** a well-formed warning dict — never an exception. The same test passes on this RTX 4070 dev box (returns `None` because qwen3-embed sits at 77.9 %, just under the 80 % gate) and on a CPU-only CI image (returns `None` because the GPU gate fails fast).
To run the full guard suite:
```
cd Tlamatini
python manage.py test agent.test_embedding_memory_guard --verbosity=2
```
Forty-nine tests in roughly 2.3 seconds, no database setup, no GPU required.
## 35. Multi-Turn execution pipeline
Below the toolbar checkbox, here is what really happens when you tick **Multi-Turn**:
```
1. FRONTEND
User types message + ticks Multi-Turn
→ WebSocket sends {message, multi_turn_enabled: true,
exec_report_enabled: ?, acpx_enabled: ?,
ask_execs_enabled: ?}
↓
2. WEBSOCKET CONSUMER (consumers.py)
Saves to DB, broadcasts user message, queues LLM retrieval
↓
3. RAG INTERFACE (rag/interface.py)
Bypasses prompt-shape validation (acpx_enabled OR multi_turn_enabled)
Bypasses path-access validation
Passes flags into the chain payload
↓
4. UNIFIED RAG CHAIN (rag/chains/unified.py)
Retrieves docs, builds enhanced input
Bypasses file-listing short-circuit
Filters tools through agent.acpx.filter_acpx_tools(tools, acpx_enabled)
↓
5. CAPABILITY-AWARE EXECUTOR (mcp_agent.py)
Picks the relevant tool subset (default cap: 20 tools)
Builds a request-scoped MultiTurnToolAgentExecutor
↓
6. MULTI-TURN TOOL LOOP
for i in 1..unified_agent_max_iterations (default 4096):
LLM call with bind_tools(selected_tools)
if tool_calls:
for each call (after dedup + quota):
if ask_execs_enabled and call is state-changing:
BLOCK on browser Proceed/Deny (ExecPermissionBroker)
if DENIED: record denial, HALT the whole chain
execute, append ToolMessage
continue
if pure text: that's the final answer, exit loop
↓
7. EXEC REPORT (if exec_report_enabled)
Capture every state-changing tool call into _exec_report_entries
Render <table class="exec-report-...">
Append to llm_response BEFORE save_message (strict ordering)
If a tool was DENIED (Ask Execs): append the red "Execution
interrupted" banner (always, regardless of exec_report_enabled)
↓
8. WEBSOCKET BROADCAST
{message, tool_calls_log, multi_turn_used}
↓
9. FRONTEND
appendChatMessage() renders prose, then exec-report tables / denial banner
if Multi-Turn ran with ≥1 SUCCESSFUL agent (and user not anonymous)
→ render "Create Flow" button (no whole-answer classifier gate;
the flow is built from only the successful agents)
```
### Self-healing model steps — she never hangs, never loses your work, never lies (2026-07-06)
Every time Tlamatini talks to the model during a Multi-Turn run (step 6 above), that call goes through a **self-healing invoker** (`agent/self_healing.py`). If the model hiccups — a slow network, an overloaded server, an empty response — she does not freeze and she does not give up:
- **She never hangs.** Each attempt has a time limit (`unified_agent_llm_step_timeout_seconds`, 80 s by default); an attempt that runs over is abandoned, not waited on. She then tries again with a *different* tactic — plain retry, a short back-off, trimming the oldest messages, or a plain no-tools call — up to `unified_agent_llm_step_max_tactics` (4096) times. Only **you** (pressing Cancel) — or running out of tactics — stops her.
- **She never throws away work.** If she truly cannot reach the model but agents have already run, she finishes from that real work instead of discarding it, so your **Create Flow** button and **Exec report** still appear and the answer is honest about what actually got done.
- **She never lies about it.** However the run ends, a short recovery note is prepended to the answer telling you exactly what she went through, and while she is retrying you see live status messages appear in the chat.
### The Ask-Execs permission round-trip (step 6, expanded)
The Multi-Turn executor is **synchronous** and runs in a worker thread (`sync_to_async(ask_rag, thread_sensitive=False)`), so it cannot `await` a WebSocket reply directly. `agent/exec_permission.py::ExecPermissionBroker` bridges the gap:
1. The consumer registers one broker per request, keyed by user id, before invoking the chain (and tears it down in a `finally`).
2. Before a state-changing tool runs, the executor calls `broker.request_permission(detail)`, which emits an `exec_permission_request` frame onto the consumer's event loop (`asyncio.run_coroutine_threadsafe`) and then **blocks on a `threading.Event`**.
3. The browser shows the modal and replies with an `exec-permission-response` frame; the consumer routes it through `resolve_permission(user_id, request_id, decision)`, which sets the event and unblocks the executor.
4. **Proceed** → the tool runs. **Deny** → `_exec_denied` is recorded and the executor returns immediately, halting the chain; the denial detail flows back through the chain → `interface.ask_rag` (`global_state['last_exec_report_denied']`) → the consumer → `response_parser` renders the red banner.
The round-trip is **fail-safe**: an emit failure, a mid-flight Cancel, or a broker `close()` (e.g. the browser disconnected) all resolve to **Deny**, so an unconfirmed state-changing tool never runs. The wait loop polls `cancel_generation` on a short tick so a Cancel never deadlocks the worker thread. Read-only / polling tools are exempt from the prompt (they only observe).
The capability-aware selector scores each tool with name match (+14 exact), alias / hint phrase match (+10–12), example-request token overlap (up to +3), description token overlap (up to +10), plus a +15 history-aware boost on short follow-ups (≤4 meaningful tokens). The cap is 20 tools per request by default — lowered from 50 after observing keyword inflation pulled in everything.
## 36. ACPX runtime mechanics
ACPX is a Python port of OpenClaw's ACPX plugin. The `agent_id` mapping, `permissionMode` vocabulary, and SKILL.md frontmatter contract match OpenClaw verbatim.
### Three transport profiles
| Transport | Used by | Drain mechanic |
|---|---|---|
| `oneshot-prompt` | claude, codex, cursor, gemini, qwen | Re-spawn the CLI fresh per turn with the prompt as a CLI arg (`claude -p "<task>"`, `codex exec "<task>"`, …). Close stdin. `proc.communicate(timeout=180)` captures stdout to EOF. **The only transport that reliably captures TUI agents on Windows** — long-lived stdin-fed children only capture the outbound prompt because TUI CLIs detect a piped stdout and refuse to flush. |
| `json-acp` | tlamatini self-host | Strict ACP envelope: drain until `{"done": true}` line. |
| `tui-repl` | kiro, kimi, iflow, kilocode, opencode, pi, droid, copilot | Long-lived REPL. Drain via the **transport-aware idle rule**: armed after `startup_grace + idle_seconds` even with zero events (a silent TUI is, by definition, finished). |
### The idle-rule fix
Before the redesign, every TUI spawn burned the full 45 s timeout because the idle rule required `event_count > 0`, but TUIs almost never emit stdout when fed JSON on a piped stdin (Node block-buffers stdout to 64 KB when piped). The transport-aware idle rule fires on `tui-repl` even with zero events. **Per-leg latency dropped from ~91 s to ~9 s.**
### Per-event payload trimming
Every event-returning tool caps each event body at `max_event_chars` (default 2048) before returning it to the LLM. A chatty REPL paste-back of a long document cannot blow the LLM's context budget on the next iteration. Trimmed events carry `_truncated: true`.
### Permission gating
| Mode | What it does |
|---|---|
| `approve-reads` (default) | Reads auto-approved; writes / shell / network / db need a prompt. |
| `approve-all` | Auto-approves everything. **Flagged DANGEROUS.** |
| `deny-all` | Lockdown — even `acp_spawn` returns `PERMISSION_DENIED`. `acp_doctor` still works. |
Plus a non-interactive policy (`deny` / `fail`) for unattended runs.
### Persisted artifacts on disk
| Path | What |
|---|---|
| `<stateDir>/<session>.json` | `AcpSessionRecord` (session_id, agent_id, cwd, paths, pid, timestamps, closed). |
| `<stateDir>/<session>.transcript.ndjson` | Per-session full transcript: one `{"direction":"out", "text":..., "ts":...}` line per outbound, one `{"direction":"in", "raw":..., "ts":...}` per inbound. |
| `~/.tlamatini/skill-audit/<YYYY-MM>/<epoch>_<skill>_<id8>.ndjson` | Per-`invoke_skill` audit. Append-only. Replayable byte-for-byte. |
### ACPX visual canvas counterpart
The **ACPXer** workflow agent is the canvas counterpart of the 12 LLM-facing tools. One ACPXer node = one ACPX session lifecycle. It mirrors the runtime mechanics inline (in ~120 lines) because workflow agents in the pool run as separate Python subprocesses and cannot import `agent.acpx`. The transcript format is byte-identical, so an ACPXer transcript is interchangeable with a chat-driven one.
## 37. Database models
13 models in `agent/models.py`. The ones that matter day-to-day:
| Model | Purpose |
|---|---|
| `Agent` | Type registry — one row per agent type in the sidebar (`idAgent`, `agentName`, `agentDescription`, `agentContent`). |
| `Mcp` | UI toggle rows for MCP context providers. |
| `Tool` | UI toggle rows for unified-agent tools. |
| `ChatHistory` | Chat messages, with per-user isolation via `conversation_user`. |
| `AgentMessage` | Per-message records, `conversation_user` foreign key for user isolation. It is the ONLY place the LLM's conversation memory lives (re-read before every answer), which is why the chat card's **Drop** deletes one row and needs no reconnect. |
| `AcpAgent` | Mirrors `agent_registry.py + config.json` overrides. Reconciled by `service.boot_acpx()` on every start. |
| `Skill` | Mirrors `SKILL.md` packages. Reconciled by `service.boot_skills()`. |
| `AcpSession` | One row per real (or recently-real) ACP child. |
| `SkillInvocation` | Append-only audit row per `invoke_skill` call. |
| `ChatAgentRun` | Per wrapped chat-agent run — run_id, status, runtime_dir, log_path, start/end times. |
| `SessionState` | Per-user session state (24-hour TTL). |
| `AgentProcess` | Tracked PIDs for canvas agents. Cleared on shutdown. |
| `Omission` | Secret-redaction patterns for context. |
## 38. The application log (`tlamatini.log`)
`Tlamatini/manage.py` defines a `_TeeStream` wrapper that replaces `sys.stdout` and `sys.stderr` **before Django initializes**. Every print, every Django logger (they all use `StreamHandler`), and every tool's stdout/stderr lands in both the console and a single file:
| Mode | Path |
|---|---|
| Source | `Tlamatini/tlamatini.log` (next to `manage.py`) |
| Frozen (PyInstaller) | Next to the executable |
Characteristics:
- **Truncated on every restart** (mode `'w'`).
- **No rotation, no size cap.** Long sessions grow unbounded — copy or rename before restart if you need history.
- **Stream-level**, upstream of Django's logging config — picks up `print()` calls and third-party stdout too.
- **HTTP GET noise filter**: successful `"GET /…" 200/304` lines are silenced (the per-poll runtime status pings flooded the log otherwise). Non-2xx/3xx GETs stay visible — real failures are still surfaced.
When debugging an issue, `tlamatini.log` is the first artifact to consult.
## 39. ASCII / box-drawing diagrams in chat
LLM-generated ASCII art / flowcharts / column layouts render in the chat with a fixed-width font and preserved whitespace. The LLM is instructed (rule 13 in `prompt.pmt`) to wrap diagrams in `BEGIN-DIAGRAM` / `END-DIAGRAM` markers. There is also auto-detection: any run of consecutive lines containing box-drawing characters (`│┃|─━┌┐└┘├┤┬┴┼╭╮╯╰`), arrow glyphs (`▲▼►◄→←↑↓`), or ASCII-art runs (`+`, `-`, `=`, `|`) is wrapped automatically. Both pipelines emit `<pre class="ascii-diagram">…</pre>` HTML.
The `v1.48.14` parser hardening makes that two-pass behavior lossless. Explicit diagrams are replaced by NUL-wrapped `DGRM_<n>` sentinels while Markdown is processed; if a later auto-detected diagram temporarily captures an earlier sentinel, restoration expands the nested placeholder to a fixed point. A Markdown thematic break such as `---` after a blank line stays a thematic break rather than becoming a false diagram. Any malformed residual sentinel or NUL byte is removed before HTML reaches the browser, so users never see a bare `DGRM_0` or an empty diagram box.
---
# Part VII — Configuration Reference
The main file is `Tlamatini/agent/config.json`.
| Mode | Resolution order |
|---|---|
| Source | `Tlamatini/agent/config.json` |
| Frozen | `<install-dir>/config.json` next to the executable |
| Both | `CONFIG_PATH` env var, if set, wins over both |
## 40. LLM settings
```json
{
"embeding-model": "Nomic-Embed-Text:latest",
"chained-model": "deepseek-v4.1-flash:cloud",
"ollama_base_url": "http://127.0.0.1:11434",
"ollama_token": "",
"ANTHROPIC_API_KEY": "config you api key here by claude",
"GEMINI_API_KEY": "config your api key here by gemini",
"enable_unified_agent": true,
"unified_agent_model": "deepseek-v4.1-flash:cloud",
"unified_agent_base_url": "http://127.0.0.1:11434",
"unified_agent_temperature": 0.0,
"unified_agent_max_iterations": 4096,
"chat_agent_limit_runs": 100
}
```
| Key | What it does |
|---|---|
| `embeding-model` | RAG embedding model. |
| `chained-model` | Primary chat model. |
| `unified_agent_model` | Multi-Turn tool-loop model. Can differ from `chained-model`. |
| `unified_agent_max_iterations` | Hard cap on the tool loop. Default 4096. |
| `unified_agent_temperature` | 0.0 for deterministic. |
| `ollama_token` | Bearer token for authenticated remote Ollama. |
| `ANTHROPIC_API_KEY` / `GEMINI_API_KEY` | Top-level keys for Tlamatini's own cloud paths (image analysis, Opus client). |
| `enable_unified_agent` | Master switch for the tool-calling chain. |
| `chat_agent_limit_runs` | Wrapped-run listing cap. |
| `kali_server_url` | Base URL of the MCP-Kali-Server (`server.py`) on your Kali box. Tlamatini is the **embedded client** — the `chat_agent_kalier` tool auto-injects this as the default `server_url` on every run, so chat pentest prompts never repeat the address (the LLM may still override per-call). Default `http://127.0.0.1:5000` (works for WSL2 localhost-forwarding or an SSH tunnel); editable via `Config -> URLs`. |
You can still edit `config.json` by hand, but you no longer have to for the common cases. The chat navbar's `Config -> Models` dialog submits all 38 registered model, engine and voice settings, including values on hidden tabs. `Config -> URLs` writes the endpoint / host / port subset. The backend validates before `config_loader.save_config_updates()` atomically merges the submitted section into the active source/frozen `config.json`, preserving unrelated keys. The model API requires the complete 38-key object, not a one-key patch. Reconnect chat after changed client settings; already running agents must reload/restart.
### Model settings, inheritance and runtime paths
The complete [38-field reference](docs/model_configuration.md#complete-setting-reference)
lists every global key, initial default, local/provider/Ollama kind, fallback and
agent YAML path. Core and Vision each have 7 settings, Speech 7, Workflows 6,
Documents 5, Monitoring & messaging 6. Search spans the categories; engine and voice
fields are selectors. There are 21 model-backed agent types, not 38 separate agents.
Missing registered agent fields and quoted `"@config"` inherit central settings.
Literal canvas values remain overrides. Wrapped chat starts from globals and then
applies explicit tool arguments; standalone MCP preserves literal template choices
and applies invocation overrides last. Parametrizer and FlowCreator preserve these
strings. Saving never rewrites old workflows or downloads weights.
Talker requires Orpheus-compatible audio tokens and one of its five voices.
Whisperer offers `faster-whisper`, `cloud-groq`, `cloud-openai`; an empty cloud model
uses its provider default, while cleanup remains a separately enabled stage.
Video-Analyzer's `transcription.model` is local and independent of Whisperer's
engine. Empty LaTeXer repair model disables that stage. Credentials, hosts, audio
devices, sampling controls and external ACPX/MCP provider settings remain separate.
Source configuration lives at `Tlamatini/agent/config.json`; frozen configuration
lives beside the executable. `CONFIG_PATH` overrides either. Frozen services import
the compiled registry and copy portable helpers from the resolved agents root,
never a synthetic `_internal` source path. The offline `check_agent_runtimes` gate
executes in both modes and is mandatory before frozen packaging; a source snapshot
is not needed for ordinary execution. [Dated verification](docs/model-configuration-verification.md)
covers 89 runtime preparations, 21 loaders and actual File-Creator execution.
Any failed image/video observer or merger rejects that attempt and accumulates a
visible fatal error. Tlamatini continues its configured recovery routes and existing
retry tactics with the same model identities. A retired HTTP 410 model requires an
explicit user configuration change; do not substitute a survivor or raw notes.
Robotics still requires two explicit independent passes for `PASS_OK`.
## 41. RAG settings
```json
{
"chunk_size": 3000,
"chunk_overlap": 800,
"max_chunks_per_file": 50,
"k_vector": 100,
"k_bm25": 100,
"k_fused": 150,
"enable_bm25": true,
"rrf_k": 60,
"fetch_k": 300,
"max_doc_chars": 150000,
"max_context_chars": 250000,
"context_budget_allocation": {
"high_relevance": 0.60,
"architecture": 0.20,
"related": 0.15,
"documentation": 0.05
},
"use_llm_extractor": true,
"use_long_context_reorder": true,
"metadata_extraction": {
"enable_code_structure": true,
"enable_file_role_classification": true,
"enable_dependency_tracking": true,
"enable_cross_references": true
},
"retrieval_strategy": {
"enable_multi_stage": false,
"enable_query_expansion": true,
"enable_hierarchical_context": true,
"enable_context_budget_allocation": true
}
}
```
## 42. Internet search settings
```json
{
"internet_classifier_model": "deepseek-v3.2:cloud",
"internet_classifier_max_iterations": 4,
"internet_hint_words_mode": "extend",
"internet_hint_words": [],
"web_summarizer_model": "deepseek-v3.2:cloud",
"web_context_max_chars": 10000
}
```
## 43. MCP services
```json
{
"mcp_system_server_host": "127.0.0.1",
"mcp_system_server_port": 8765,
"mcp_files_search_server_port": 50051,
"mcp_files_search_server_max_workers": 10,
"mcp_files_search_model": "deepseek-v3.2:cloud",
"max_lines_search_files": 1024
}
```
### 43.1. External MCP defaults and private runtimes
The **External ▸ MCPs** dialog is the operator surface for universal MCP connections. Tlamatini speaks stdio, Streamable HTTP, SSE, and WebSocket transports, keeps at most five catalog servers active, and exposes ten built-in supervisor tools for import, activation, diagnostics, calls, waits, and runtime management.
Every installation contains two official catalog entries defined in `agent/external_mcp_defaults.py`:
| Default | Purpose | Initial state |
|---|---|---|
| `memory` | Persistent knowledge graph from `@modelcontextprotocol/server-memory`; its data file is `%LOCALAPPDATA%\Tlamatini\memory\memory.json`. | Inactive |
| `sequential-thinking` | Structured, revisable step-by-step reasoning from `@modelcontextprotocol/server-sequential-thinking`. | Inactive |
Neither default is activated without the user. Existing edits win over shipped defaults, and deleting a default writes its key to `_removed_defaults` so a later launch does not resurrect it. Re-importing that server explicitly clears the tombstone. This seeding happens in code when `load_catalog()` reads the user catalog because `external_mcps.json` is preserved across self-updates.
Most community MCP servers launch through `npx` or `uvx`. `agent/runtime_provisioner.py` can acquire Node/npm/npx/pnpm and uv/uvx once into `%LOCALAPPDATA%\Tlamatini\runtimes`: no administrator rights, no system `PATH` mutation, and no multi-gigabyte runtime bundle in the installer. The operation is fail-open, backgrounded during startup, downloaded from official upstreams, checksum-verified where upstream publishes checksums, and installed through an atomic partial-directory swap. The runtime strip in the dialog reports each tool's location; click **Install now** for an explicit installation, or simply activate an npx/uvx server and let its background connection provision what is missing. Multi-Turn can inspect or initiate the same path through `external_mcp_runtime_status` and `external_mcp_runtime_install`.
Public and private builds deliberately differ. A normal public build generates a secret-free catalog containing only the two inactive defaults and aborts if a live-looking credential reaches a secret-shaped environment field. A keyed/private build may include the maintainer catalog after `regen_secrets.py` has rehydrated it. Never hand-copy a private `external_mcps.json` into a public release.
### 43.2. External MCP Adder skill
The `adding-external-mcp` skill is the supported guided path for bringing a new server into Tlamatini. It keeps setup deterministic and prevents a configuration record from being mistaken for a healthy connection:
1. Classify the server as stdio, Streamable HTTP, SSE, or WebSocket from its official launch instructions.
2. Build a secret-separated `mcpServers` entry; never place a live credential in a public catalog or generated document.
3. Import the entry through `external_mcp_import` instead of editing user state blindly.
4. Run `external_mcp_doctor` before the first activation so transport, runtime, endpoint, and placeholder-secret failures are named.
5. Activate only with operator intent and keep the global active-server cap at five.
6. Wait for a healthy connection with `external_mcp_wait`; catalog presence alone is not readiness.
7. Inspect `external_mcp_status` and `external_mcp_list_tools` before selecting a remote operation.
8. Call the proven remote tool through `external_mcp_call`, or let Multi-Turn bind its `ext__<server>__<tool>` wrapper lazily.
The append-only Deep Internet Research starter prompt (migration `0194`, prompt 118, Getting Started, `sort_rank=100`) demonstrates this ecosystem at full scale. It requests a long, link-rich search and explicitly selects Multi-Turn plus Exec Report; Sequential Thinking, Memory, External MCPs, and workflow agents remain tools the operator can authorize rather than hidden prerequisites.
## 44. ACPX settings
The whole `acpx` block is **optional**. When missing or partial, every value falls back to a safe default. On first boot of an upgrade build, `agent/acpx/service.py::boot_acpx()` calls `ensure_acpx_block_in_config_json()` and **appends the documented default block to your existing `config.json` atomically**.
```json
{
"acpx": {
"cwd": "C:/Development/Tlamatini",
"stateDir": "C:/Users/angel/.tlamatini/acpx-state",
"probeAgent": "gemini",
"permissionMode": "approve-reads",
"nonInteractivePermissions": "deny",
"timeoutSeconds": 180,
"agents": {
"claude": {
"command": "C:/Users/angel/AppData/Roaming/npm/claude.cmd",
"env": { "ANTHROPIC_API_KEY": "sk-ant-api03-..." }
},
"gemini": {
"command": "C:/Users/angel/AppData/Roaming/npm/gemini.cmd",
"env": {
"GEMINI_API_KEY": "AIza...",
"GOOGLE_API_KEY": "AIza..."
}
},
"qwen": { "command": "qwen-code", "env": { "DASHSCOPE_API_KEY": "sk-..." } },
"codex": { "command": "codex", "env": { "OPENAI_API_KEY": "sk-..." } }
}
}
}
```
| Key | Default | Description |
|---|---|---|
| `cwd` | Django process cwd | Default working directory used when `acp_spawn` is called without `cwd`. |
| `stateDir` | `~/.tlamatini/acpx-state/` | Directory for `<session>.json` and `<session>.transcript.ndjson`. |
| `probeAgent` | first resolvable | `agent_id` used by `acp_doctor`'s `--version` probe. |
| `permissionMode` | `approve-reads` | One of `approve-reads`, `approve-all` (DANGEROUS), `deny-all`. |
| `nonInteractivePermissions` | `deny` | What to do when a permission prompt fires unattended. `deny` or `fail`. |
| `timeoutSeconds` | 120 | Per-turn timeout for the embedded runtime. |
| `agents` | `{}` | Per-`agent_id` command + `env` overrides on top of the built-in registry. |
### API key setup — two layers, in plain English
**Layer 1 — top-level config keys** (consumed by Tlamatini's own cloud calls):
```json
{ "ANTHROPIC_API_KEY": "sk-ant-...", "GEMINI_API_KEY": "AIza..." }
```
**Layer 2 — per-`agent_id` `env` injection** (consumed by the spawned ACP child):
```json
{
"acpx": {
"agents": {
"claude": { "env": { "ANTHROPIC_API_KEY": "sk-ant-..." } },
"gemini": { "env": { "GEMINI_API_KEY": "AIza...", "GOOGLE_API_KEY": "AIza..." } }
}
}
}
```
How the merge works at spawn time:
1. `build_agent_registry(overrides, env_overrides)` merges per-agent `env` on top of `DEFAULT_ACP_AGENTS[agent_id].env` (override wins on conflict).
2. `AcpSession.spawn_child()` builds the child env as `{**os.environ, **self.spec.env}` — parent process env first, then per-agent override on top. **Explicit `acpx.agents.<id>.env` value wins over an exported shell variable.**
Canonical env var per CLI:
| Agent | Env var(s) | Where to put it |
|---|---|---|
| `claude` | `ANTHROPIC_API_KEY` | Both layers (top-level for Tlamatini, per-agent for the spawned `claude`). |
| `gemini` | `GEMINI_API_KEY` (newer builds also accept `GOOGLE_API_KEY`) | Both layers; per-agent should set both names. |
| `qwen` | `DASHSCOPE_API_KEY` | `acpx.agents.qwen.env`. |
| `codex` | `OPENAI_API_KEY` | `acpx.agents.codex.env`. |
| `cursor` | (own credential store) | usually no env injection needed. |
| `copilot` | (`gh auth login`) | usually no env injection needed. |
| `pi`, `droid`, `iflow`, `kilocode`, `kimi`, `kiro`, `opencode` | per upstream | check each CLI's docs. |
> **Security warning.** `config.json` is tracked by git. Three safer patterns:
>
> 1. **Don't commit the keys.** Use the `regen_secrets.py` script:
> ```bash
> python regen_secrets.py --mode push-able # rewrite secrets to placeholders before commit
> python regen_secrets.py --mode keyed # restore from data.keys (gitignored) for local use
> ```
> Or run `git update-index --skip-worktree Tlamatini/agent/config.json` after configuring locally.
> 2. **Use shell env vars in development.** Export the keys; leave `acpx.agents.<id>.env` empty.
> 3. **Rotate immediately if a key has been pushed.** Any leaked key should be revoked at the provider before reuse.
### The `setup-new-acpx-key` skill (the easy button)
Instead of editing `config.json` by hand, in chat (Multi-Turn + ACPX ticked):
> "Use `invoke_skill` with `setup-new-acpx-key` to register my Anthropic key for the `claude` agent_id." (paste the key)
The skill walks itself through writing `data.keys`, patching both `config.json` layers, optionally extending `regen_secrets.py`, and verifying via `acp_doctor`.
## 45. Image interpreter
```json
{
"image_interpreter_base_url": "http://127.0.0.1:11434",
"image_interpreter_model": "jcyhsiao/qwen3.5cloud:latest",
"image_interpreter_temperature": 0
}
```
## 46. Advanced options
```json
{
"history_summary_enable": true,
"history_summary_trigger_tokens": 150,
"history_keep_last_turns": 3,
"performance": { "enable_caching": true, "cache_embeddings": true, "parallel_processing": true, "max_workers": 12 },
"logging": { "log_retrieval_metrics": true, "log_context_size": true, "log_query_rewrites": true, "verbose_metadata": true },
"load_hidden": true,
"ssl_verify": false,
"max_input_tokens": 5000,
"keep_last_turns": 3
}
```
---
# Part VIII — Deploying & Packaging
## 47. The three-step build pipeline
```
build.py ──► build_uninstaller.py ──► build_installer.py
│ │ │
▼ ▼ ▼
pkg.zip Uninstaller.exe dist/Tlamatini_Release/
(app bundle) (project root) ├─ Installer.exe
├─ Uninstaller.exe
├─ pkg.zip
└─ _internal/
```
The final distributable is `dist/Tlamatini_Release/` — zip the folder, share it.
### Step 1 — `build.py`
```bash
python build.py
```
Installs deps, runs `collectstatic`, executes PyInstaller, copies required payloads (including `README.md` and bundled `jd-cli/`), runs migrations, creates the default user (`user`/`changeme`), renames the exe to `Tlamatini.exe`, copies agent templates, bundles support scripts (`flow_file_associations.ps1`, `register_flw.ps1`, `register_fpmt.ps1`, `unregister_flw.ps1`, `unregister_fpmt.ps1`, `CreateShortcut.ps1`, `Tlamatini.ps1`, `.ico`), and zips it all into **`pkg.zip`**.
`build.py` is strict: missing `README.md`, missing `jd-cli/`, or missing `jd-cli.bat` causes a non-zero exit instead of a silent half-package.
### Step 2 — `build_uninstaller.py`
```bash
python build_uninstaller.py
```
Builds `uninstall.py` into a single `--onefile` Tkinter exe. Output: `Uninstaller.exe` at the project root.
### Step 3 — `build_installer.py`
```bash
python build_installer.py
```
Requires `pkg.zip` and `Uninstaller.exe` from steps 1 and 2. Builds `install.py` with `--onedir --windowed` and a splash screen, copies `pkg.zip` and `Uninstaller.exe` into `dist/Installer/`, and assembles `dist/Tlamatini_Release/` with SHA-256 verification.
### Private keyed build and contact synchronization
`build_complete_private_release.py` is the maintainer-only keyed path and must never be published. Before building, it synchronizes contact records from the development-tree `contacts.json` and a same-machine frozen installation into gitignored `contacts.private.json`: names are compared case- and accent-insensitively, aliases are unioned, and existing non-empty values win. That merged private file may be bundled only when the explicit private-build opt-in is set. Public builds still ship an empty contacts book, and `TlamatiniSourceCode/` never carries contact files or other contact PII.
## 48. Versioning
Tlamatini follows [Semantic Versioning 2.0.0](https://semver.org/) — `MAJOR.MINOR.PATCH` — but the **single source of truth is a git tag**, not a number sitting in any source file. You never hand-edit a version anywhere. You tag, then you build, and the three build scripts in §47 each bake the resolved value into the artefact they produce.
### What the three numbers mean
- **MAJOR** bumps when something that already shipped breaks for the user: the `.flw` file schema changes, an Agent Contract is removed, an LLM tool is renamed, a public endpoint URL changes. The first `2.0.0` is the first release where loading an old `.flw` might not just work.
- **MINOR** bumps when you add a backward-compatible feature: a new agent (ACPXer was a minor bump), a new toolbar checkbox, a new SKILL package, a new HTTP endpoint, a new optional field on an existing API.
- **PATCH** bumps for backward-compatible fixes: the conjunction-parser fix, the exec-report ordering fix, the ACPX `oneshot-prompt` capture fix — anything that closes a regression without changing surface.
Pre-releases use the standard SemVer suffixes — `2.0.0-alpha.1`, `2.0.0-beta.1`, `2.0.0-rc.1`. They sort **before** the final release, so `2.0.0-rc.2` < `2.0.0` for the Windows installer registry and for Python tooling alike.
### Cutting a release in five commands
```powershell
git status # clean tree, on main
git tag -a vMAJOR.MINOR.PATCH -m "Release MAJOR.MINOR.PATCH: <one-liner>" # annotated tag, an unused version (1.92.4 is taken)
git push origin vMAJOR.MINOR.PATCH
python build.py
python build_uninstaller.py
python build_installer.py
```
All three build scripts pick the tag up from `git describe --tags` automatically. The final artefact lands in `dist/Tlamatini_Release_v1.92.4/`, named for the version so the file you hand to a user is unambiguous before they even unzip it. The latest released version is `1.92.4`; pass `--version 1.92.4` to the complete wrappers to pin that number. Tlamatini deliberately never puts a `.devN`, `+gSHA` or `.dirty` suffix into its version string — it always reports the base tag, which is the designed behaviour described in `VERSIONING.md`. That is exactly why the five commands above start with `git status` on a clean tree: a release cut from a dirty tree would ship a version number that says nothing about what is actually inside it.
### Where the version shows up in a running install
The build computes the version once and bakes it into four surfaces:
- **`Tlamatini/agent/_version.py`** — generated at build time, gitignored, read at runtime by `agent.version.get_version()`. This is what every in-process surface reads.
- **Win32 `VERSIONINFO`** — `Tlamatini.exe`, `Installer.exe`, and `Uninstaller.exe` all carry the version in their resource fork. Right-click the file → Properties → Details → ProductVersion.
- **Release folder name** — `dist/Tlamatini_Release_v1.85.0/`.
- **Runtime surfaces** — the About dialog renders `Tlamatini v{{ version }}` (Django context processor); after the release tag/build, the startup banner prints `--- [VERSION] Tlamatini 1.85.0` to both the console and `tlamatini.log`; `GET /agent/version/` returns `{"version":"1.85.0","commit":"abc1234","date":"…","source":"generated"}` as an **open** endpoint suitable for a health-check.
If the four surfaces ever disagree, your build was run with a stale `$env:TLAMATINI_VERSION` or against an out-of-date `_version.py` — clear them and re-run `build.py`.
### What happens if you don't tag
The build never fails for "no version" — and the version surface is always a clean SemVer like `1.1.1`. The resolver returns the **bare base tag** reachable from HEAD; distance / commit / dirty state are deliberately stripped from the displayed version:
| Situation | Version baked in |
|---|---|
| Tag exists, HEAD exactly on `v1.2.0` | `1.2.0` |
| Tag exists, HEAD 17 commits past, clean tree | `1.2.0` |
| Tag exists, HEAD 17 commits past, uncommitted edits | `1.2.0` |
| No tags at all | `0.0.0` |
| Not a git repo (e.g. download zip) | `0.0.0+unknown` |
No `.devN`, no `+gSHA`, no `.dirty` ever appears in the version string. Distance from the tag and dirty state are git concerns and live in `git status` / `git describe --long --dirty`, not in the user-facing version.
### Overriding the resolver
There are four sources of the version, in precedence order:
1. `--version X.Y.Z` on the build script's command line (highest).
2. `$env:TLAMATINI_VERSION` exported in the shell.
3. `git describe --tags --abbrev=0 --match 'v[0-9]*'` against the working tree — the bare base tag, no distance/dirty suffix (the normal path).
4. The sentinel `0.0.0+unknown` (lowest — only fires when there is no git at all).
`build.py` exports `$env:TLAMATINI_VERSION` after it resolves, so `build_installer.py` and `build_uninstaller.py` in the same shell see exactly the same value — the three artefacts cannot disagree. Even on an untagged commit, the git-derived dev version stays consistent across all three.
The full contract — including the recovery path for a mis-tagged release, the runtime resolver internals, the file-by-file integration map, and the FAQ — lives in [`VERSIONING.md`](VERSIONING.md) at the repo root.
## 49. What the installer does
When an end user runs `Installer.exe`:
1. Opens the shared setup window (`setup_wizard.py`, since v1.82.0) to choose and review the installation directory; before it changes any file it prepares and verifies a recovery copy (`setup_recovery.py`, journal in `Temp/.setup-recovery`), restores an interrupted run on the next start, keeps errors on screen with **Retry** and **Change folder**, and logs to `%LOCALAPPDATA%\Tlamatini\SetupLogs`.
2. Extracts `pkg.zip` into `<install_path>/Tlamatini/`.
3. Locks agent venv permissions.
4. Writes `config.json` with installation settings.
5. Copies `Uninstaller.exe` into the install dir.
6. Creates desktop and Start Menu shortcuts (`Tlamatini.lnk`).
7. Registers `.flw` extension to open with Tlamatini.
8. Cleans the PyInstaller bundle path from helper subprocess environments so PowerShell helpers and Explorer restarts don't stall.
## 50. What the uninstaller does
1. Refuses to start while Tlamatini is still running from that folder (Retry or Exit — there is no "continue anyway"), then asks for confirmation.
2. Stops the workers that installation started, removes its shortcuts and its Installed Apps entry, and unregisters the `.flw` / `.fpmt` associations it owns (the shell is refreshed without restarting Explorer).
3. Deletes the application files **except**:
- `agents/` — always kept (your agents; companion apps such as Tlamatini-FlowPills keep reading it);
- `application/`, `applications/`, `content_generated/`, `context_files/`, `doc_generated/` and `Templates/` — kept whenever they hold at least one file. `Templates/` is where STM32er, ESP32er, Arduiner, ESPHomer, Unrealer and LaTeXer put the projects they create, so it holds your own code; it was missing from the list until 2026-10-09, when uninstalling `C:\Tlamatini` erased Angela's code.
- `Temp/` is scratch and is always erased.
4. The last screen names the folders it kept; files Windows still holds are removed right after the uninstaller closes.
5. Since v1.82.0 the removal runs inside the same recovery transaction (`setup_recovery.py`): a verified recovery copy is prepared before files are removed, a file that stays locked for five seconds is named (with the program holding it when Windows can tell, which is never closed for you), and the removal stops before deleting anything; Installed Apps is checked to be really gone, and only Tlamatini's own registry values and shortcut are touched (`setup_windows.py`).
⚠️ The uninstaller is not part of an in-app update: **About ▸ Check for updates** keeps the `Uninstaller.exe` already installed. A new Installer replaces it.
## 51. Frozen-mode behavior
The Multi-Turn implementation carries frozen-build awareness in supporting runtime code:
- `config_loader.py` resolves `CONFIG_PATH`, then executable-local `config.json`, then module-local.
- `FileSearchRAGChain` resolves its default `config.json` from the executable directory in frozen mode.
- Template-agent discovery checks both `<install_dir>/agents` and `<install_dir>/Tlamatini/agent/agents`.
- `_get_agents_root()` in `chat_agent_runtime.py` resolves from `sys.executable` in frozen mode, from `__file__` in source mode — both paths are logged at INFO level.
- `_resolve_python_executable()` tries `PYTHON_HOME`, then bundled `python.exe` beside the frozen executable, then PATH.
---
# Part IX — The Command Deck (API + WebSocket)
## 52. WebSocket protocol
Endpoint: `ws://<host>/ws/agent/`. (A second, separate endpoint — `ws://<host>/ws/prompt-flow-panel/` — carries the Prompt Flow Panel's run commands: `start`, `pause`, `resume`, `stop`, `reply` and a `ping` heartbeat. It serves one run per connection and refuses anonymous sockets; see the chapter *The Prompt Flow Panel* near the front of this book.)
### Direct voice input: `/ws/chat-voice/`
This authenticated same-origin WebSocket prepares a resident Whisperer worker without opening the microphone. Client messages are `{"action":"start","run_id":"unique-id"}` and `{"action":"cancel","run_id":"unique-id"}`. The server reports preparing/ready, starting, real recording telemetry, transcribing and a terminal result/empty/error/cancelled event; cancellation can first report cancelling. It rejects anonymous or foreign-origin sockets and overlapping commands. Text results return to the browser, which submits once through `/ws/agent/` with the current options. No raw audio travels through this browser socket and no new public port is introduced. Full frame and size limits: [microphone protocol](docs/chat-microphone-design.md#transport-and-ownership).
### Client → Server (chat)
```json
{
"message": "Your question here",
"multi_turn_enabled": true,
"exec_report_enabled": true,
"acpx_enabled": true
}
```
Optional toggles. `multi_turn_enabled=false` falls back to legacy one-shot.
### Client → Server (control)
| Type | Purpose |
|---|---|
| `set-canvas-as-context` | Use the current canvas file as context |
| `unset-canvas-as-context` | Remove the canvas file from context |
| `set-directory-as-context` | Load a directory as context |
| `set-file-as-context` | Load a single file as context |
| `cancel-current` | Cancel the current generation |
| `reconnect-llm-agent` | Rebuild the current LLM/RAG chain |
| `clean-history-and-reconnect` | Clear chat history and rebuild |
| `drop-message` | Delete ONE saved message (`message_id`) of this user's conversation — the card's **Drop** button; no rebuild. Answered with a `message-dropped` frame (`ok`, `message_ids`, or a `reason`) sent to every open tab |
| `clear-context` | Remove persisted context and rebuild |
| `cancel-all` | Cancel all active generation |
| `save-files-from-db` | Persist canvas / DB-backed files |
| `enable-llm-internet-access` | Enable internet access for the LLM |
| `disable-llm-internet-access` | Disable internet access for the LLM |
| `view-context-dir-in-canvas` | Show the current context directory tree in the canvas |
| `set-file-omissions` | Update file omission patterns |
| `set-mcps` | Persist MCP enablement |
| `set-tools` | Persist tool enablement |
| `set-agents` | Persist agent enablement |
### Server → Client
```json
{ "message": "Processing request...", "username": "Tlamatini" }
```
```json
{ "type": "session-restored", "context_type": "directory", "context_path": "/path/to/project" }
```
A Multi-Turn message also carries `tool_calls_log` and `multi_turn_used`. The Create Flow button appears whenever ≥1 agent in that log executed successfully; the old `answer_success` classifier flag was removed 2026-07-06.
A message that was SAVED to the conversation (your prompt, her answer) also carries `message_id`, the row's id; the card keeps it so its **Drop** button can name the row. Live status lines carry none. The first page render passes the same id as `id` in `initial_messages`.
## 53. HTTP endpoints
The backend currently exposes 104 routes. Highlights:
### Pages
| Endpoint | Method |
|---|---|
| `/` | GET/POST (login) |
| `/welcome/` | GET |
| `/agent/` | GET (chat) |
| `/agentic_control_panel/` | GET (designer) |
| `/logout/` | GET |
### Data loading
| Endpoint | Method |
|---|---|
| `/load_canvas/<filename>/` | GET |
| `/load_prompt/<prompt_name>/` | GET |
| `/load_omissions/<omission_name>/` | GET |
| `/load_mcp/<mcp_name>/` | GET |
| `/load_tool/<tool_name>/` | GET |
| `/load_agent/<agent_name>/` | GET |
| `/load_agent_description/<agent_name>/` | GET |
| `/load_agent_config/<agent_name>/` | GET |
### Agent management
| Endpoint | Method |
|---|---|
| `/save_agent_config/<agent_name>/` | POST |
| `/deploy_agent_template/<agent_name>/` | POST |
| `/ensure_agent_exists/<agent_name>/` | GET |
| `/execute_starter_agent/<agent_name>/` | POST |
| `/execute_ender_agent/<agent_name>/` | POST |
| `/check_starter_log/<agent_name>/` | GET |
| `/check_ender_log/<agent_name>/` | GET |
| `/check_agents_running/<agent_name>/` | GET |
| `/check_all_agents_status/` | GET |
| `/read_agent_log/<agent_name>/` | GET |
| `/restart_agent/<agent_name>/` | POST |
| `/restart_agents/` | POST |
| `/asker_choice/<agent_name>/` | POST |
| `/execute_flowhypervisor/<agent_name>/` | POST |
| `/check_flowhypervisor_alert/<agent_name>/` | GET |
| `/validate_flow/` | GET |
### Flow Compiler & Agent Contracts (since commit `0bea21d`, May 2026)
| Endpoint | Method | Notes |
|---|---|---|
| `/agent/compile_flow/` | POST | Backend Flow Compiler. Body: `{ "mode": "dry_run"\|"write", "flow": <ACP snapshot> }`. Save / Validate use `dry_run`; Start uses `write` to materialize `config.yaml` and `interconnection-scheme.csv` into the session pool. |
| `/agent/flow_from_tool_calls/` | POST | Chat Create-Flow normalizer. Body: `{ "tool_calls_log": [...], "flow_data": <legacy draft> }`. Returns a registry-canonical, secret-redacted `.flw` JSON. |
| `/agent/agent_contracts/` | GET | Returns the live `AgentContract` registry summary — connection-field shape, parametrizer source-fields, secret paths, singleton/long-running/never-starts-targets/excluded-from-validation flags. Used for diagnostics and for any out-of-tree client (e.g. a future MCP server) that needs to introspect the agent surface. |
### Connection updates (canvas auto-configuration)
`/update_<agent>_connection/<agent_name>/` — 83 live connection endpoints (2026-10-19), one per agent type that saves its canvas wiring the moment a wire is drawn: Raiser, Emailer, Monitor-Log, Ender, Starter, Whatsapper, Croner, Mover, Sleeper, Cleaner, Deleter, Notifier, Executer, Pythonxer, Stopper, RecMailer, Asker, Forker, Counter, Shoter, Camcorder, Globber, Grepper, Editor, Recorder, Whisperer, AudioPlayer, VideoPlayer, Talker, SSHer, SCPer, TeleTlamatini, ACPXer, Telegrammer, SQLer, Prompter, Gitter, Dockerer, MCP Doctor, Instant Messaging Doctor, PSer, Kuberneter, Apirer, Unrealer, Blenderer, Playwrighter, Reviewer, Analyzer, Jenkinser, Crawler, Summarizer, FlowHypervisor, Mouser, Windower, Kalier, Discoverer, Nmapper, PDFer, LaTeXer, PPTXer, STM32er, ESP32er, ESPHomer, Arduiner, File-Interpreter, Image-Interpreter, Video-Analyzer, NetSpeed-Calculator, File-Creator, File-Extractor, Kyber-KeyGen, Kyber-Cipher, Kyber-DeCipher, Parametrizer, FlowBacker, Barrier, J-Decompiler, De-Compresser, Keyboarder, Googler, OR and AND. FlowCreator has no connections; Mongoxer and Monitor Netstat have no live endpoint, and their wiring is written by the Flow Compiler when the flow is validated or started.
Plus the Parametrizer-specific pair:
| Endpoint | Method |
|---|---|
| `/get_parametrizer_dialog_data/<agent_name>/` | GET |
| `/save_parametrizer_scheme/<agent_name>/` | POST |
### Session & pool
| Endpoint | Method |
|---|---|
| `/session_state/` | GET |
| `/save_session_state/` | POST |
| `/clear_session_state/` | POST |
| `/clear_pool/` | POST |
| `/cleanup_session/` | POST |
| `/clear_agent_logs/` | POST |
| `/clear_pos_files/` | POST |
| `/reanimate_agents/` | POST |
| `/save_paused_agents/` | POST |
| `/load_paused_agents/` | GET |
| `/delete_paused_agents/` | POST |
| `/delete_agent_pool_dir/<agent_name>/` | POST |
| `/get_session_running_processes/` | GET |
| `/kill_session_processes/` | POST |
### Open in… external editors
| Endpoint | Method |
|---|---|
| `/agent/detect_installed_apps/` | GET — returns which of File Explorer / VS Code / Antigravity are installed |
| `/agent/open_in_app/` | POST — accepts `app_id` plus `directory` or `agent_name`; resolves the current session pool instance directory |
---
### Chat image ingest (screenshot paste / file drop)
| Endpoint | Method |
|---|---|
| `/agent/paste_image/` | POST — multipart `image` field. Re-encodes the clipboard bitmap to JPEG (Pillow; alpha flattened onto white; 25 MB cap)