RFC-0006-TEMPORAL-DESIGN-PATTERNS.md
Rama main · réplica de lectura del repositorio original.
# RFC-0006 — Temporal Design Patterns for OpenCompany
> **Current-state reading rule:** defect lists, measurements and code sketches below are anchored to their cited baseline commits. Some fixes have since landed. Use the [documentation audit](docs-internal/DOCUMENTATION_AUDIT.md), current runtime references and recorded generation flags for shipped behavior; proposed paths and later rollout phases remain designs until implemented. This audit does not change the accepted owner decisions.
- **Status:** Draft, 2026-10-24. Owner decisions T1-T20 are in §3.1. The amendments to RFC-0005 listed in §20 and the corrections to RFC-0004 listed in §21 were made in the same commit. On 2026-10-25 the Phase 1b plan was simplified (§22.2), and RFC-0007 amended §9.6 and §22.5. Phase 1b's docs pass (1b.45, 2026-11-02 and 2026-11-03) added notes wherever Phase 1b built something differently. Fixes after Phase 1b (2026-11-04 and 2026-11-05) are listed in §22.2's Done list and RFC-0005 §24.4.
- **Created:** 2026-10-24
- **Scope:**
- every Temporal workflow and activity in `server/services/temporal/` and `server/nodes/scheduler/cron_scheduler/`;
- the code that starts, signals, updates and cancels them: `server/services/deployment/`, `server/services/events/`, `server/services/chat/`, `server/services/approvals/`, `server/services/employees/`;
- the plugin framework's activity behaviour: `server/services/plugin/`, and the nodes whose behaviour depends on Temporal semantics (Timer, approvalGate, every sending node);
- worker topology and configuration (`worker.py`, the standalone worker), and the replay and identity tests in `server/tests/temporal/`.
- **Relationship:**
- Complements RFC-0004 (using the SDK and platform fully). It covers what RFC-0004 skipped: a pattern-by-pattern review of Temporal's design-pattern catalogue, AI cookbook, best practices, Python samples and Nexus pages. §21 corrects the RFC-0004 statements that verification found wrong or that a decision superseded.
- Amends RFC-0005 (the event OS) wherever a pattern or an owner decision changes a phase's design (§20). RFC-0005 stays the plan of record; this RFC adds detail and fixes gaps.
- Adds Phase 1b to RFC-0005's rollout: fixes that need no workflow versioning and should land before Phase 2 (§22).
- **Baseline:** HEAD 4f73cb10 (2026-10-24), temporalio 1.30.0, Temporal CLI pinned to v1.9.1 (server 1.32.0) for new installs. Line numbers are as of that commit and drift.
- **Sources:** docs.temporal.io as of 2026-10-24 and temporalio/samples-python at 33f7dc29 (2026-10-22). See §2.
<!-- architecture-suite:start -->
## Architecture diagrams
Source baseline: `56f0b397`. [Complete diagram suite](docs/diagrams/architecture/README.md).
**Event OS control plane and resource lifecycle — Proposed target · accepted roadmap.**
[](docs/diagrams/architecture/14-event-os-control-plane-v1.png)
[Editable Mermaid](docs/diagrams/architecture/14-event-os-control-plane.mmd) · [Source evidence and limitations](docs/diagrams/architecture/14-event-os-control-plane.brief.md)
<!-- architecture-suite:end -->
## How to read this
- **Owner:** §1 (summary), §3 (decisions), §24 (defaults chosen and what is still open).
- **Reviewer:** §4 (the catalogue at a glance), §5 (rules every workflow follows), and §20 (what changes in RFC-0005).
- **Implementer:** §6 to §19, one section per pattern category or source. Each pattern has the same parts: what the docs prescribe, SDK support, what OpenCompany does today with file and line, a verdict, recommendations with their versioning and phase, remarks on pitfalls, and a Python sketch where something is missing or misused. Then §22 (rollout) and §23 (acceptance gates).
- **Quotations of RFC-0005:** where an entry in §6 to §19 describes "the RFC-0005 plan", it quotes RFC-0005 as reviewed, before the amendments in §20.
Verdicts use six words:
- **Adopted correctly:** the code follows the pattern.
- **Adopted with deviations:** the code follows it with gaps, listed.
- **Misused:** the code uses the pattern's mechanism in a way the docs warn against, or that breaks a rule OpenCompany has set.
- **Missing and beneficial:** OpenCompany should adopt it.
- **Planned in RFC-0005:** RFC-0005 already designs it, and this RFC says whether that design follows the docs.
- **Not applicable:** it solves a problem OpenCompany does not have, with the reason.
Priorities:
- **P1:** fix before Phase 2: in Phase 1b, or in Phase 1 for T17's fairness and partition. These fixes need no workflow versioning.
- **P2:** fix with the RFC-0005 phase named.
- **P3:** later or never.
## 1. Summary
Temporal publishes a catalogue of design patterns for durable workflows: orchestration, messaging, entities, external waits, transactions, retries, batches, quality of service, latency and worker configuration. It also publishes an AI cookbook, best practices, Python samples and the Nexus pages. This RFC checks OpenCompany against every one of them, page by page, and says what to change.
**What we found.** OpenCompany's Temporal code follows the documented patterns in shape. Of the 45 design-pattern entries:
- 26 are adopted with deviations;
- 4 are misused;
- 5 are missing and would help;
- 3 are already planned in RFC-0005;
- 7 do not apply.
None is adopted with no deviation at all. The deviations sit exactly where the pages put their warnings: what happens on a cancel, a retry, a restart or a long wait. Verified examples:
- **Side effects can happen twice.** Every message a graph node sends retries up to three times with no guard, including the reply after the approval gate in every employee that Hire builds. Tool calls such as an HTTP POST, a GitHub pull request or a Vercel deploy run again after a broken-off attempt. Both contradict E12. (§11.2)
- **Agent children are terminated, not cancelled.** Four agent-child starts take the default parent close policy, so a parent that closes early kills its agents without their cleanup. Browser claims, permits and job rows are left behind. (§6.2)
- **A bug in workflow code makes its own fix fail replay.** Eight workflow types run cleanup activities on any exception, bugs included, and three swallow bugs per event. (§18.2)
- **Waits do not survive restarts.** The Timer node sleeps inside an activity, and a graceful shutdown fails it. Reviewed team-job delivery strands its draft when a retry lands within its lease. The approval gate polls inside a 24-hour activity. (§9.5, §9.6)
- **Some calls have no deadline:** every client-side Update call, a streamed LLM call (the 600 s client timeout is per read), and every Gemini call. (§7.3, §16.3)
- **Some failures are fatal or silent.**
- The controller fails within seconds to about 20 seconds of database trouble while it holds a backlog.
- Compensations give up after 3 or 10 attempts (until 1b.20 made them retry without limit).
- The Start path carries saved credentials into Temporal input.
- No Temporal metric is exported, so no alert can fire. (§12.3, §10.2, §17.8, §17.10)
**What changes.** The owner settled twenty questions on 2026-10-24 (§3.1). In short:
- **Fresh start confirmed.** Old Temporal databases are deleted at the pinned release (T5).
- **Polls become retried activities** (T7).
- **Owner chat starts at once,** with no debounce (T9).
- **Owner actions write an intent** and are delivered in the background (T10).
- **Stuck turns park** instead of failing (T11).
- **Stop does not wait for drafts** (T12).
- **Side effects are redone only when idempotent** (T13).
- **Agent transcripts stay in history** (T6).
- **Hosted instances run workers in a second process** (T8).
- **Fairness ships with the fresh start** (T17).
- **Lost supervisors are rebuilt in place** (T20).
**How it ships.**
- **Phase 1b** (§22.2) is a new step before RFC-0005's Phase 2: fixes that need no workflow versioning (each leaves the replayed command sequence unchanged), one commit each, simplified on 2026-10-25, with RFC-0007's approvals track inside it. Side effects come first, then lifecycle, compensation, retries, platform, and tests and docs.
- **Design-level changes** land in the RFC-0005 phases they belong to (§22.3). RFC-0005 was amended in the same commit (§20), and 34 RFC-0004 statements that verification found wrong or that a decision superseded were corrected (§21).
- **What does not apply:** Nexus, eager workflow start, delayed start, asynchronous completion and MapReduce trees, each for a stated reason (§19, §14.4, §9.4, §18.15, §12.5).
## 2. Method and sources
- **Docs** (all fetched as raw markdown on 2026-10-24 by appending `.md` to each page URL):
- the design-pattern catalogue (https://docs.temporal.io/design-patterns): the index, 10 category overviews and 35 patterns, 46 pages. The same pages existed on 2026-10-16; only the Saga page changed since.
- the AI section and its cookbook, 19 pages;
- the best-practices section (13 pages, four of them new since 2026-10-16) and the Python SDK best practices (6 pages);
- the Nexus section and the standalone activity and operation pages, 15 pages.
- **Code samples:** temporalio/samples-python at commit 33f7dc29 (2026-10-22), 57 sample folders. It targets temporalio 1.34.0 or later; every call it makes was checked against the installed 1.30.0 source.
- **Research:** twelve parallel passes, one per pattern category or source. Each read its pages in full and audited the code at HEAD.
- **Verification:** four independent passes re-checked every P1 claim, every Misused verdict, every defect statement and every figure this RFC quotes. They tried to refute each one against the code, the 1.30.0 and 1.34.0 SDK sources, the CLI and dev-server source, and the recorded 0.3.1 replay histories. Of 493 claims checked:
- 388 were confirmed;
- 88 were corrected;
- 4 were refuted;
- 13 could not be verified without a live server.
This RFC states only confirmed or corrected findings.
- **Owner questions:** four rounds on 2026-10-24 (§3.1).
- **Drafting:** six passes wrote §6 to §19 from the verified notes. Six more passes applied the owner's last decisions, shared numbering and cross-references.
- **Not done:** nothing was run against a live server. Timings and event counts come from the recorded replay histories, which were made with stub activities, so they are upper bounds on orchestration cost, not measurements of a live instance.
## 3. Decisions
### 3.1 Owner decisions (2026-10-24, four rounds)
| # | Decision | Changes |
|---|---|---|
| T1 | **Sources.** The review covers the design-pattern catalogue plus the AI cookbook, the best practices, the Python samples and Nexus. | §2 |
| T2 | **Role.** This RFC amends RFC-0005 in the same commit wherever a pattern changes a phase's design. | §20 |
| T3 | **Depth.** A full entry per pattern, with Python sketches. | §6-§19 |
| T4 | **Priority.** When two patterns pull in different directions, reliability wins: no lost events and no stuck or duplicated runs, even at some cost in latency or money. | throughout |
| T5 | **Fresh start confirmed.** RFC-0005 E16 stands: the first pinned release deletes old Temporal databases, and every install runs server 1.32. The code that keeps an install's old database and server (commit 49177427) is reverted in Phase 1. Server 1.31 and 1.32 features need no per-install gating once that release ships. | RFC-0005 §3.1 E16, §24.4 |
| T6 | **Agent transcripts stay in history.** They keep travelling in LLM-step inputs, bounded by the byte budget, compaction and continue-as-new. RFC-0005's rule that nothing over 64 KB enters Temporal (I7) gets an exception for agent loops. | RFC-0005 §6.2 I7 |
| T7 | **Polls are retried activities.** Each poll trigger gets one small workflow that runs a single activity. The activity fetches, stores the new items and the cursor in the database in one transaction, then ends with a BENIGN error so Temporal retries it after the interval. History stays near zero. This replaces one Schedule per poll. | RFC-0005 §3.1 E5, §15 |
| T8 | **Hosted tier S splits workers out.** Hosted instances run Temporal workers in a second process; the desktop stays one process. First, activities bound to the backend process (the WhatsApp bridge, the phone emulator, the Claude session pool, the socket broadcaster, the encryption key) are routed to a queue only that process polls. | RFC-0005 §20 |
| T9 | **No debounce for owner chat.** The turn starts at once; messages sent while it runs merge at the agent's next step (merge point B1). The docs say timers are accurate only from one second up. | RFC-0005 §11 |
| T10 | **Owner actions write an intent first.** Stop, Resume, approval decisions and Interrupt write a database row and answer at once (§7.3 applies the same to Start and Apply). A background deliverer sends the Update with a stable id until it is applied, and the UI shows "applying". For approval decisions, RFC-0007 (2026-10-25) replaces the Update with a nudge Signal, sent again by the approvals watchdog; the workflow re-reads the row. | RFC-0005 §10.2, §13.4, §17.5; RFC-0007 §4.6 |
| T11 | **Stuck turns park.** A turn that cannot continue for a reason that may clear (a provider outage past the step's time budget, an exhausted key or allowance, an account problem) suspends where it stopped. It holds no worker slot, raises an attention item, and continues on Resume, top-up, or when the outage ends. | RFC-0005 §10.3, §14 |
| T12 | **Stop does not wait for drafts.** Stop completes while an approval draft waits. A decision taken while stopped is sent on Resume. | RFC-0005 §10.2, §17.5 |
| T13 | **Side effects are redone only when idempotent.** A side-effecting tool call that broke off is run again only if its plugin declares itself idempotent or uses a provider idempotency key. Otherwise it is recorded as `unknown` and the employee is told. This extends E12 from sends to every side-effecting tool. | RFC-0005 §3.1 E12, §17.3 |
| T14 | **A release that cannot replay an open run starts anyway.** The affected generations are marked "needs Reset". This applies to replay failures only; a release that fails its readiness check still rolls back (RFC-0005 §21.2). | RFC-0005 §21.2, §24.1 |
| T15 | **Outgoing content rules.** Every outgoing message passes built-in rules (a saved credential, a secret-shaped token, a recipient outside the conversation) and the owner's own rules, even with Ask first off. A match holds the message for the owner. Amended by the owner on 2026-10-25: no key checks ("the LLM router will handle the API keys"), so the saved-credential and secret-token rules and 1b.7 are dropped. The recipient rule and owner rules ship in RFC-0005 Phase 7, where the approval gate calls the approvals service in one line (RFC-0007 §3). | RFC-0005 §17 |
| T16 | **Phase 1 is finished after this RFC,** by the same session. | RFC-0005 §24.4 |
| T17 | **The fresh-start release adds fairness and one partition.** The dev server gets `matching.enableFairness=true` and one task-queue partition (read and write) in the release that creates every Temporal database empty, because partitions can be lowered safely only on a new database. Retention and search attributes at namespace creation stay deferred, as decided on 2026-10-23 (RFC-0004 D4). The Start-snapshot secret fix ships in Phase 1b. | RFC-0005 §13.2, §20, §24.2 |
| T18 | **The step budget and parking ship with Phase 5, for new generations only** (the generation flag `scheduler_version=1`). Existing generations keep today's unlimited LLM-step retries and fail-and-pause. A budget without parking would fail turns and lose their input. | RFC-0005 §14, §24.2 |
| T19 | **A parked turn waits at most 7 days.** Then its input is dead-lettered with an attention item. RFC-0005's per-class turn deadlines become "slow turn" alerts that do not count parked time. (A 24-hour backstop for turns that are stuck without being parked is a default, §24.1.) | RFC-0005 §13.1, §18.1, §23 |
| T20 | **A lost supervisor is rebuilt in place.** A terminated or failed supervisor is started again from the database (`ALLOW_DUPLICATE_FAILED_ONLY`), keeping chat and Context. After 3 rebuilds within an hour it stops and asks the owner to Reset. | RFC-0005 §10.1 |
Decisions already in force: RFC-0005 §3.1 (E1-E16) and RFC-0004 §17, including Q6: a total time budget per LLM step. §11 of this RFC proposes the values, and T18 places them in Phase 5.
## 4. The catalogue at a glance
One row per entry in §6 to §19. Priority P1 means Phase 1b (the ids are listed in §22), P2 means the RFC-0005 phase named, and P3 means later or never.
### 4.1 Design patterns
| § | Pattern | Verdict | P | Action |
|---|---|---|---|---|
| 6.1 | Task orchestration (overview) | Adopted with deviations | P1 | Explicit child options, named bounds, named loser cleanup; lint (1b.40, dropped) |
| 6.2 | Child Workflows | Adopted with deviations | P1 | Agent children request cancellation (1b.12, moved to the cancellation lane with the re-raise); cancel-and-drain and durable Claude Code and RLM delegations (cancellation lane) |
| 6.3 | Parallel Execution | Adopted with deviations | P2 | One failure policy (running nodes finish, nothing new starts); graph by reference; delegation-window fix |
| 6.4 | Pick First (race) | Adopted with deviations; RFC-0005 turn-deadline race planned (deviates) | P1 | A chat run past 24 h cancels its workflow (1b.14); for Phase 4, slow-turn alerts (T19) and a 24-hour no-progress backstop (a default, §24.1) |
| 7.1 | Workflow messaging (overview) | Adopted with deviations | P1 | The handler checklist (§5 rules 12-15) |
| 7.2 | Signal with Start | Missing and beneficial (as start-with-state); RFC-0005 relay planned (deviates) | P2 | Triggers in the controller's start input; the stream cursor as the entity's start input |
| 7.3 | Request-Response via Updates | Adopted with deviations; RFC-0005 owner Updates planned (follows) | P1 | A deadline on every Update call (1b.15); stable ids, slim results, the T10 deliverer |
| 7.4 | Event Accumulator | Adopted with deviations; RFC-0005 debounce planned (follows) | P2 | Dead-letter poison events; no owner-chat debounce (T9); absolute debounce deadline; carry cursors |
| 8.1 | Entity and lifecycle (overview) | Not applicable (selection guide) | P3 | The database-as-truth rules (§5.1) |
| 8.2 | Entity Workflow | Adopted with deviations; RFC-0005 plan deviates | P2 | Decommission the Workspace controller; turn numbers from admission; a seeded `applied_seq`; supervisor rebuilt in place (T20) |
| 8.3 | Continue-As-New | Adopted with deviations | P2 | Slim controller carry, cancellation fix, rollover at the first boot of each release; slim agent carry (1b.19) |
| 8.4 | Updatable Timer | Planned in RFC-0005 (follows, misses details) | P2 | One timer over absolute deadlines; no zero timeouts; no sub-second windows; one instance-wide upkeep sweep |
| 9.1 | External interaction (overview) | Not applicable (selection guide) | P3 | "Where each wait goes" (§5 rule 2) |
| 9.2 | Polling External Services | Adopted with deviations | P1 | Bounded attempts, without heartbeats (1b.17); one retried poll activity per trigger in Phase 3 (T7) |
| 9.3 | Long-Running Activity | Adopted with deviations | P1 | Activities re-raise cancellation (1b.2); long waits fit one attempt (1b.4, dropped); checkpoint external job ids |
| 9.4 | Delayed Start | Not applicable | P3 | No adoption; never `start_delay` on ids addressed by Signal-With-Start |
| 9.5 | Delayed Callback (webhooks) | Misused (Timer node; webhooks until Phase 2) | P1 | Timer node fixes (1b.8); a durable Timer behind a flag; the Phase 2 inbox for webhooks |
| 9.6 | Approval | Misused (the wait) | P1 | Reviewed delivery re-attaches (1b.9); approved sends wait while stopped (1b.5); stranded drafts expire (1b.10); ApprovalWorkflow for held tool calls (RFC-0007, AP.4a and AP.4b), then the gate and delivery (Phase 7) |
| 10.1 | Distributed transactions (overview) | Adopted with deviations | P1 | The transaction rule in DESIGN.md (1b.45) |
| 10.2 | Saga | Adopted with deviations | P1 | Compensations retry until they land (1b.20); Reset closes everything it started (1b.21); Start's compensation in reverse (1b.22); interrupted Resets finish (1b.23) |
| 10.3 | Early Return | Adopted with deviations | P1 | A bounded Workspace Update-with-Start (1b.15); later behind the T10 deliverer |
| 11.1 | Error handling (overview) | Adopted with deviations | P1 | Shared retry constants (with 1b.9, 1b.16 and 1b.20); the lint (1b.41) dropped |
| 11.2 | Fixed Count of Retries | Adopted with deviations | P1 | Effect classes and the redo guard (1b.1); the legacy tool path honours declared policies (1b.3) |
| 11.3 | Fixed Wall-Time Retries | Missing and beneficial | P2 | A per-step budget of 15 min (cloud) and 45 min (local); expiry parks the turn (Phase 5, new generations, T18) |
| 11.4 | Non-Retryable Errors | Misused | P1 | LLM-step bugs and `PayloadsTooLarge` non-retryable (1b.26); compaction on the step's policy (1b.27) |
| 11.5 | Delayed Retry | Adopted with deviations | P2 | A provider wait longer than the remaining budget parks the turn (Phase 5) |
| 11.6 | Fast/Slow Retries | Adopted with deviations | P2 | Parking is the slow phase (Phase 5); controller spawn backoff and dead letter behind a patch |
| 11.7 | Retry Alerting via Metrics | Missing and beneficial | P1 | The attention item and a quieter log now (1b.30); the metric counter in Phase 8 |
| 11.8 | Resumable Activity | Adopted with deviations | P2 | One park mechanism (Phase 5; T11, T18, T19) |
| 12.1 | Batch processing (overview) | Adopted with deviations | P1 | A real-server test of the 2,000-Update limit (1b.43) |
| 12.2 | Fan-Out with Child Workflows | Adopted with deviations | P2 | Admission caps for new generations; a cap on open team tasks |
| 12.3 | Batch Iterator | Adopted with deviations | P1 | Controller paging retries without limit (1b.16); bounded poll pages (1b.18) |
| 12.4 | Sliding Window | Missing and beneficial | P2 | Permit waits inside the workflow with a wake Signal; Reset releases permits |
| 12.5 | MapReduce Tree | Not applicable | P3 | Do not adopt |
| 13.1 | QoS and throughput (overview) | Overview | P2 | One enforcement point per resource; "reserved share" means admitted turns |
| 13.2 | Downstream Rate Limiting | Misused | P2 | Frozen class-queue routes for LLM steps; per-credential buckets; never cap `machina-tasks` |
| 13.3 | Priority | Planned in RFC-0005 (follows, misses details) | P2 | An explicit key on every top-level start; entities keyed by lane |
| 13.4 | Fairness | Planned in RFC-0005 (follows, misses details) | P1 | Fairness and one partition in the fresh-start release (T17); the employee as fairness key |
| 14.1 | Performance and latency (overview) | Overview | P2 | A timing mode with production settings; a pinned metrics port |
| 14.2 | Local Activities | Missing and beneficial | P2 | AgentWorkflow progress broadcasts as best-effort local activities (`progress_local_version=1`, after SDK 1.34.0); new types schedule no UI-only activity |
| 14.3 | Early Return + Local Activities | Adopted with deviations (Workspace tasks); Not applicable (chat) | P3 | Only if measured |
| 14.4 | Eager Workflow Start | Not applicable | P3 | Do not adopt |
| 15.1 | Worker configuration (overview) | Adopted with deviations | P1 | A placement axis; one main-queue registration (1b.36) |
| 15.2 | Worker-Specific Task Queues | Adopted with deviations; process affinity missing | P1 | The pool always polls (1b.34); the standalone guard (1b.35); an instance queue before the T8 split |
| 15.3 | Activity Dependency Injection | Adopted with deviations | P1 | One main-queue registration (1b.36); no worker builder |
### 4.2 AI agents, best practices, samples and Nexus
| § | Entry | Verdict | P | Action |
|---|---|---|---|---|
| 16.1 | Durable AI and the cookbook | Adopted with deviations | P2 | Priority and fairness keys (Phase 5); summaries without user text (Phase 8) |
| 16.2 | Agentic loop with tool calls | Adopted with deviations | P1 | Non-retryable failures (1b.26); compaction's policy (1b.27) |
| 16.3 | LLM call timeouts | Adopted with deviations | P1 | 30-minute attempts for local endpoints and a classified request timeout (1b.25; no in-attempt deadline, no Gemini timeout); the step budget in Phase 5 (T18) |
| 16.4 | Hello world (OpenAI Responses) | Adopted with deviations | P3 | Settle node-level SDK retries with RFC-0004 Q9 |
| 16.5 | Retry policy from HTTP responses | Adopted with deviations | P2 | Honour `x-should-retry`, retry 409, cap Retry-After (Phase 5) |
| 16.6 | Deep research | Adopted with deviations | P3 | Summaries (Phase 8) |
| 16.7 | Structured outputs | Adopted with deviations | P3 | Stop offering a tool after 3 refusals in a turn |
| 16.8 | Claim check | Adopted with deviations | P1 | Slim the carry (1b.19); relief before every step (Phase 4); the I7 exception and its bounds (T6) |
| 16.9 | Guardrails | Adopted with deviations | P1 | Built-in content rules (1b.7, dropped by the owner); owner and recipient rules (Phase 7) (T15) |
| 16.10 | Human in the loop | Misused | P1 | See §9.6 |
| 16.11 | Agent memory and state | Adopted with deviations | P1 | The redo guard (1b.1) |
| 16.12 | Durable MCP server | Not applicable | P3 | A durable route only if Claude Code employees should send |
| 16.13 | Multi-agent with ADK | Adopted with deviations | P3 | Tool errors built with `json.dumps` (Phase 4) |
| 16.14 | Framework cookbook pages | Not applicable | P3 | Keep the native loop |
| 16.15 | Framework plugin samples | Not applicable | P3 | Ideas only |
| 16.16 | OpenRouter samples | Planned in RFC-0005 (deviates) | P2 | Provider exhaustion and long outages park the turn (Phase 5) |
| 16.17 | Bedrock and AgentCore samples | Planned in RFC-0005 (follows) | P2 | No change |
| 16.18 | Google GenAI sample | Not applicable | P3 | None |
| 16.19 | Tracing and metrics samples | Not applicable | P2 | gen_ai metric names, no content (Phase 8) |
| 16.20 | Workflow Streams sample | Adopted with deviations | P3 | Keep the chat hub |
| 17.1 | Best practices (index) | Not applicable | P3 | None |
| 17.2 | Namespace best practices | Adopted with deviations | P2 | Retention and search attributes at creation, deferred per RFC-0004 D4 (T17) |
| 17.3 | Multi-tenant patterns | Planned in RFC-0005 (follows) | P2 | See §13.4; no ids in metric labels |
| 17.4 | Managing APS limits | Missing and beneficial | P1 | A history-cost counting script (1b.44, dropped); jittered budget timers; no UI-only activities |
| 17.5 | Workflow cost optimization | Adopted with deviations | P2 | No UI-only activities in new types; graphs by reference |
| 17.6 | Cost governance | Not applicable | P3 | Release tags in fleet telemetry |
| 17.7 | Cloud access control | Not applicable | P3 | Telemetry key hygiene |
| 17.8 | Security controls | Misused (Start path) | P1 | Sanitized Start snapshot (1b.31, 1b.32); named ports and netpolicy (1b.33) |
| 17.9 | Worker deployment and performance | Adopted with deviations | P1 | Graceful shutdown (1b.37); the T8 worker split |
| 17.10 | Alerting on worker metrics | Missing and beneficial | P1 | A stuck-runs attention item (1b.38b); the SDK metrics and starter alerts in RFC-0005 Phase 8 |
| 17.11 | Error handling | Adopted with deviations | P1 | Explicit chain-unique activity ids; cancellation (1b.2); the send guard (1b.1) |
| 17.12 | Pre-production testing | Missing and beneficial | P2 | Replay open runs before polling and mark "needs Reset" (T14, Phase 9a) |
| 17.13 | Knowledge hub | Adopted with deviations | P1 | Stale comments (1b.45) |
| 17.14 | Python best practices (index) | Not applicable | P3 | None |
| 17.15 | Python debugging | Adopted with deviations | P2 | Core logs at WARN; opt-in span exporter (Phase 8) |
| 17.16 | Python error handling | Adopted with deviations | P1 | The bug guard (1b.13); bugs stay stuck and visible |
| 17.17 | Python testing suite | Adopted correctly | P1 | Node-cancel and per-type bug tests |
| 17.18 | Python SDK sandbox | Misused | P1 | The determinism lint (1b.40, dropped); new types sandboxed (Phase 4) |
| 17.19 | Python sync vs async | Adopted with deviations | P1 | Loop-lag watchdog (1b.39); the T8 second process (Phase 9) |
| 18.1 | Pattern-to-sample map | - | - | 12 of 35 patterns have no Python sample |
| 18.2 | Exit cleanup and handler waits | Misused | P1 | The bug guard (1b.13) |
| 18.3 | Message handlers and Update-with-Start | Adopted correctly | P3 | None |
| 18.4 | Workflow basics (`hello`) | Adopted correctly | P3 | None |
| 18.5 | Observability samples | Missing and beneficial | P1 | Task failures at ERROR (1b.29); BENIGN retries (1b.28) |
| 18.6 | Polling | Adopted with deviations | P2 | T7's poll workflow (Phase 3) |
| 18.7 | Timers | Planned in RFC-0005 (follows) | P2 | Versioned `alarms_changed` (Phase 6) |
| 18.8 | Schedules | Adopted correctly | P3 | Cron jitter (planned) |
| 18.9 | Patching | Adopted with deviations | P2 | Four-stage retirement; dormant patches after SDK 1.34.0 |
| 18.10 | Replay | Adopted with deviations | P2 | Replay open runs at upgrade (Phase 9a) |
| 18.11 | Worker-specific task queues | Adopted with deviations | P2 | The instance queue (T8) |
| 18.12 | Worker processes | Planned in RFC-0005 (follows) | P2 | Second process on hosted tier S (T8) |
| 18.13 | Worker versioning | Not applicable | P3 | Declined for topology |
| 18.14 | Long-running activities | Adopted with deviations | P1 | Cancellation (1b.2); heartbeat interval from the timeout |
| 18.15 | Asynchronous activity completion | Not applicable | P2 | Human waits move to ApprovalWorkflow, a nudge Signal plus a timer: held tool calls since RFC-0007, the gate and delivery in Phase 7 |
| 18.16 | Resource pool | Adopted with deviations | P3 | Reset releases permits (planned for 1b.21, not built there; the age reclaim still frees them) |
| 18.17 | Interpreters and batches | Adopted correctly | P3 | None |
| 18.18 | Codecs, external storage, converters | Not applicable | P3 | References first; a codec only with an owner decision (RFC-0004 Q13) |
| 18.19 | Eager start and local activities | Not applicable | P3 | Drop RFC-0004's eager-start experiment |
| 18.20 | Test techniques | Adopted correctly | P3 | None |
| 18.21 | Samples that do not apply | Not applicable | P3 | None |
| 19.1-19.10 | Nexus, its operations, patterns, security and standalone activities | Not applicable | P3 | Keep out; correct RFC-0004 §4.2, §6.6, §9.6 and §15 |
### 4.3 Counts
Across the 45 design-pattern entries (10 category overviews and 35 patterns; the catalogue's index page has no entry), counting each entry's main verdict:
- Adopted correctly: none.
- Adopted with deviations: 26.
- Misused: 4 (Delayed Callback, Approval, Non-Retryable Errors, Downstream Rate Limiting).
- Missing and beneficial: 5.
- Planned in RFC-0005: 3.
- Not applicable, or an overview with no verdict of its own: 7.
Most of OpenCompany's Temporal code follows the documented patterns in shape. The deviations are in the details the pages call out: cancellation, retries, waits, and what happens on a restart.
## 5. Rules every workflow and activity follows
These rules distil the patterns in §6 to §19 into checks a reviewer can apply to any change in `server/services/temporal/`, the plugin framework or a new workflow type. Each names the section that explains it. Where a lint or test enforces a rule, the rule says so (§22 schedules them).
### 5.1 The database holds the truth
1. **Workflows hold handles, cursors and timers, not facts** (RFC-0005 I10).
- Anything the owner sees, or that a restart, a Reset or a fresh Temporal database must recover, lives in the app database first. That covers control state, approvals, sends, budgets, alarms and conversation cursors.
- Turn numbers and any id that feeds an idempotency or spend key live there too, and every entity start (the first, a restart after passivation, a rebuild) seeds its state from the database.
- A Signal is a hint to look (I1). (§8.1, §8.2)
2. **A wait goes where it can survive what it waits through.** (§9.1)
- Waiting seconds for an answer: an activity, within one attempt.
- Waiting for a person or a date: an Update plus a durable workflow timer.
- Waiting for a provider to recover: activity retries, then a parked turn (T11).
- Never sleep inside an activity for longer than one attempt can last.
- Never put `start_delay` on an id that the relay addresses with Signal-With-Start: any delivery would bypass the delay (§9.4).
### 5.2 Workflow code
3. **Determinism is checked, not assumed.**
- The 14 existing types and RFC-0007's ApprovalWorkflow run unsandboxed (`sandboxed=False`). A determinism lint covers their workflow code: the methods of `@workflow.defn` classes and the helpers they call, not activities that share the module. It checks for no I/O, no wall clock, no unseeded randomness, no `asyncio.wait` or `asyncio.as_completed` (their ordering is not deterministic; `workflow.wait` and `workflow.as_completed` are), and no command-issuing code in a signal handler. That lint was 1b.40, dropped on 2026-10-25, so today only the replay gate checks this.
- New RFC-0005 types run in the sandbox, passing through only side-effect-free leaf modules. (§17.18)
4. **Cleanup runs only when the workflow is really exiting.**
- Code that issues commands in `finally`, `except Exception` or `except BaseException` first re-raises a workflow bug (`is_workflow_bug`). Otherwise a bug records cleanup commands, marks rows failed, and makes the fixed release fail replay.
- `asyncio.TimeoutError` counts as a failure, not a bug. (§18.2)
5. **Every child start states its policies.**
- Each child start passes `parent_close_policy`, `cancellation_type` and `id_reuse_policy` explicitly.
- Agent children use `REQUEST_CANCEL` with `WAIT_CANCELLATION_COMPLETED`; per-event runs use `ABANDON`.
- A lint was to reject a child start without them (1b.40, dropped on 2026-10-25). (§6.2)
6. **Know what replay compares.**
- On replay Temporal compares only a command's kind, its activity or child type, and its id.
- Changing timeouts, retry policies, task queues or a parent close policy needs no patch.
- Changing a cancellation type does need a patch whenever a replayed history can contain a cancel of that activity or child, because the cancellation type decides whether a cancel emits a command. (§6.2; corrects RFC-0004 §8.5)
7. **Every fan-out names its bound, and every race names its loser's fate.** That means who cancels the loser, whether the workflow waits for its cleanup, and how long that wait may last. (§6.3, §6.4)
8. **Timers run over absolute deadlines.**
- One live timer per workflow over named absolute deadlines, through a shared helper.
- Never `wait_condition` with a timeout of zero or less.
- No sub-second timers: the docs say timers are accurate only from one second up (https://docs.temporal.io/workflow-execution/timers-delays). (§8.4)
9. **Close cleanly.**
- Continue-as-new and completion happen only in the main method, after `await workflow.wait_condition(workflow.all_handlers_finished)`, with no `await` between the final buffer check and the close.
- A long-lived type rolls over when the server suggests it, or at 1,800 Updates (90% of the 2,000-per-run limit), whichever comes first. (§7.4, §8.3, §12.1)
10. **Activity ids are idempotency keys.** Wherever an id feeds an idempotency or spend key, it is explicit and unique across the continue-as-new chain. The SDK's default activity id is a per-run counter that restarts after continue-as-new. (§17.11)
11. **No UI-only activities in new types.** Progress broadcasts become best-effort local activities or ride an adjacent activity. Today they are 35-52% of an AgentWorkflow history. (§14.2, §17.4)
### 5.3 Messages
12. **Handlers stay small.** Signal handlers only buffer or set flags. Rejections that need no I/O live in Update validators, which reject without writing history. (§7.1)
13. **Every client-side Update call that a request or a loop waits on runs under a deadline** (`update_calls.bounded_update`, 1b.15); a background transition's own Updates may wait. `rpc_timeout` bounds one request only: after the Update is accepted, `execute_update` waits for the outcome without it. (§7.3; corrects RFC-0004 §8.2)
14. **Owner actions go through intents** (T10).
- Stop, Resume, approval decisions, Interrupt, Start and Apply write an intent row and answer at once.
- A background deliverer sends the Update by workflow id, with a stable id and a per-attempt deadline, until it is applied. An approval decision is the exception (RFC-0007 §4.6): the request writes the row and nudges the waiting `ApprovalWorkflow` with a Signal, and the chat watchdog nudges again until the decision is read.
- No request path waits on Temporal (RFC-0005 I2). (§7.3, §10.2)
15. **Updates carry absolute values and a revision.**
- Handlers are idempotent by revision across continue-as-new, because Update-id dedupe works only within one run.
- A stale revision is rejected in the validator. (§7.3, §8.2)
### 5.4 Activities, retries and side effects
16. **Named retry policies only.**
- Every activity call site uses a shared policy constant. A lint rejects inline policies in the workflow modules `server/tests/test_retry_policies.py` lists; the lint for every module was 1b.41, dropped on 2026-10-25.
- A paid or side-effecting call never retries without a time budget.
- One exception: the LLM step and compaction on generations without `scheduler_version=1` (RFC-0005 Phase 5) keep unlimited retries, under the retry alert (1b.30). A budget without parking would fail their turns (T18).
- An unlimited policy is otherwise allowed only for idempotent bookkeeping, compensations, queue paging, permit waits, the approval gate's wait for the owner, the Pi driver's turn, T7's poll activities (each retry is the next poll), and sends and deliveries made at-most-once by their claim (the egress outbox; reviewed delivery's `inflight` mark, 1b.9). Each is declared, and the retry alert covers all of them except the planned retries: T7's polls, and the permit waits, approval gate and Pi driver that `PLANNED_WAITS` lists (1b.30). (§11.1; rewords RFC-0004 AG9)
17. **Classify every failure.**
- User and configuration errors fail fast as non-retryable: `NodeUserError`, permission errors, `PayloadsTooLarge` (Core's type since SDK 1.31; `PayloadSizeError` before) and bugs inside the LLM step.
- Rate limits retry as BENIGN with the provider's `next_retry_delay`.
- Planned retries (T7's poll attempts, permit waits, the approval gate, the Pi driver) are BENIGN and excluded from retry alerts. (§11.4, §11.5, §11.7)
18. **Side effects say whether they may repeat** (T13, E12).
- One service table, `services/approvals/repeats.py`, lists the calls that may run again: they change nothing outside, or make the same change again. Every other call that changes something outside OpenCompany is `unsafe`. No plugin declares anything (1b.1a, on the owner's direction of 2026-10-25).
- An `unsafe` call runs under a claim guard: if an attempt broke off after the provider might have acted, the next attempt records `unknown` and tells the employee instead of calling again.
- This applies to graph nodes and tool calls alike, sends included. (§11.2, §16)
19. **Cancellation reaches the code.**
- Activities re-raise cancellation instead of returning a failure envelope (RFC-0004 H2).
- A worker shutdown becomes a retryable error, so the next worker resumes the work. (§9.3)
20. **Long activities heartbeat.**
- The heartbeat interval is a third of `heartbeat_timeout`, at most 30 s.
- They record external job ids in a database row before waiting on the job, and also in heartbeat details. A retry then re-attaches to the job instead of starting it again. Heartbeat details alone are lost when the activity is scheduled again after a rollover. (§9.3)
### 5.5 Capacity and order
21. **One enforcement point per resource.**
- Turns are limited by admission (RFC-0005 §13.3), providers by per-credential buckets.
- Queue-wide rate caps go only on a queue with one downstream and one fixed limit, and never on `machina-tasks`, where a cap turns off eager activity dispatch. (§13.1, §13.2)
22. **Every top-level workflow start sets an explicit priority key.** An unkeyed start runs at 3, which is RFC-0005's internal class; unkeyed children and activities inherit their parent's key. An AST test enforces this. (§13.3)
### 5.6 Payloads and secrets
23. **References, not bytes.**
- Media travels as `FileRef`, and events over 64 KB travel by reference (RFC-0005 I7).
- Agent transcripts are the one exception (T6). They stay within its bounds: at most 393,216 bytes of messages before every step, each step input under 1 MiB, the continue-as-new carry under 1.9 MB and the stored conversation under 1 MB. (§16.8)
24. **No secrets in Temporal inputs.** Snapshots of node parameters are sanitized on every path, Start included, and activities read credentials from the saved row. (§17.8)
### 5.7 Workers and tests
25. **One worker builder.**
- Every Worker, embedded, standalone, test or recorder, comes from one function.
- That function makes dependencies ready before polling, registers from one list (`workflow_registry.py`), and checks the registration against what workflows schedule.
- Built narrower in 1b.36 (d379ab3e): one main-queue registration (`_main_queue_activities`, `temporal_plugins()`) serves the embedded manager, the standalone worker and `create_worker`. There is no builder function and no registration check.
- The pool always polls the queues frozen runs can target.
- Activities bound to the backend process run on the instance queue (T8). (§15)
26. **Every limit we rely on has a real-server test.** That includes Updates per run, history size and continue-as-new suggestions. Every change to workflow code passes the N-1 replay gate and the golden identity test. (§12.1, §17.17)
### 5.8 Changes that span systems
27. **One local transaction where possible; a saga only across systems.**
- A change that touches the database and Temporal, or a provider, writes its intent in one local transaction, and a worker carries it out (an outbox).
- Irreversible effects go last.
- A saga only where several systems change: compensations run last in, first out, each on its own.
- An effect that cannot be compensated (a sent message, a refund) is guarded before it happens by an Undo window and a claim at a revision; after it, an unclear outcome is reported as `unknown`, never undone.
- An owner's action writes an intent and answers; a deliverer applies it (T10). (§10.1; `docs-internal/DESIGN.md` §5)
## 6. Task orchestration
Three pages cover how one workflow splits, parallelizes and races work. OpenCompany uses all three mechanisms where the docs put them: agents run as child workflows, graph nodes run as parallel activities, and races happen inside activities. The gaps are in what happens to work that is still running when its parent stops or fails.
Terms used below:
- A **Child Workflow** is a workflow started by another workflow, its parent, with its own id and history.
- A **command** is what workflow code asks the server to do: schedule an activity, start a child, start a timer, record a patch marker. On replay the SDK checks the code's commands against the recorded history, so a change to which commands a workflow issues needs versioning: `workflow.patched(...)` inside a type that old and new runs share, or a per-generation `resource_manifest` flag frozen into the workflow's input (RFC-0005 §24.1 rule 2).
- The **parent close policy** decides what the server does to a child that is still running when its parent closes: TERMINATE (kill it; none of its code runs), REQUEST_CANCEL (send it a cancellation request, so its cleanup runs) or ABANDON (leave it running).
- The **cancellation type** decides what the parent's handle does when the parent itself cancels the child or activity: TRY_CANCEL (request the cancel and resolve at once; the default for activities), WAIT_CANCELLATION_COMPLETED (resolve only when the child or activity has finished cancelling; the default for children) or ABANDON (resolve at once and send no cancel).
- The **id reuse policy** decides whether a new execution may take the id of a closed one.
### 6.1 Task orchestration patterns (overview)
**Docs:** https://docs.temporal.io/design-patterns/task-orchestration-patterns. A chooser for the three patterns of this section:
- "A process is large or reused across Workflows": Child Workflows "with independent histories".
- "Independent work can run at the same time": Parallel Execution "with a concurrency bound".
- "Several approaches compete and you want the first to finish": Pick First "and cancel the losers".
- Scale-out goes to Batch Processing, latency to Performance & Latency.
**SDK:** none of its own (§6.2 to §6.4).
**OpenCompany today:**
- Agents are child workflows. MachinaWorkflow schedules graph nodes as parallel activities. AgentWorkflow runs a turn's tools one at a time, delegations as a window of children, and Task Manager work as ABANDON runners. The controller starts one child run per event. Races exist only inside activities. (§6.2 to §6.4.)
- No rule says which close policy, cancellation type and id reuse policy a child start uses. Four agent-child starts, at six call lines, leave the parent close policy at the SDK default, TERMINATE (table in §6.2).
- No fan-out names its bound. MachinaWorkflow schedules every ready node (`server/services/temporal/workflow.py:640-874`); the controller starts one child per event without a cap (`server/services/temporal/workflow_control_workflow.py:431-483`).
- Work that should stop is either killed without cleanup or left running: an uncontrolled run's failure abandons its running nodes (`workflow.py:907-908`), a cancelled root's agent children are terminated (§6.2), and a chat run past 24 hours leaves its workflow running (§6.4).
- Nothing enforces determinism. All 14 workflow definitions pass `sandboxed=False`, so the SDK's sandbox warning for `asyncio.wait` can never fire, and no test forbids it.
**Verdict:** Adopted with deviations. The three mechanisms sit where the docs put them; bounds and the cleanup of losers are stated nowhere.
**Recommendation:**
1. **State three orchestration rules and lint them.** §5 adopts them as rules: (a) every child start passes `parent_close_policy`, `cancellation_type` and `id_reuse_policy` explicitly; (b) every fan-out names its bound; (c) every race names who cancels the loser and whether the workflow waits for its cleanup. The checks ship in the determinism lint, 1b.40, which §17.18 specifies. The orchestration part this entry adds to it: a child start without the three explicit options; `asyncio.wait` and `asyncio.as_completed` in workflow modules; a direct `execute_activity` or `start_activity` call inside a `@workflow.signal` handler. (Pi `pause` and `resume` reach an activity indirectly, through `_forward`; §7.3 R6 removes them.) The same commit writes out the current defaults at the child starts that rely on them; passing a default explicitly changes nothing on replay. The retry-policy lint over the same workflow modules is 1b.41 (§11.1).
- Versioning: none (a test, and explicit defaults).
- When: Phase 1b (1b.40), after 1b.13. Dropped on 2026-10-25 (§22.2), so the lint waits. Until the cancellation lane ships 1b.12, the agent-child starts state today's policies explicitly. Effort S. Risk: none at run time.
**Remarks:** The overview points to Batch Processing. OpenCompany has no record-batch workload, so those pages map only to the controller's one child per event, its queue paging and the delegation window (§12.2 to §12.4).
### 6.2 Child Workflows
**Docs:** https://docs.temporal.io/design-patterns/child-workflows. A child "executes as an independent Workflow with its own Workflow ID, event history (51,200 event limit), and lifecycle". Use a child for a separate id, work that "may outlive the parent", reuse, another task queue or an independent history; use an activity for external calls ("Consider Activities first").
- Python: `workflow.execute_child_workflow(...)` waits for the child to finish; `workflow.start_child_workflow(...)` "returns a handle once the child has started".
- Parent close policy: TERMINATE for "Tightly coupled processes", ABANDON for "Fire-and-forget, long-running tasks", REQUEST_CANCEL for "Graceful cleanup". "The default policy is TERMINATE, which kills children when the parent closes." The cancellation concept page adds: "Anything a terminated Workflow would have cleaned up is left as it was: reserved inventory stays reserved, held locks" stay held (https://docs.temporal.io/encyclopedia/workflow/cancellation-and-termination).
- "You must wait for the child to start before the parent completes." The Python guide adds: from a non-main coroutine, "make sure the Parent Workflow doesn't complete before that call resolves".
- Best practices: unique ids, "Generate deterministic IDs based on business keys" (the page's own samples use `workflow.uuid4()`); choose the policy; handle `ChildWorkflowError`; "Limit parallelism" (a hard limit of 2,000 pending children per parent, and at most 1,000 children per parent in total is recommended); "Set timeouts"; typed stubs; "Monitor child executions".
- Pitfalls: "Treating Child Workflows like Activities", unbounded children in a loop, "Ignoring the Parent Close Policy", omitting ids, not handling child failures. The comparison table rates an "Async Lambda" as "Independent history: No; Can outlive parent: No".
**SDK:**
- 1.30.0: `workflow.start_child_workflow(..., cancellation_type=WAIT_CANCELLATION_COMPLETED, parent_close_policy=TERMINATE, id_reuse_policy=ALLOW_DUPLICATE, execution_timeout, run_timeout, task_timeout, retry_policy, memo, search_attributes, static_summary, static_details, priority)` (`temporalio/workflow/_workflow_ops.py:291-316`). Children have no `id_conflict_policy`.
- 1.34.0 adds only the experimental `event_groups` parameter; the enums and defaults are unchanged.
- In both versions, cancelling a workflow cancels only its main task (`temporalio/worker/_workflow_instance.py:598-606` in 1.30.0, `:660-668` in 1.34.0). A child that the code is not awaiting at that moment gets no cancel; the server applies its parent close policy when the parent closes.
- Cancellation fixes after 1.30.0, per the release notes: 1.31 "cancellations no longer swallowed in some circumstances"; 1.32 no nondeterminism error when a try-cancel child finishes after its cancel; 1.33 duplicate cancel commands de-duplicated (`_request_cancel`, present in the 1.34.0 source and absent in 1.30.0).
**OpenCompany today:**
Every child start in production code. The 27 recorded scenarios in `server/tests/temporal/fixtures/0.3.1/` and `fixtures/0.3.1-cancel/` record the same values. "Default" is the SDK default. No child start sets an execution or run timeout.
| Parent -> child | Start | Parent close | Cancellation type | Id reuse |
|---|---|---|---|---|
| MachinaWorkflow -> AgentWorkflow or PiCodingAgentWorkflow | `server/services/temporal/workflow.py:826` (execution control on), `:828` (off); id `<root id>-agent-<node id>` (`:170-172`) | TERMINATE (default) | default | ALLOW_DUPLICATE (default) |
| DelegatedTaskWorkflow -> agent | `server/services/temporal/agent_workflow.py:595-598` | TERMINATE (default) | default | default |
| AgentWorkflow -> agent (`delegate_to_*`) | `agent_workflow.py:1740-1744` | TERMINATE (default) | default | default |
| AgentWorkflow -> DelegatedTaskWorkflow (Task Manager runner) | `agent_workflow.py:1894-1899` (options `:1878-1891`, with `EventWorkflowId`) | ABANDON | default | default |
| EmployeeJobWorkflow -> lead agent | `server/services/temporal/employee_job_workflow.py:117` (on), `:120` (off) | TERMINATE (default) | default | default |
| PiInlineDelegationWorkflow -> DelegatedTaskWorkflow | `server/services/temporal/pi_inline_delegation.py:365-369` | REQUEST_CANCEL | WAIT_CANCELLATION_COMPLETED (explicit) | default |
| WorkspaceTaskControllerWorkflow -> NodeInvocationWorkflow | `server/services/temporal/workspace_tasks_workflow.py:114-121` | REQUEST_CANCEL | explicit | REJECT_DUPLICATE |
| NodeInvocationWorkflow -> Workspace agent | `server/services/temporal/node_invocation.py:34-39` | REQUEST_CANCEL | explicit | default |
| Controller (push and poll) -> MachinaWorkflow per event | `server/services/temporal/trigger_listener_workflow.py:495`, `:497`; `polling_trigger_workflow.py:398-402`; entered from `workflow_control_workflow.py:755-765`, `:888-895` | ABANDON | default | ALLOW_DUPLICATE_FAILED_ONLY |
| CronTriggerWorkflow -> MachinaWorkflow | `server/nodes/scheduler/cron_scheduler/_workflow.py:203` (options `:176-189`) | ABANDON | default | ALLOW_DUPLICATE_FAILED_ONLY |
- **What follows the docs.**
- Agent loops are children, with their own history, continue-as-new (`agent_workflow.py:2982-3069`) and control Updates; tools are activities. NodeInvocationWorkflow runs a plain node as an activity and only an agent as a child (`node_invocation.py:31-49`). No child is used as a function call.
- Ids are deterministic and built from business keys (table).
- Starts resolve before a parent can finish. Cron and listeners return after the start resolves (`cron_scheduler/_workflow.py:186-214`; `trigger_listener_workflow.py:493-497`). Task Manager runner starts run in asyncio tasks that are awaited in call order before the turn ends (`agent_workflow.py:2015-2021`, `:2374-2381`), and the rollover guard refuses continue-as-new while any is live (`:2993-2996`). An exception that escapes the tool loop leaves runner-start tasks unawaited.
- Child failures: MachinaWorkflow turns a finished child's exception into a node failure (`workflow.py:1082-1133`); AgentWorkflow returns a failed delegation to the model (`agent_workflow.py:2574-2628`); DelegatedTaskWorkflow and EmployeeJobWorkflow record a terminal state and re-raise (`agent_workflow.py:692-702`; `employee_job_workflow.py:121-126`).
- Timeouts: none on children, on purpose (`workflow.py:148-155`; `agent_workflow.py:186-191`): runs may stay paused for months and timeout timers keep running through a pause. This departs from "Set timeouts" deliberately (RFC-0005 §27).
- Parallelism: delegations per turn are capped by `max_concurrent_subagents` (default 3, `agent_workflow.py:1451-1453`) and per root by a database permit (`server/services/temporal/agent_activities.py:2247-2339`).
- Monitoring: runner and child ids are recorded by `agent.register_task_execution` (`agent_workflow.py:553-561`, `:601-609`, `:1748-1757`). Workflow types travel as strings on purpose, guarded by `server/tests/temporal/test_workflow_identity_golden.py` and `server/services/temporal/workflow_registry.py`.
- **Where a parent closes early, so TERMINATE kills an agent without cleanup:**
1. *An uncontrolled run fails.* The first failed node ends the loop (`workflow.py:907-908`); the run completes with `success: False` (`:913-929`) and the server terminates its agent children. Manual canvas Runs are uncontrolled: execution-control fields are copied only for an admitted generation (`server/services/temporal/executor.py:152-173`; `server/routers/websocket.py:968-975`).
2. *Chat Stop outlasts its grace period.* Stop moves the run to `stopping` (`server/services/chat/ledger.py:598-639`). The watchdog cancels the run's MachinaWorkflow 45 to about 75 seconds later (`ledger.py:714-728`; `stop_grace_s` 45 and `watchdog_interval_s` 30 in `server/config/chat_defaults.json`). MachinaWorkflow waits in `workflow.wait_condition` (`workflow.py:1120`) with no `CancelledError` handler, closes Cancelled, and the server terminates its AgentWorkflow. A run that stops cooperatively inside the grace closes normally.
3. *Apply with "stop work".* `server/services/employees/safe_apply.py:112-116` cancels running executions found by `EventWorkflowId`. Agent children carry no search attributes (`workflow.py:820-822`), so only the parent's TERMINATE reaches them. Task Manager agents are the exception: their runner carries `EventWorkflowId` (`agent_workflow.py:1882-1891`), is cancelled directly and awaits its agent (`:610`), so that agent gets a real cancel. A plain agent in a tool call does not stop on it: it turns the cancelled tool call into a tool error and keeps working (`agent_workflow.py:2574-2582`; only Workspace and browser-runtime agents re-raise). `fixtures/0.3.1-cancel/task_runner_cancel` records this: the assignee answers and completes, and the runner records the delegation as finished.
4. *A child-start collision.* `WorkflowAlreadyStartedError` at `workflow.py:826` or `:828` is a `FailureError` (`temporalio/exceptions.py:53`) and fails the whole run instead of one node. Any other start failure becomes a `RuntimeError`, which fails the Workflow Task instead (`temporalio/worker/_workflow_instance.py:918-920`).
5. *An AgentWorkflow cancelled while its delegations run.* asyncio passes the cancel only to the handle being awaited. The other started children in `delegation_handles` (`agent_workflow.py:1467`) are terminated when the agent closes. `_cleanup_cancelled_delegations` (`:1518-1566`) settles the task rows and permits but cancels no handle.
- **What termination skips.** AgentWorkflow's `fail_job(cancelled=True)` (employee runtime-v2 leads only, `agent_workflow.py:826-836`), its browser-claim release (`:842-857`), its delegation cleanup, and the cleanup of its own children. Consequences:
- *Browser claims.* A claim left by a terminated agent keeps the profile `BrowserBusy` for every other task (`server/services/browser_owners.py:245-258`). Reset releases it only while the terminated execution is still retained (24 hours on the dev server) and only if it belongs to the generation being reset (`server/services/browser_reset.py:23-45`, `:60-75`). A manual canvas Run's claim, or one older than retention, is never released, and it also defers the profile cleanup of a deleted employee (`server/nodes/browser/__init__.py:94-98`).
- *Subagent permits.* A leaked permit holds one of the root's delegation slots (default limit 3) for up to 3 hours; the next acquire for that root then releases it (`server/core/database.py:4143-4187`).
- *Employee job rows.* A runtime-v2 lead started by MachinaWorkflow records its own failure only from workflow code (`agent_workflow.py:826-829`; `employee_job_workflow.py:28-66`), so its job stays non-terminal; no automatic repair exists. EmployeeJobWorkflow leads are settled by the parent's shielded `employee.job.failed` (`employee_job_workflow.py:121-126`) unless the parent itself is terminated.
- Reset is not on this list: it terminates the whole tree on purpose (`server/services/deployment/handlers.py:974-1017`) after expanding by `RootWorkflowId` (`:862-900`); RFC-0004 H12 and D9 cover its cancel-first variant.
- **Async Lambda: delegations from Claude Code and RLM.** `_execute_delegated_agent` runs a delegated agent as `asyncio.create_task(run_child_agent())` (`server/services/handlers/tools.py:889-890`) under a process-local semaphore (`:78-81`, `:475`).
- From Claude Code, this path is reached only through Task Manager `assign_task` to a non-Coding teammate: the MCP bridge injects `ai_service` and `database` only for `coding_agent`, `taskManager` and `_builtin_check_delegated_tasks` with `pi_harness_version == 1` (`server/services/cli_agent/workflow_tools.py:337-355`). A Claude Code `delegate_to_*` call to any other agent already fails with "Agent delegation requires ai_service and database in config" (`tools.py:401-405`), a separate defect.
- From RLM, both direct delegate tools and Task Manager reach it (`server/services/rlm/adapters.py:269-272`; Task Manager's in-process branch, `server/nodes/tool/task_manager/__init__.py:198-209`).
- A Coding Agent target becomes durable through PiInlineDelegationWorkflow, but only when `pi_harness_version == 1` (`tools.py:378-386`).
- The task runs in the process that handled the call: the API process for the `/mcp/ide` bridge (`server/main.py:406`, `:668`), the worker process of the RLM activity for RLM. It is lost on restart (the status lookup then answers `not_found`, `tools.py:1061-1062`), invisible to Temporal, untouched by Stop and Reset (`server/services/workflow.py:833-858` cancels CLI sessions only), its TeamTask row stays `running` (`server/core/database.py:3698`), and its permit self-heals only after 3 hours.
**Verdict:** Adopted with deviations. Ids, awaited starts, ABANDON for per-event runs and the child-versus-activity split follow the docs. Two deviations: four agent-child starts keep TERMINATE although agents own cleanup (browser claims, permits, job rows), and delegations from Claude Code and RLM run as Async Lambdas.
**Recommendation:**
1. **R1. Agent children get REQUEST_CANCEL.** Pass `parent_close_policy=workflow.ParentClosePolicy.REQUEST_CANCEL` and an explicit `cancellation_type=workflow.ChildWorkflowCancellationType.WAIT_CANCELLATION_COMPLETED` at the six call lines: `workflow.py:826` and `:828` (through `child_start_kwargs`, `:820-822`), `agent_workflow.py:595-598` and `:1740-1744`, `employee_job_workflow.py:117` and `:120`. The Workspace agent child already does this (`node_invocation.py:34-39`).
- Versioning: none. On replay, core compares only a child's id and type (and an activity's id and type); the parent close policy is never compared. The checks are in the core sources shipped in the 1.30.0 wheel, under `temporalio/bridge/sdk-core/crates/sdk-core/src/worker/workflow/machines/`: `child_workflow_state_machine.rs:233-258` and `activity_state_machine.rs:371-394`. The docs agree: changing "The Child Workflow Options in a call to spawn a Child Workflow Execution" is listed among changes "that would not result in non-determinism errors" (https://docs.temporal.io/workflow-definition). The golden identity test is unchanged, because it captures only the `id` and `activity_id` keywords (`server/tests/temporal/workflow_identities.py:36-40`). `test_replay_corpus.py` must stay green.
- When: the cancellation lane (§22.4), together with a re-raise of a cancelled tool call and LLM step in AgentWorkflow (`agent_workflow.py:2574-2578`, `:1260-1264`) under `agent-delegation-drain-v1`. The owner moved it out of Phase 1b on 2026-10-25 (1b.12). Without the re-raise, a plain agent that receives the cancel keeps calling the model and tools after its parent closed, where TERMINATE stops it today. RFC-0004 decided R1 (D9, RFC-0004 §8.4). Effort S, plus the re-raise.
- Risk: a closed parent now leaves its child running its cleanup for a while (see Remarks on id reuse).
2. **R2. Cancel and drain.** On cancellation, MachinaWorkflow cancels every handle in `running` and waits for them, with a bound (sketch); AgentWorkflow's `_cleanup_cancelled_delegations` does the same for every started handle in `delegation_handles` before its database cleanup. With the same patch, node activities get `cancellation_type=WAIT_CANCELLATION_COMPLETED`, so the drain waits for the activity's own cleanup. §6.3 rec 1 adds the failure half of the same patch.
- Versioning: `workflow.patched("machina-close-and-drain-v1")` in MachinaWorkflow, read once before the scheduling loop, and `workflow.patched("agent-delegation-drain-v1")` in AgentWorkflow. The cancellation-type change must ship inside the same patch: a cancellation type is not compared on replay, but it decides whether a cancel emits a RequestCancel command and when the workflow sees the outcome, and core raises a nondeterminism error when a non-ABANDON activity or child records a start or completion after its cancel ("Non-Abandon cancel mode activities cannot be completed after being cancelled", `activity_state_machine.rs:779-781`; the child equivalent at `child_workflow_state_machine.rs:146-149`). Here core is stricter than the workflow-definition page, which lists activity and child option changes as safe. The repo already gated the same change for Workspace nodes behind `workspace-node-cancellation-v1` (`workflow.py:856-861`).
- When: P2, in a cancellation lane after Phase 1 (which ends with the 1.34.0 upgrade and its 1.31 and 1.33 cancellation fixes) and Phase 1b, and before RFC-0005 Phase 4, whose Interrupt and 24-hour no-progress backstop (a default, §24.1; §6.4 rec 2) cancel a turn and expect its agents to clean up. 1b.11 recorded six cancel-path scenarios from 0.3.1 into `server/tests/temporal/fixtures/0.3.1-cancel/`:
- a root cancelled while its agent child runs, uncontrolled (the chat Stop grace path) and controlled;
- an uncontrolled branch failure (a manual Run with two branches, one failing);
- a cancelled Task Manager runner, whose assignee swallows the cancel;
- the same runner cancelled while its generation is stopped, whose assignee ends cancelled;
- a stopped team job cancelled before its lead starts.
The corpus still has no ActivityTaskFailed, because node failures complete with a failure envelope. Effort M.
- Risk: a node that never sees the cancel holds the run until the bound. The drain assumes node activities report cancellation; today `NodeExecutor` turns `CancelledError` into a `{success: False, error: "Cancelled"}` result (RFC-0004 H2), which resolves the handle but records the activity as Completed. 1b.2 (§9.3) fixes H2 in Phase 1b, before this lane.
3. **R3. Durable delegations from Claude Code and RLM** (default chosen; §24 lists the alternative). When Temporal is on, every delegation reached through `_execute_delegated_agent` starts a durable runner instead of an asyncio task: a new workflow type `InlineDelegationWorkflow` that generalizes PiInlineDelegationWorkflow's admission (an admission row with a request digest, a deterministic runner id from root, parent, target and call, `REJECT_DUPLICATE`, `memo` with the digest; `pi_inline_delegation.py:120-230`) to every agent type in `AGENT_WORKFLOW_TYPES`, starts `DelegatedTaskWorkflow` as its child with REQUEST_CANCEL (as `pi_inline_delegation.py:365-369` does), and is started with `EventWorkflowId` so Reset and Apply find it (RFC-0004 H12 notes the Pi inline runner lacks it). It must cover both entry paths: Task Manager's in-process branch and the direct `delegate_to_*` tools; the Claude Code bridge then passes `database` and the trusted scope for those calls too.
- Versioning: none (a new workflow type; old runs never meet it).
- When: P2, an independent lane after Phase 1, before hosted tier S splits workers into a second process (T8): process-local tasks and semaphores do not survive that split. Effort L.
- Risk: admission and permission checks (`assert_runtime_access`) must run on the trusted scope, not on model-supplied arguments.
4. **R4. The Task Manager runner rejects duplicates.** Start it with `id_reuse_policy=WorkflowIDReusePolicy.REJECT_DUPLICATE` (`agent_workflow.py:1878-1881`), like every other do-once start (`server/services/employees/jobs.py:240-243`, `pi_inline_delegation.py:219-226`, `workspace_tasks_workflow.py:118`), and treat `WorkflowAlreadyStartedError` as "already queued". Versioning: the policy needs none; the new exception handling changes commands only after a failed start, so ship it under the next AgentWorkflow patch. When: P3. Effort S. Risk: none known; duplicates are near impossible because runner ids include the iteration and the task id.
5. **R5. Agent children carry `EventWorkflowId`.** Pass `search_attributes=event_workflow_search_attributes(workflow_id)` on agent child starts. Only safe-apply's stop-work sweep then reaches them directly (`safe_apply.py:112-116`); the manager's sweep filters to the two listener types (`server/services/deployment/manager.py:1176-1185`). Versioning: none (an option). When: P3. Effort S. Risk: an agent child may get two cancel requests (direct and from its parent), which is harmless.
6. **R6. A failed child start fails only its node.** Catch `WorkflowAlreadyStartedError` around `workflow.py:824-829` and record it as that node's failure. Versioning: it changes commands only after a failed start, which fails the run today, so put it under `machina-close-and-drain-v1`. When: P2 with R2. Effort S. Risk: only `WorkflowAlreadyStartedError` is caught; other start failures still fail the Workflow Task and stay visible.
**Remarks:**
- R1 and R2 are both needed. REQUEST_CANCEL sends the cancel when the parent closes and does not wait, so the parent's result cannot reflect the child's cleanup. The drain alone does not cover parents that close without running code: termination, or a failure path that skips the handler.
- Replay never re-sends a recorded command. Children started before R1 ships keep TERMINATE; only children started afterwards get REQUEST_CANCEL; an in-flight run can hold both.
- AgentWorkflow already runs its cleanup after catching `CancelledError`, shielding the activities (`agent_workflow.py:826-859`). That is Python's form of "cleanup outside the canceled scope". It runs only when a cancel reaches the workflow while it is not in a tool call or LLM step that turns the cancel into an error result. R1 sends the cancel; the cancellation lane's re-raise makes it reach the cleanup. In `fixtures/0.3.1-cancel/`, `task_runner_cancel` shows the swallow, and `task_runner_paused_cancel` shows an assignee that waits for Resume ending cancelled.
- Id reuse after R1. Per-event roots use ALLOW_DUPLICATE_FAILED_ONLY, so a Cancelled or Terminated root id can run again. If its old agent child is still cleaning up, the new start of `<root id>-agent-<node id>` raises `WorkflowAlreadyStartedError` and, until R6, fails the new run. The controller treats a duplicate start as accepted when the accumulator is on (`workflow_control_workflow.py:468-472`), so repeats are rare; R2's drain closes the window.
- The parent close policy also applies when a parent continues as new. AgentWorkflow's rollover guard refuses to continue while delegations or runner starts are live (`agent_workflow.py:2982-2996`); keep it after R1, or a rollover would cancel live delegations.
- Pending-child limit: only the controller's count follows the event rate. Whether ABANDON children count as pending while the parent runs was not verified; the docs' "incomplete Child Workflows" suggests they do. At 2,000 the start command fails and the workflow task retries until a child closes, stalling the employee's triggers. Not reachable at today's volumes (unmeasured); RFC-0005's one turn per conversation bounds it by design.
- Tests that pin today's policies: `server/tests/temporal/test_agent_workflow.py:394` (runner ABANDON) and `server/tests/temporal/test_browser_task_dispatch.py:28` (REQUEST_CANCEL). Unit tests that assert agent-child start options may need updating.
- If a release ever ships without a needed patch, T14 starts it anyway and marks the affected generations "needs Reset". The patches above keep open runs resumable instead.
**Sketch** (R2 in MachinaWorkflow, SDK 1.30.0; AgentWorkflow's delegation drain has the same shape):
```python
# server/services/temporal/workflow.py, MachinaWorkflow._run_graph
DRAIN_PATCH = "machina-close-and-drain-v1"
_DRAIN_BOUND = timedelta(minutes=2)
drain_v1 = workflow.patched(DRAIN_PATCH) # read once, before the scheduling loop
...
if drain_v1:
# Same patch: a replayed history can only hold a cancel of this activity if it has the marker.
start_kwargs["cancellation_type"] = workflow.ActivityCancellationType.WAIT_CANCELLATION_COMPLETED
...
try:
while True:
... # the existing scheduling loop (workflow.py:647-911)
except asyncio.CancelledError:
if drain_v1 and running:
for handle in running.values():
handle.cancel() # activity: RequestCancelActivityTask; agent child: cancel request, then wait
try:
await workflow.wait_condition(
lambda: all(h.done() for h in running.values()),
timeout=_DRAIN_BOUND, # a node that never sees the cancel must not hold the run
)
except asyncio.TimeoutError:
workflow.logger.warning("Cancel drain bound reached; closing with work still cancelling")
raise # controlled runs then unregister in run()'s finally (workflow.py:503-513)
```
### 6.3 Parallel Execution
**Docs:** https://docs.temporal.io/design-patterns/parallel-execution. Start every independent operation, then wait for the results.
- Python: `asyncio.gather(*[workflow.execute_activity(...) for item in items])`; "Avoid premature blocking. Schedule all Activities before waiting for any results."
- Controlled parallelism: chunks of `max_parallel`, each awaited before the next ("Limit concurrency").
- Errors: `asyncio.gather(..., return_exceptions=True)`; waiting for all "fails on the first error by default"; "Support partial results."
- Limits and pitfalls: at most 2,000 pending activities ("Scheduling more causes Workflow Task failures"), counted separately from children, Signals and cancellation requests; "Blowing the 4 MB gRPC message limit ... if their combined inputs are large. Batch scheduling across multiple Workflow Tasks"; history growth toward 51,200 events; Workflow memory for many futures; "Parallel execution makes tracing more difficult"; "Rate limit".
- Not a fit for dependent operations or strict ordering.
- The in-flight figure "aim for 500" is on the Batch Processing page, not this one (https://docs.temporal.io/design-patterns/batch-processing-patterns).
**SDK:** activity and child handles are asyncio tasks (`temporalio/workflow/_activities.py:61`, `_workflow_ops.py:53`), so `asyncio.gather` works. `workflow.wait` and `workflow.as_completed` are the deterministic replacements for the asyncio versions (`temporalio/workflow/_asyncio.py`, the same in 1.34.0 apart from line endings). Paginated workflow-task completions (SDK 1.33 with server 1.32, behind `history.enableWorkflowTaskCompletionPagination`, default false) are the only server-side relief for the 4 MB limit; the recombined size must still fit, and an over-limit completion then fails the Workflow Task and retries forever instead of terminating the execution. T5 removes the server-version gate, not the size rule. 1.34.0's experimental Event Groups could group a turn's commands in the UI; RFC-0004 Q12 keeps preview features off.
**OpenCompany today:**
- **MachinaWorkflow schedules continuously.** Ready nodes come from `_find_ready_nodes` in graph order (`server/services/temporal/workflow.py:1244-1264`), all start at once (`:674-874`), and completions are taken one at a time by `wait_condition` and an ordered scan (`_wait_any_complete`, `:1101-1133`). This beats the docs' chunks (no wait for a chunk's slowest member) and is deterministic: insertion order, never `asyncio.wait`.
- **No in-flight bound.** `running` grows to every ready node (`:640-874`); only the graph's width limits it.
- **Inputs carry the graph.** Every node input embeds `nodes`, `edges`, the whole `parameter_snapshot` and upstream outputs (`workflow.py:708-724`); agent tool inputs embed them too (`server/services/temporal/agent_workflow.py:2170-2195`). K ready nodes in one task send K copies in one completion. Recorded toy graphs are small: AgentWorkflow start 1,738 bytes and the `node.chatReply.v1` input 1,612 bytes (`server/tests/temporal/fixtures/0.3.1/machina_chat_agent.json`); real sizes are unmeasured. Already RFC-0004 M6 and D7; RFC-0005's turn loads the graph by reference (RFC-0005 §10.2).
- **Three failure policies.** Uncontrolled Temporal runs (manual canvas Runs and generations without execution control) stop at the first failure and abandon running siblings (`workflow.py:907-908`). Controlled runs close admission and let in-flight branches finish, so independent nodes not yet started never start (`:656-657`). The in-process executor cancels the remaining tasks and waits for them (`server/services/execution/executor.py:441-458`).
- **A turn's tool calls run one at a time** (`agent_workflow.py:2035-2369`; the `await _execute_plugin_tool_activity(...)` at `:2363-2369`). Concurrency exists only for Task Manager `assign_task` preflights (`start_activity` plus `asyncio.gather(..., return_exceptions=True)`, `:1903-1992`, the page's error-handling shape) and `delegate_to_*` children. The in-process loop is sequential too (`server/services/agent_runtime.py:359-388`). 39 lines under `server/nodes` declare `"readonly": True` in plugin `annotations` (for example `server/nodes/filesystem/file_read/__init__.py:45`), but `agent.prepare_payload` records no annotations.
- **The delegation window is ordered, not sliding,** although its comment calls it "a bounded sliding window" (`agent_workflow.py:1442-1446`). Handles are awaited in model order. On the uncontrolled path the next delegation starts only after the earliest call finishes without raising (`:2259-2270`, `:2343-2348`); the controlled path starts it after any completion (`:2666-2668`). Most child failures come back as `success: False` results and do start the next one; a child that raises (terminated, cancelled, or failed by an uncaught error) blocks it. The never-started delegation's call then reports `KeyError` (`:2259`, caught at `:2574`). For a team lead, `agent.finish_delegation` is then called for a task that was never queued; it raises "Delegated team task does not exist" three times (`server/services/temporal/agent_activities.py:2383-2384`), and the lead AgentWorkflow fails. Uncontrolled path only.
- **Controller poll loops.** One asyncio task per polling trigger (`server/services/temporal/workflow_control_workflow.py:332`, `:526-528`) catches and logs its own errors (`:849-866`); rollover cancels and gathers them (`:612-615`). This is not the docs' isolation: a cancel that lands while a poll awaits its activity arrives as `ActivityError`, which `except Exception` swallows, so the gather can wait forever and stall the rollover (RFC-0004 H8). Legacy controllers are fully exposed and v1 controllers in a narrower window (§8.3). Under T7, polls leave the controller for new generations.
- Results are ordered deterministically: tool messages in model order, preflight results zipped in order (`:1980-1989`), delegations awaited in order.
**Verdict:** Adopted with deviations. The scheduling loop is better than the docs' sample; the deviations are three failure policies, no bound, inputs that copy the graph, a delegation window that can fail a team lead, and poll loops that swallow a cancel.
**Recommendation:**
1. **One failure policy for graph runs** (default chosen; §24 lists the alternative): when a node fails, running siblings finish and no new node starts. This is what controlled runs already do. Uncontrolled MachinaWorkflow runs adopt it under the same marker as §6.2 R2 (`machina-close-and-drain-v1`): with the marker, a failure sets `ready = []` instead of `break` (sketch). The in-process executor stops cancelling (`executor.py:441-458`) and waits for running tasks without starting new ones.
- Versioning: the MachinaWorkflow change is inside §6.2 R2's patch; the in-process change needs none. Ship both in the same release.
- When: P2, the cancellation lane with §6.2 R2. Effort M.
- Why this default: T4 puts reliability first, and T13 records a side-effecting call that broke off as `unknown` (§11.2, 1b.1). Cancelling running siblings when an unrelated node fails turns known outcomes into unknown ones; starting nothing new stops further side effects.
- Risk: a failed run keeps spending on branches whose result is no longer used.
2. **Graph by reference before any added parallelism.** RFC-0004 M6 and RFC-0005 §10.2 already plan it for turns. Every concurrent command multiplies the embedded graph and parameter snapshot, so items 4 and 5 wait for it. Versioning: the turn's frozen input flag (`event_os_version=1`, RFC-0005 §10.4). When: P2, RFC-0005 Phase 4. Effort M (planned). Risk: activities read the graph from the database, so a generation's snapshot must stay immutable while its runs are open.
3. **Fix the delegation window on the uncontrolled path.** Start the next delegation after any outcome of the one being awaited, failure included, as the controlled path does (`agent_workflow.py:2666-2668`); never call `agent.finish_delegation` for a delegation that was never queued; correct the "sliding window" comment. The same patch refills on any completion (§12.4 rec 3): wait with `workflow.wait(..., return_when=asyncio.FIRST_COMPLETED)` on the started delegations and start the next as soon as any of them finishes, while still appending results in call order. Today both paths start the next delegation only after the one awaited in call order completes (`agent_workflow.py:2343-2348`, `:2666-2668`).
- Versioning: `workflow.patched("agent-delegation-window-v2")` (command order changes after a failed delegation, and whenever a later delegation finishes first).
- When: P2, the cancellation lane, not later: for any team lead it fails the whole lead AgentWorkflow. Effort S. Risk: low. §12.4 rec 3 refers here.
4. **Cap in-flight nodes per run.** Take at most N ready nodes (for example 100, under the batch page's "aim for 500"). It changes commands only for graphs wider than N; gate it with a frozen input flag. When: P3. Effort S. Risk: a very wide graph runs slower; no graph today comes near the bound (unmeasured).
5. **Read-only batches run concurrently** (default chosen; §24 lists the alternative). When every call in one model response is a tool whose plugin declares `annotations["readonly"] is True`, has no approval spec and parsed cleanly, start them together (at most 8) and append results in call order; any other batch runs one at a time as today. `agent.prepare_payload` records each tool's annotations in the prepared tool index, and a per-generation flag, `tool_concurrency_version=1` in the generation's `resource_manifest`, frozen into the run's input, turns the batching on (RFC-0005 §24.1 rule 2).
- Versioning: `tool_concurrency_version=1` (command order changes for such batches). No run-level version is recorded in `agent.prepare_payload`'s result, since T18 leaves rule 2 unamended; only new generations batch.
- When: P3, after item 2. Effort M.
- Risk: the batch is only as safe as the plugins' `readonly` annotations; each concurrent call must also be admitted by execution control (`begin_action`) so Stop's checkpoint counts it.
**Remarks:**
- Keep the FIRST_COMPLETED loop: the docs' chunked "controlled parallelism" makes every chunk wait for its slowest member.
- Several completions in one activation are handled in insertion order, one per loop pass, with new starts in between; replay produces the same order.
- RFC-0004 §8.1 item 3 (replace `_wait_any_complete` with `workflow.wait(..., return_when=FIRST_COMPLETED)`) changes no commands; do not bundle it with a command change.
- Pending activities stay far below 2,000 (graph width; sequential tools). Preflights are one per `assign_task` call in a turn, which the model controls; a per-turn cap (for example 20) is cheap insurance (model behaviour unverified).
- Per-call error isolation already holds in AgentWorkflow: a failed tool becomes an error tool message (`agent_workflow.py:2574-2628`).
- Tracing: activity ids are node ids (`workflow.py:847`) and stable tool ids (`agent_workflow.py:2350-2354`).
- An execution whose task completion exceeds 4 MB is terminated ("the Temporal Service terminates the Workflow Execution", https://docs.temporal.io/troubleshooting/blob-size-limit-error). T6 keeps agent transcripts in LLM-step inputs, but only one LLM step per agent runs at a time; item 5 is where copies multiply.
**Sketch** (items 1 and 5, SDK 1.30.0; helper names are illustrative):
```python
# (1) MachinaWorkflow: one failure policy, under §6.2 R2's marker (drain_v1)
if errors and (self._execution_control.enabled or drain_v1):
ready = [] # start nothing new; running nodes finish
...
if not result.get("success"):
errors.append(error_info)
if not self._execution_control.enabled and not drain_v1:
break # pre-patch histories: abandon running siblings
# (5) AgentWorkflow: a model batch that is entirely read-only runs at once
readonly_batch = (
run_input.get("tool_concurrency_version") == 1 # the generation's flag, frozen at admission
and 1 < len(calls) <= _MAX_READONLY_BATCH # 8
and all(_is_readonly_call(tool_index, call) for call in calls)
)
if readonly_batch:
handles = []
for index, call in enumerate(calls):
info = tool_index[call["name"]]
await self._execution_control.begin_action() # Stop's checkpoint counts each call
handles.append(workflow.start_activity(
_tool_activity_name(info), args=[_tool_payload(index, call)],
activity_id=_tool_activity_id(info["tool_node_id"], iteration, index),
**_tool_options(info)))
outcomes = await asyncio.gather(*handles, return_exceptions=True)
for index, (call, outcome) in enumerate(zip(calls, outcomes)):
self._execution_control.end_action()
_append_tool_outcome(messages, index, call, outcome) # model order; today's error text
else:
... # today's sequential loop
```
### 6.4 Pick First (Race)
**Docs:** https://docs.temporal.io/design-patterns/pick-first. Run several activities that reach the same goal, take the first result and cancel the rest.
- Python: `asyncio.create_task(workflow.execute_activity(...))` per candidate; `done, pending = await workflow.wait([task1, task2], return_when=asyncio.FIRST_COMPLETED)`; cancel each task in `pending`; return `done.pop().result()`.
- Activities heartbeat and re-raise `asyncio.CancelledError`: "Activities do not cancel instantly (they wait for the next heartbeat)".
- Cleanup variant: `cancellation_type=workflow.ActivityCancellationType.WAIT_CANCELLATION_COMPLETED`, then await each pending task, swallowing `CancelledError`.
- Pitfalls: missing heartbeats; "Not waiting for cancellation cleanup"; "Ignoring errors from the winning Activity"; "Forgetting to cancel remaining Activities", which "continue running indefinitely".
- Not a fit when "Activities have side effects that should not be cancelled".
**SDK:** `workflow.wait` returns ordered lists and rejects coroutines ("Passing coroutines is forbidden, use tasks explicitly", `temporalio/workflow/_asyncio.py:89-118`). Handles have `.cancel()`. `ActivityCancellationType` is TRY_CANCEL (default), WAIT_CANCELLATION_COMPLETED or ABANDON (`temporalio/workflow/_activities.py:71-82`), unchanged in 1.34.0, which also de-duplicates a repeated activity cancel command. The worker throttles heartbeats to min(0.8 x heartbeat timeout, 60 s) by default (`temporalio/worker/_worker.py:125-126`); OpenCompany does not override it.
**OpenCompany today:**
- **No race in workflow code:** no `workflow.wait` or `asyncio.wait` in any workflow. The one hit in a workflow module (`server/services/temporal/employee_job_workflow.py:20`) is inside the activity `deliver_employee_job`.
- **In-activity races follow the docs.** `_until_chat_run_stops` races the model call against a 1-second database stop watcher, cancels the loser in `finally` and gathers both, so the winner's error surfaces through `task.result()` (`server/services/temporal/agent_activities.py:419-444`). Its docstring states the docs' trade-off: "a heartbeat-borne cancel arrives too late to stop a stream mid-sentence". `_await_with_llm_heartbeats` (`:378-398`) and `deliver_employee_job` (`employee_job_workflow.py:11-27`) clean up the same way.
- **Chat Stop** is a cooperative race against a grace timer: the run moves to `stopping`, the model step returns its partial answer and unstarted tools are skipped; 45 to about 75 seconds later the watchdog cancels the run's workflow (`server/services/chat/ledger.py:598-639`, `:714-728`). The shape is right; the cancel reaches only the root, whose agent child is then terminated (§6.2).
- **The chat run timeout cancelled nothing** (until 1b.14, e6fb360a, which cancels the run's workflow first). A running run past `max_running_s` (86,400 s) is failed as `timed_out` (`ledger.py:766-767`), and `_sweep_one` is not even given `temporal_cancel` (`:701-703`; compare `:722-726`). `is_stopping` matches only `stopping` and `stopped` (`:642-645`), so the LLM stop watcher and the tool skip never see the `error` run, and `post_reply` still writes for a terminal run (`:334-370`). The workflow keeps running, may keep billing model calls, and the freed lane admits a second run on the same conversation: the page's "Forgetting to cancel remaining Activities", at run level. Detection takes 24 hours plus up to one 30-second sweep; the clock restarts at Resume and paused controlled runs are skipped (`:742-765`).
- **Uncontrolled failure:** the first failure "wins" and siblings are not cancelled (§6.3).
- **Pi driver, the docs' cleanup variant:** `_await_request` waits for "operation recorded or driver done"; `_stop_driver(cancel=True)` cancels the driver activity, which runs with WAIT_CANCELLATION_COMPLETED, and awaits it (`server/services/temporal/pi_workflow.py:180-216`).
- **Not races, correctly:** the Undo window is settled by a database claim at the approval revision, and a started send is never cancelled (`server/services/temporal/approved_tool_call_workflow.py:41-75`), as the page advises for side effects. Workspace Take over is a process-local lease (`server/nodes/browser/_live_control.py:300-352`, `server/nodes/mobile/_control.py:30-50`). `wait_for_delegation` lets only the waiter time out and never cancels the child (`server/services/handlers/tools.py:946-976`).
- **Planned in RFC-0005:** turn deadlines per class (RFC-0005 §13.1: 15, 15, 30 and 60 minutes), the turn as a child with REQUEST_CANCEL and WAIT_CANCELLATION_COMPLETED (RFC-0005 §10.1), Interrupt that cancels a turn "at its next merge point" with its inputs returned to pending (RFC-0005 §13.4), and "Turn hangs" (RFC-0005 §23). Missing:
1. Turn n+1 must not start until turn n's handle is done, or two turns share one conversation key (I4).
2. A bound on that wait, and a last resort when cleanup hangs. A workflow can only request cancellation of another workflow; termination needs an activity that calls the client.
3. The winner may be a failure: a failing turn and a firing deadline in one activation need a deterministic rule.
4. A turn's clocks must stop while the generation is paused (the chat watchdog restarts its window at Resume; Temporal timers keep running through a pause, `workflow.py:148-155`) and while the turn is parked (T11). That makes each an updatable timer. T19 also turns the per-class deadlines into "slow turn" alerts; cancelling is left to a 24-hour no-progress backstop, a default (§24.1; recommendation 2).
5. Cancelling a turn helps only if the turn passes the cancel to its agents (§6.2 R1 and R2).
6. A cancelled turn's agent can still finish a model step and save after turn n+1 started; the consumption transaction must check that the saving turn still holds the conversation.
**Verdict:** Adopted with deviations. In-activity races follow the docs; the run-level timeout left its loser running until 1b.14. The RFC-0005 turn as a child follows the docs; its turn-cancel race is Planned in RFC-0005 (deviates) on the six points above.
**Recommendation:**
1. **A chat run that hits its 24-hour limit cancels its workflow** (default chosen; §24 lists the alternative). Done (1b.14, e6fb360a). Pass `temporal_cancel` to `_sweep_one` and, in its `timed_out` branch, cancel `run.temporal_workflow_id` and `run.temporal_run_id` as `_sweep_stopping` does (`ledger.py:714-728`). This 24-hour limit covers today's chat runs; time waiting on an approval does not count (RFC-0007 D2, §4.7). RFC-0005 turns are bounded by the 24-hour no-progress backstop (a default, §24.1; recommendation 2), which never counts time parked, paused or waiting for the owner; from RFC-0005 Phase 5 (`scheduler_version=1`) a parked turn waits at most 7 days (§11.8; T18, T19).
- Versioning: none (client code).
- When: Phase 1b (1b.14). Until the cancellation lane ships 1b.12, the server terminates the root's agent children when the cancelled root closes; the lane's re-raise then stops a plain agent on the cancel. Effort S.
- Why this default: T4. A run that keeps going after its run row ended is a second writer on the conversation and an unbounded spend.
- Risk: an answer that would have arrived after 24 hours is lost; the owner sees `timed_out`, as today.
2. **Specify the turn's clocks and its cancel race in RFC-0005** (RFC-0005 §10.2, §13.1, §13.4, §18.1, §23; sketch):
- T19: the per-class turn deadlines (15, 15, 30 and 60 minutes for C1 to C4) become "slow turn" alerts. The entity raises an attention item and a metric, and cancels nothing.
- A backstop cancels a turn after 24 hours with no completed step (default chosen, §24.1). It never counts time the turn is parked, paused or waiting for the owner (an approval draft, the owner's browser). Each completed step restarts it.
- Both clocks are named deadlines on §8.4's one live timer, and both stop while the generation is paused or the turn is parked, so parked time never counts toward either. Parking exists only for generations with `scheduler_version=1` (RFC-0005 Phase 5, T18); a parked turn waits at most 7 days, then its input is dead-lettered with an attention item (§11.8, T19).
- A turn that finished, even by failing, in the same activation as the backstop wins.
- Otherwise the entity cancels the turn, waits up to `_TURN_CANCEL_GRACE` (2 minutes) for it to close, then terminates the turn and its agent children by id through an activity; only then may turn n+1 start. Interrupt uses the same wait.
- Versioning: none (ConversationWorkflow is a new type).
- When: P2, RFC-0005 Phase 4, after §6.2 R2; the parked-time rules arrive with parking in Phase 5 (T18). Effort M.
3. **Fence consumption by the turn's lease.** The transaction that saves a transcript and advances `consumed_seq` (RFC-0005 §9.5 step 5) checks that the saving turn still holds its `sched_leases` row (RFC-0005 §13.3); a cancelled or terminated turn's late save is refused. Versioning: none (a database transaction). When: P2, RFC-0005 Phase 4 design text. Effort S. Risk: the refused step's model spend is already settled; only its transcript is dropped.
**Remarks:**
- Heartbeat delivery bounds every cancel. Node activities heartbeat every 30 seconds from a background task when start-to-close exceeds the heartbeat timeout (`server/services/plugin/base.py:1248-1263`), and the 2-minute heartbeat timeout gives a 60-second throttle. A cancel rides the next heartbeat response, so it reaches a node activity about 0-60 seconds after it is sent (about 30 seconds on average), and an LLM step (1-minute heartbeat timeout, `server/services/temporal/agent_workflow.py:99`) within about 48 seconds. That is why the chat path polls a 1-second stop flag.
- The 24-hour start-to-close (`workflow.py:167`) also ends a wedged node attempt, after which the retry policy may run it again (3 attempts by default). A drain or a turn-cancel wait still needs its own bound.
- Side effects: on Stop, Interrupt or the no-progress backstop, an in-flight send is cancelled like any other activity. T13 then records it as `unknown` (1b.2, §9.3; the redo guard is 1b.1, §11.2), unless its plugin is idempotent or uses a provider idempotency key.
- Record a replay fixture of the chat Stop grace path before changing any cancel path (1b.11).
**Sketch** (the turn's clocks inside ConversationWorkflow: RFC-0005 Phase 4, parking from Phase 5; SDK 1.30.0):
```python
# self._deadlines is §8.4's helper: one live timer for all of the entity's named deadlines.
turn = await workflow.start_child_workflow(
"MachinaWorkflow", turn_input, id=f"turn-{self._conv_id}-{self._turn_no}", # number from admission (§8.2)
parent_close_policy=workflow.ParentClosePolicy.REQUEST_CANCEL,
cancellation_type=workflow.ChildWorkflowCancellationType.WAIT_CANCELLATION_COMPLETED,
)
now = workflow.now()
self._deadlines.set("slow_turn", now + _SLOW_TURN[self._turn_class]) # 15/15/30/60 min: alert only
self._deadlines.set("no_progress", now + _NO_PROGRESS) # 24 h, no completed step (§24.1)
while True:
due = await self._deadlines.wait(turn.done)
if turn.done(): # checked first: a turn that ended, even by failing, wins
outcome = self._turn_outcome(turn)
break
if "slow_turn" in due: # a latency target (T19): attention item and metric, no cancel
self._deadlines.set("slow_turn", None)
await self._report_slow_turn()
if "no_progress" in due:
turn.cancel() # its agents clean up only with §6.2 R1 and R2
try:
await workflow.wait_condition(turn.done, timeout=_TURN_CANCEL_GRACE)
except asyncio.TimeoutError: # a workflow cannot terminate another one: an activity does
await workflow.execute_activity(
"conversation.terminate_turn", {"turn_id": turn.id}, # the turn and its agent children
start_to_close_timeout=timedelta(seconds=60), retry_policy=DEFAULT_ACTIVITY_RETRY)
outcome = "no_progress" # RFC-0005 §23, "Turn hangs": the turn is cancelled, then the retry policy
break
self._deadlines.set("slow_turn", None)
self._deadlines.set("no_progress", None)
self._release_turn(outcome) # only now may turn n+1 start (I4)
# Handlers only move these deadlines; the helper re-arms on the new minimum (§8.4):
# - a completed step (the progress RFC-0005 §18.1's stuck-turn alarm watches): no_progress = now + 24 h;
# - a pause, or a park (T11, from Phase 5): both are removed and their remaining time is kept;
# - a wait for the owner (an approval draft, the owner's browser): no_progress alone, the same way;
# - when the pause, park or wait ends: each is set again at now + its remaining time.
```
## 7. Workflow messaging
Three pages cover how callers talk to running workflows. Terms used below:
- A **Signal** is a one-way message to a workflow. The server records it and returns; the caller gets no answer and the workflow may handle it later.
- An **Update** is a message that the workflow validates, records and answers. A **validator** is a read-only check that runs before the Update is written to history; when it raises, the Update is rejected and leaves no history.
- **Signal-With-Start** (SWS) starts a workflow if none is running under the id and delivers a Signal to it, in one call. **Update-With-Start** does the same with an Update.
- **Continue-as-new** closes the current run and starts a fresh one under the same id with new input, to keep history bounded.
### 7.1 Workflow messaging patterns (overview)
**Docs:** https://docs.temporal.io/design-patterns/workflow-messaging-patterns. A chooser: "You want to send a message without checking whether the Workflow is running": Signal with Start. "You need a result back from the Workflow, with validation": Request-Response via Updates. "You receive many events and want to process them in batches": Event Accumulator. The rules come from the three pattern pages and the concept pages:
- **Order.** In each activation, handlers run before the main method, including the first activation after Signal-With-Start or continue-as-new (https://docs.temporal.io/handling-messages).
- **Exceptions.** A Signal handler that raises `ApplicationError`, or lets an activity exhaust a finite retry policy, fails the whole workflow; any other exception fails the Workflow Task, and "the Workflow will get stuck". In an Update handler, a validator exception rejects the Update; an `ApplicationError` or an exhausted activity or child fails only that Update; in the Python SDK any other exception fails the Workflow Task too (only a `FailureError` or `asyncio.TimeoutError` fails just the Update).
- **Dedupe.** Updates are deduplicated by Update id "per Workflow run"; Signals need an application key. Neither request ids nor Update ids survive continue-as-new.
- **Updates need a worker.** With none polling, "Your request will be retried by the SDK Client indefinitely. You can use asyncio.timeout to impose a timeout" (https://docs.temporal.io/develop/python/workflows/message-passing).
- **Limits.** 10 in-flight and 2,000 total Updates per run; the server suggests continue-as-new at 90% of the 2,000. `history.maximumSignalsPerExecution` defaults to 10,000; the dynamic-config reference says further Signals fail with InvalidArgument, while the event page says the execution is terminated. The docs disagree; either way, continue as new first. Throughput: "below 5/sec per workflow instance" (accumulator page).
- **Closing.** "Complete all handlers before using Continue-as-New"; "Use Continue-as-New from your main Workflow Definition method".
**SDK:** everything this section needs exists in 1.30.0: `Client.start_workflow(..., start_signal=, start_signal_args=, id_reuse_policy=, id_conflict_policy=, priority=, rpc_timeout=)` (`temporalio/client/_client.py:527-565`); `execute_update_with_start_workflow` and `WithStartWorkflowOperation`; `WorkflowUpdateStage`; `workflow.all_handlers_finished`; `HandlerUnfinishedPolicy`; `workflow.current_update_info`; `Info.is_continue_as_new_suggested`, `get_current_history_length` and `get_current_history_size`; `Priority(priority_key, fairness_key, fairness_weight)`. 1.33.0 made continue-as-new from an Update handler fail the Workflow Task (OpenCompany never does it) and moved Update-With-Start to Core's long-poll timeout. 1.34.0 removes the private `start_workflow` keywords `callbacks`, `links` and `request_id` (unused here) and adds `WorkflowAlreadyStartedError.first_run_id` (server 1.32 or later) and `Info.original_execution_run_id`.
**OpenCompany today:**
- **Events are Signals routed through Visibility.** `dispatch.emit` runs one Visibility query per event and sends a plain `on_event` Signal by workflow id (`server/services/events/dispatch.py:155-165`, `:251-258`); it drops the event when the query fails or finds nothing (`:163-165`, `:186-191`). The consumer is the per-generation controller (`server/services/temporal/workflow_control_workflow.py:336-359`).
- **Control Updates with validators** on eight workflow types: the controller, MachinaWorkflow (`server/services/temporal/workflow.py:249-263`), DelegatedTaskWorkflow (`server/services/temporal/agent_workflow.py:500-514`), AgentWorkflow (`:759-773`), EmployeeJobWorkflow (`server/services/temporal/employee_job_workflow.py:78-92`), PiInlineDelegationWorkflow (`server/services/temporal/pi_inline_delegation.py:329-343`), CronTriggerWorkflow (`server/nodes/scheduler/cron_scheduler/_workflow.py:88-101`) and PiCodingAgentWorkflow (`server/services/temporal/pi_workflow.py:49-77`).
- **Updates without validators:** Workspace `submit` and `reset` (`server/services/temporal/workspace_tasks_workflow.py:82-193`); Pi `register_driver`, `submit_operation` and `assistant_ready` (`pi_workflow.py:79-107`); the controller's `pause_admissions` (`workflow_control_workflow.py:255-264`).
- **Update-With-Start:** Workspace tasks only (`server/services/node_invocations.py:44-74`).
- **Other Signals:** the controller's `register_trigger`, `reset` and legacy `pause` and `resume`; Pi `pause` and `resume`, which run an activity (`pi_workflow.py:64-73`); MachinaWorkflow's `on_event`, which buffers into `_matched_events` that nothing reads (`workflow.py:427-447`).
- **Signal-With-Start:** not found in `server/`.
- **`@workflow.init`:** 9 decorators in 8 production files.
- **Request-response outside Temporal:** chat admission (`server/services/chat/ledger.py:252-331`); approval decisions by compare-and-swap on a revision (`server/services/approvals/decisions.py`); Stop and Resume intent rows (`server/services/deployment/handlers.py:2295-2310`); Apply claims, re-driven every 10 seconds by `recover_applies` (`server/services/employees/safe_apply.py:285-289`, `server/services/employees/activation.py:55-72`).
**Verdict:** Adopted with deviations. Control needs an answer, so it uses Updates; events need none and must queue while paused, so they use Signals. That matches the chooser. Delivery is at-most-once through Visibility (RFC-0004 H1, replaced in RFC-0005 Phases 2 and 4), and the client-side rules are missing (§7.3).
**Recommendation:**
1. **Adopt a handler checklist** as rules in §5, for every new or changed handler: (a) Signal handlers only buffer or set flags, and drop a malformed payload with a log line instead of raising; (b) every rejection that needs no I/O lives in a validator; (c) every client Update call runs under a deadline (§7.3 R1, 1b.15) and passes a stable id where it is retried and the handler cannot fail transiently after acceptance; (d) continue-as-new or completion happens only in the main method, after `wait_condition(all_handlers_finished)`, with no `await` between the final buffer check and the close; (e) every Signal payload carries an idempotency key, carried across continue-as-new; (f) Signals and Updates are counted per run, not only events; (g) owner actions (Start, Stop, Resume, Apply, approval decisions, Interrupt) go through intent rows and the deliverer (T10, §7.3 R7), and any other request path that sends an Update (Workspace submit) does so under a deadline and answers "pending" when it expires.
- Versioning: none. The lintable parts were 1b.40 (§6.1 rec 1, §17.18), dropped on 2026-10-25 (§22.2). When: Phase 1b. Effort S. Risk: none at run time.
**Remarks:**
- The accumulator page says "Signal handlers must not call activities or yield to the scheduler"; the Python page allows `async def` handlers that run activities. Both hold: Python permits it, but such a Signal handler fails the whole workflow if its activity exhausts a finite retry policy. Pi `pause` and `resume` are that case (`DEFAULT_ACTIVITY_RETRY`, 3 attempts, `server/services/temporal/_retry_policies.py:62-67`).
- RFC-0004 §8.2 already lists deadlines, stable ids, validators for Pi and Workspace, and counting Signals and Updates; none of it has landed.
### 7.2 Signal with Start
**Docs:** https://docs.temporal.io/design-patterns/signal-with-start. One atomic call starts the workflow if it is not running and delivers the Signal; if it runs, only the Signal is delivered. Python: `client.start_workflow(Wf.run, id=..., task_queue=..., start_signal="add_item", start_signal_args=[...])`. The call returns when the server has accepted the Signal, not when it is processed.
- Best practices: derive the workflow id from the entity; "Implement Signal idempotency. Track processed operation IDs"; "Use WorkflowInit. Initialize state before Signals are delivered (... Python's `__init__`)"; "Handle unbounded execution. Use Continue-As-New"; "Use ALLOW_DUPLICATE_FAILED_ONLY for entity Workflows" (while the same page says both ALLOW_DUPLICATE and ALLOW_DUPLICATE_FAILED_ONLY "work well"); include operation ids; return early on duplicates.
- Pitfalls: no idempotency ("Signals can be delivered more than once (for example, client retries)"); "Unbounded history growth ... will hit the 51,200 event or 10K Signal limit"; "Losing pending Signals on Continue-As-New. Drain all pending Signals before calling Continue-As-New, and pass unprocessed ones as input to the new execution"; expecting a return value. On a Signal that races continue-as-new, "Temporal prevents this race".
- Not a fit for "one-time operations (use REJECT_DUPLICATE policy instead)" or for request-response that needs synchronous confirmation ("use Update with Start").
**SDK:** present in 1.30.0 as listed in §7.1. Each call gets a fresh `request_id` (`temporalio/client/_impl.py:327`) that the caller cannot set, in 1.30.0 or 1.34.0, so the server deduplicates only the SDK's own RPC retries. `id_conflict_policy` is passed through as UNSPECIFIED (`_impl.py:332-335`); the server's default for Signal-With-Start is to use the running execution. `priority` travels with the start request (`_impl.py:355-356`), so it most likely applies only when the call starts a run (the server side was not verified). `rpc_timeout` bounds the single RPC; a deadline error is not retried. Signal-With-Start cannot use eager start. The workflow-side `workflow.signal_with_start_workflow` is experimental; RFC-0004 Q12 keeps preview features off.
**OpenCompany today:**
1. **Controllers correctly avoid Signal-With-Start.** A controller's existence is lifecycle state, so an event must never create one (RFC-0004 §10.1 item 2); `dispatch.emit` signals by id only (`server/services/events/dispatch.py:251-258`).
2. **The controller is started empty, then signalled: the "start, then signal" shape the accumulator page warns about.**
- `_start_controller` starts `WorkflowControlWorkflow` with state, revision and scope but no triggers (`server/services/deployment/handlers.py:614-657`). The deployment manager then sends one `register_trigger` Signal per trigger node (`server/services/deployment/manager.py:952-982`).
- Until a trigger is registered, `on_event` finds no matching spec and drops the event without queueing it (`server/services/temporal/workflow_control_workflow.py:336-359`). Before the first registration the `ControlEventTypes` search attribute is absent, so dispatch treats the controller as match-all and does signal it (`dispatch.py:175-184`, `:217-248`).
- Nothing guards the window. The generation stays `starting` until after `_await_deployment_setup` (`handlers.py:2106-2156`), and producers deliver to `starting` generations: chat answers "now" for `starting` (`server/services/chat_thread.py:72`, `:84-92`), and webhook routes read every row in `WORKFLOW_CONTROL_ACTIVE_STATES`, which includes `starting` (`server/nodes/trigger/webhook_trigger/_routes.py:36-46`). Events in that window are lost.
- A crash between the start and the Signals leaves a live but empty controller. `_converge_interrupted_start` fails the generation when no trigger registered but treats a partial registration as complete (`handlers.py:1708-1745`, the test `if triggers_registered or not graph_has_triggers`); the boot re-arm then registers the rest.
- The controller already supports start-with-state: `run` seeds `triggers` from its input (`workflow_control_workflow.py:488-533`), and for execution-control v1 `@workflow.init` seeds the same carried state before any handler runs (`:62-92`). Every continued run already takes this path.
3. **Legacy controllers seed nothing in `__init__`** (`workflow_control_workflow.py:90-92`), so an `on_event` in the first activation after continue-as-new is dropped. Every generation started on current code is v1 (`server/services/deployment/control.py:160`), and T5's fresh start fails every older generation, which owners Reset and Start as v1. No live path remains after the first pinned release.
4. **RFC-0005 relay (planned, RFC-0005 §9.4).** `client.start_workflow("ConversationWorkflow", id=<entity id>, start_signal="deliver", start_signal_args=[batch], id_reuse_policy=ALLOW_DUPLICATE, priority=Priority(priority_key=<class>, fairness_key=<employee id>))`; at most one Signal-With-Start in flight per stream and 4 per entity per second; `delivered_seq` is committed after acceptance; the entity drops `seq <= applied_seq`, carried across continue-as-new (RFC-0005 §9.5). Ids `conv-<workflow_id>-g<gen>-<lane>-<thread>`; the supervisor `emp-<workflow_id>-g<gen>` is started at Start (RFC-0005 §10.1) and rebuilt in place when lost (rec 5, T20). Passivation: "owner 24 h, customer 30 min, others 10 min", and "the server rejects completion while new Signals are unhandled" (RFC-0005 §11). Subscriptions are compiled "in the same transaction that moves the generation to `running`" (RFC-0005 §9.3).
**Verdict:**
- Today: Missing and beneficial, as start-with-state rather than Signal-With-Start. It gives the atomicity Signal-With-Start provides without letting events create controllers.
- RFC-0005: Planned in RFC-0005 (deviates). It follows the docs on deterministic ids, a stable idempotency key (`seq`, a gap-free high-water mark, which is better than a seen-set), batching, a rate below 5 per second, buffer-only handlers, `@workflow.init`, a continue-as-new trigger well below 10,000 Signals (4,000) and commit after acceptance. It misses the start input after passivation, the Reset race, the instant EG12 measures from, and a deadline on the call.
**Recommendation:**
1. **Start the controller with its triggers in the start input.** The deployment manager builds every trigger spec first and `_start_controller` passes them as `triggers` (sketch). `register_trigger` stays only for the boot re-arm. For a listener id it already knows, it changes nothing today (`workflow_control_workflow.py:327-329`); after 1b.32 (§17.8) it refreshes only that listener's sanitized `parameter_snapshot`.
- Versioning: none in workflow code. Old histories replay with their recorded input; the input shape is the one every continued run already uses. New first runs move their search-attribute upsert and poll starts into the first task, which the golden identity test does not lock.
- When: P2, the controller-hardening lane, with H10's slim carry. It is not needed for EG12: RFC-0005 delivers events only from `running` (RFC-0005 §9.3; rec 3), which comes after the Signals were accepted, and Signals are applied in arrival order. It closes the window where today's Visibility path loses chat messages, webhooks and unscoped events sent to a `starting` controller, and the crash window. Effort M.
- Risk: specs embed the graph and parameter snapshot (RFC-0004 H10). N specs in one start input can exceed the 2 MB payload limit, so Start fails at once instead of at the first rollover (`ControllerContinuationTooLarge`, `:705-711`). Land the slim carry with it. Each spec's snapshot must be the one 1b.31 (§17.8) sanitizes on the Start path, so no secret enters the start input; rec 1 lands after it.
2. **Pass the stream cursor as the entity's start input.** The Signal-With-Start call in RFC-0005 §9.4 passes no workflow argument. Pass only the stream cursor, `consumed_seq` (as `{"applied_seq": consumed_seq}`, §8.2 rec 2), read in the same transaction as the batch, and seed `applied_seq` from it in `@workflow.init` on every start, a restart after passivation included, because the `deliver` handler of a new run runs before `run`. Workflow arguments take effect only when the call starts a run, which is exactly the passivated case. Without it, a restarted run cannot tell consumed inputs from new ones. The turn number is not in the start input: §8.2 rec 2 allocates it in the admission transaction (RFC-0005 §13.3), which no restart can reuse, so turn ids, agent child ids and spend keys stay unique after passivation. Versioning: none (ConversationWorkflow is a new type). When: P2, RFC-0005 Phase 4. Effort S. Risk: the relay must read the cursor in the batch's transaction, or a stale `consumed_seq` re-applies inputs.
3. **Define EG12 by the Start commit.** Compile subscriptions in the transaction that creates the `starting` generation, and hold its streams with reason `starting` until the move to `running` releases them. An event accepted after Start committed is then delivered once the generation runs; today's RFC text would mark it `no_consumer`. Versioning: none (database and router). When: P2, RFC-0005 Phase 2 (design text and router). Effort S. Risk: a Start that fails must release or dead-letter its held streams.
4. **Close the Reset race.** The relay's generation check (RFC-0005 §9.4 step 1) and its Signal-With-Start are not atomic; a Reset committed between them lets ALLOW_DUPLICATE start an entity for a dead generation. `admission.acquire` (RFC-0005 §13.3) also checks the generation, and a dead generation's entity completes without a turn. The alternative is for Reset to take the stream leases. Versioning: none (new types). When: P2, RFC-0005 Phase 4. Effort S. Risk: none known.
5. **State the reuse policies.** ALLOW_DUPLICATE is right for the entity: passivation completes the run, and the page's ALLOW_DUPLICATE_FAILED_ONLY would refuse to restart a completed entity. The supervisor follows T20 (RFC-0005 §10.1):
- A lost supervisor (terminated or failed) is rebuilt in place from the database, keeping chat and Context.
- Start starts it with `REJECT_DUPLICATE` under the new generation's id (RFC-0005 §10.1). Each rebuild passes `id_reuse_policy=WorkflowIDReusePolicy.ALLOW_DUPLICATE_FAILED_ONLY`, so the id can be taken again only after a terminated, cancelled, timed-out or failed run, and `id_conflict_policy=WorkflowIDConflictPolicy.USE_EXISTING`, so a rebuild that meets a live run attaches to it.
- After 3 rebuilds within an hour it stops rebuilding and raises an attention item that asks the owner to Reset.
- Today's v1 controller has no such rebuild: a vanished one fails its generation and only Reset recovers it (`handlers.py:780-835`, `:1912-1914`).
- Versioning: none (new types). When: P2, RFC-0005 Phase 4. Effort M.
- Risk: the policy also accepts a start after a cancelled run, so the rebuild first checks in the database that the generation is still live.
6. **Priority applies at start.** An entity started by a C4 input would keep priority 4 for its own workflow tasks after a C1 input arrives. §13.3 rec 3 settles this: entities are keyed by lane, not by their first input, and turn children carry their inputs' class (RFC-0005 §11 step 5). The fairness key in the same `Priority` is the employee id, which never changes; fairness itself is on from the fresh-start release (T17, §13.4). Versioning: none (a start option). When: P2, RFC-0005 Phase 4, with §13.3. Effort S.
7. **Put a deadline on the relay's calls.** `rpc_timeout=timedelta(seconds=10)` on every Signal-With-Start (Phase 4) and every plain Signal to today's controllers (Phase 2). A timed-out call commits nothing; the relay re-sends and the entity deduplicates by `seq`. Versioning: none (client code). Effort S. Risk: none known.
**Remarks:**
- A client-retried Signal-With-Start that straddles the entity's continue-as-new reaches both runs; only the carried `applied_seq` makes that harmless. The controller's carried `seen_event_ids` play the same role (`workflow_control_workflow.py:695`).
- Passivation: "Temporal prevents this race" is stated for continue-as-new. For completion it is documented elsewhere: when new Signals arrive while a Workflow Task runs and the workflow tries to close, "The Temporal Service fails the Workflow Task and schedules a new one" (https://docs.temporal.io/references/workflow-task-errors, Unhandled Command). It protects the entity only if it checks its buffer after `wait_condition(all_handlers_finished)` and closes with no `await` in between (§7.4 A3). A `deliver` accepted after the close starts a new run, which is correct with ALLOW_DUPLICATE and recommendation 2.
- Signal budget: the 4,000-Signal trigger must count every Signal type (`deliver`, `inputs_consumed`, `turn_suspended`, `wake`) in the handlers; the SDK has no Signal-count API. The relay window (32 inputs or 512 KB, RFC-0005 §9.4 step 3) keeps a stuck entity from absorbing Signals toward the 10,000 server limit.
- EG7 ("10,000 messages ... cost fewer than 1,000 Signals per entity run") holds only for bursts: at one message per second every message is its own Signal. Define it as a burst test: the 10,000 messages are appended before the relay runs, so batches are full (16 inputs, about 625 Signals).
- Testing: run Signal-With-Start against `WorkflowEnvironment.start_local()` with the pinned CLI. Test the passivation race by sending `deliver` while the entity's completing task is in flight; time skipping cannot interleave it. Add an entity history to the replay corpus as soon as Phase 4 records one.
**Sketch** (SDK 1.30.0):
```python
# (1) Today's controller: triggers travel in the start input (caller-only change)
specs = {spec["listener_id"]: spec for spec in await manager.build_trigger_specs(control)}
await client.start_workflow(
"WorkflowControlWorkflow",
args=[{**control_args, "triggers": specs}], # v1 @workflow.init seeds them before any handler
id=control.controller_workflow_id, task_queue=queue, search_attributes=attrs,
)
# (2) RFC-0005 relay: the workflow argument is used only when this call starts a run
seed = {"applied_seq": consumed_seq} # the stream cursor, read in the same transaction as the batch
await client.start_workflow(
"ConversationWorkflow", seed, id=entity_id, task_queue="machina-tasks",
start_signal="deliver", start_signal_args=[batch],
id_reuse_policy=WorkflowIDReusePolicy.ALLOW_DUPLICATE, # passivation completes the run
priority=Priority(priority_key=lane_key, fairness_key=employee_id), # keyed by lane (§13.3 rec 3)
rpc_timeout=timedelta(seconds=10),
)
@workflow.defn
class ConversationWorkflow:
@workflow.init
def __init__(self, seed: dict) -> None:
self._applied_seq = int(seed.get("applied_seq") or 0) # deliver runs before run(): seed here
self._pending: list[dict] = [] # turn numbers come from admission (§8.2 rec 2), not from here
@workflow.signal
def deliver(self, batch: dict) -> None: # buffer only
for row in batch["rows"]:
if row["seq"] > self._applied_seq:
self._pending.append(row)
self._applied_seq = row["seq"]
```
### 7.3 Request-Response via Updates
**Docs:** https://docs.temporal.io/design-patterns/request-response-via-updates. `@workflow.update` plus `@<update>.validator`: the validator runs first and, if it raises, the Update is rejected with nothing written to history; otherwise the handler changes state and returns a typed result, and "The Update is recorded in Workflow history before the response is returned".
- Best practices: "Validate early"; typed exceptions; "Return quickly"; "Track Update IDs only across Continue-As-New ... Update ID deduplication is scoped to a single Workflow Run"; "Set timeouts"; atomic state changes.
- Pitfalls: long operations in the handler ("Offload long-running work to Activities"); exceeding 2,000 total Updates; "Not setting Update timeouts. Without a client-side timeout, the caller blocks indefinitely if the Worker is unavailable"; set "a stable, business-meaningful `updateId` when you want a retried Update-with-Start to attach to an existing in-flight Update"; "Updates require a Worker to be online and responsive. For fire-and-forget operations, use Signals instead".
- Limits: about 2 MB per argument or result; each Update adds events toward 51,200; "a maximum of 10 in-flight Updates per Workflow execution and a maximum of 2,000 total Updates".
- The page's Python sample calls `uuid.uuid4()` inside the handler; workflow code must use `workflow.uuid4()`.
**SDK:** everything is in 1.30.0, including `start_update(..., wait_for_stage=ACCEPTED)` for long handlers and `WorkflowUpdateRPCTimeoutOrCancelledError`. A `WithStartWorkflowOperation` is single-use ("WithStartWorkflowOperation cannot be reused", `temporalio/client/_client.py:1185-1187`), so a retried Update-With-Start builds a new one. One trap, read in the source: `rpc_timeout` does not bound `execute_update`. It bounds each request while the Update is sent, but after acceptance the SDK polls for the outcome without it (1.30.0 `temporalio/client/_workflow.py:840` and `temporalio/client/_impl.py:802`; 1.34.0 `:846` and `:889`), and both loops repeat while the server answers without an outcome. Only `asyncio.timeout` (or `asyncio.wait_for`) around the whole call bounds it. The SDK turns the resulting cancellation into `WorkflowUpdateRPCTimeoutOrCancelledError` whose cause is `CancelledError`, which looks the same as the caller being cancelled; check the timeout scope's `expired()` to tell them apart.
**OpenCompany today:**
- **What follows the docs.**
- Revision-idempotent control Updates whose validators raise non-retryable `ApplicationError` (`server/services/temporal/workflow_control_workflow.py:161-174`, `:196-198`, `:234-253`, `:311-323`; `server/services/temporal/execution_control.py:58-106`), and chain identity checked by `first_execution_run_id` (`server/services/temporal/execution_control.py:72-74`).
- Stable ids for retried control calls: `control:{revision}:admission`, `:checkpoint` and `:release` (`server/services/deployment/execution_control.py:80-81`, `:120-121`, `:150-152`, `:181-183`); reconcile and pause or resume ids (`server/services/deployment/handlers.py:1263`, `:2313-2315`, `:2405`); safe-apply ids derived from a database claim (`server/services/employees/safe_apply.py:107`, `:199`, `:231`, `:254-255`, `:264`).
- Cross-workflow Updates only through activities, as the Python guide requires (`server/services/temporal/execution_control_activities.py:17-27`; `server/services/temporal/pi_activities.py:106-112`).
- Continue-as-new only from main loops after `all_handlers_finished` (`workflow_control_workflow.py:595-716`; `server/services/temporal/workspace_tasks_workflow.py:195-215`; `server/services/temporal/agent_workflow.py:2992-3031`; `server/services/temporal/pi_workflow.py:382-387`); the controller requests rollover from Update handlers instead of continuing there (`workflow_control_workflow.py:588-593`).
- Update-With-Start for Workspace tasks uses `USE_EXISTING` and `@workflow.init`, because "Update-With-Start can invoke a handler before run()" (`server/services/node_invocations.py:52-59`; `workspace_tasks_workflow.py:39-53`).
- Update-id dedupe across continue-as-new is done with business keys, as the page advises: revisions, bounded `submissions` and `resets` carried across the Workspace rollover (`workspace_tasks_workflow.py:43-46`, `:210-215`), and Pi digests in Update ids (`pi_activities.py:263`, `:297-298`).
- **Deviations.**
1. *No deadline on any app-process Update call:* Workspace Update-With-Start (`node_invocations.py:52`); `_update_controller_state` (`handlers.py:708-712`); every call in `transition_generation` (`server/services/deployment/execution_control.py:80`, `:108`, `:120`, `:150`, `:181`, `:187`); safe-apply (`safe_apply.py:107`, `:199`, `:231`, `:254`, `:255`, `:264`). The socket waiter awaits `asyncio.shield(task)` with no timeout (`handlers.py:134-139`). Only `_finish_versioned_transition` handles `WorkflowUpdateRPCTimeoutOrCancelledError` (`handlers.py:1423-1430`); its "bounded RPC poll" comment is wrong without a deadline. Worker-side calls are bounded by their activity's 60-second start-to-close (`server/services/temporal/execution_control.py:210-229`).
2. *Random Update ids on retried calls.* `execution_control.register` and `unregister` pass no `id=` (`execution_control_activities.py:17-27`), so each of the 3 attempts of `DEFAULT_ACTIVITY_RETRY` is a new Update; `transition_generation`'s `unregister_execution` and its final controller `set_control_state` pass none either (`server/services/deployment/execution_control.py:108-111`, `:187-190`). The handlers are idempotent (`workflow_control_workflow.py:219-247`), so this costs history and in-flight slots, not correctness.
3. *Rejections inside handlers.* Workspace `submit` checks admission before and after its lock and `reset` checks its request id (`workspace_tasks_workflow.py:83-97`, `:147-148`), so every refusal ("queue full", "resetting", "admission changed") is an accepted-then-failed Update in history. The caller compensates with a fresh random Update id per submit (`node_invocations.py:246-252`). Pi `register_driver`, `submit_operation` and `assistant_ready` have no validators (`pi_workflow.py:79-107`). RFC-0004 L3.
4. *Update results copy the whole controller status.* Every controller Update returns `self.status()`, including the full `live_roots` map (`workflow_control_workflow.py:369-388`), into the controller's history and again into each child's activity result. A recorded entry costs about 135 bytes with a 25-character id (`server/tests/temporal/fixtures/0.3.1/controller_push.json`); real `<slug>-<label>-<event id>` ids make it about 200. With L live runs, each fired run writes about 2 x L entries. One caller depends on it: after a Resume, the UI's in-flight count walks `live_roots` from the final `set_control_state` result (`server/services/deployment/execution_control.py:25-27`, `:187-202`; `handlers.py:1183-1190`).
5. *Large arguments.* `replace_graph` sends the whole graph snapshot as one Update argument (`safe_apply.py:199`) and the handler copies it into every trigger spec (`workflow_control_workflow.py:303-306`), bounded by the 2 MB event limit (RFC-0004 H10).
6. *Signal handlers that do request-response work.* Pi `pause` and `resume` run `coding.pi_child_control` (30 minutes, 3 attempts) when the Pi run has live Task Manager children (`pi_workflow.py:64-73`; `server/services/temporal/pi_control.py:36-55`); an exhausted activity there fails the whole Pi workflow. Only the legacy run-pinned fan-out for unversioned generations sends them (`handlers.py:915-950`), while Pi runs only in v1 generations, so this is latent. Run pinning also loses the Signal when the target continues as new between listing and signalling (NOT_FOUND counts as "already gone", `:903-912`).
7. *`register_trigger` indexes its payload* (`spec["listener_id"]`, `spec["workflow_type"]`, `workflow_control_workflow.py:325-334`). A malformed spec raises a non-`ApplicationError` and wedges the controller in a Workflow Task failure loop (RFC-0004 M5). The producer is internal (`server/services/deployment/manager.py:971-981`): low likelihood, total impact.
- **Budgets in practice.** A fired push run costs the controller 1 Signal, 2 Updates and about 40 history events; a cron firing costs 4 Updates and about 22 events (fixtures `controller_push.json`, `cron_schedule.json`). The server suggests continue-as-new at 4,096 events or 4 MiB by default, so a controller rolls over after about 100 push runs (200 Updates) or about 190 cron firings (about 750 Updates), far below 2,000. The tightest budget is 10 in flight; whether an eleventh concurrent register fails its run was not verified (the server's error code for that limit is not in the downloaded sources).
- **Owner actions today.** Stop and Resume already write an intent row first: the generation row moves to `pausing` or `resuming` with a new revision, a background task runs the transition (`handlers.py:98-105`), and `_reconcile_control` re-drives a stale one when the next status read or the boot pass calls it (`:1223-1238`). The socket handler then waits for that task (`:134-158`). An approval decision commits by compare-and-swap, then starts the send workflow from the request path and reverts the decision if Temporal is unavailable (`server/services/approvals/handlers.py:139-159`); `recover_sends` restarts sends that never started (`server/services/approvals/reconcile.py:76-89`). A pending approval draft keeps Stop in `pausing`: every node activity, an approval gate included, is admitted as an action (`server/services/temporal/workflow.py:865-868`), reviewed delivery is one too (`server/services/temporal/employee_job_workflow.py:132-136`), and `wait_for_checkpoint` waits until no action is admitted (`server/services/temporal/execution_control.py:114-119`); hence "A checkpoint can remain open on admitted work for months" (`handlers.py:1601-1602`).
**Verdict:** Adopted with deviations. The control plane is the docs' pattern almost verbatim; the gaps are the client-side rules (deadlines, stable ids) and validator placement in two newer workflows. RFC-0005's supervisor, `interrupt_turn` and ApprovalWorkflow `decide` are Planned in RFC-0005 (follows the docs): validators and Updates as the page describes, but RFC-0005 never says who sends them, and sending them from a request path would break its own I2. T10 settles the sender. RFC-0007 (2026-10-25) dropped `decide`: ApprovalWorkflow takes a `nudge` Signal and re-reads the row.
**Recommendation:**
1. **R1. Bound every Update call.** Built narrower in 1b.15 (03504492): the helper is `bounded_update(call, seconds)`, and it bounds the Updates a request waits on. The socket's Stop and Resume answer within 10 s, a Workspace submit waits 15 s and its reset 240 s, and safe-apply waits 30 s. A transition's own Updates stay unbounded, and the 10-in-flight refusal is not mapped. The design follows: a helper `execute_update_bounded` (new `server/services/temporal/update_calls.py`, sketch) wraps each app-process call in `asyncio.timeout`: about 15 seconds for Workspace Update-With-Start, 30 seconds for controller and safe-apply calls. On expiry it raises `UpdatePending`: the durable intent row or the deterministic id makes a retry safe. `_finish_versioned_transition` treats `UpdatePending` like today's pending branch (keep the intent and the closed admission). Safe-apply treats it as "leave the claim for `recover_applies`" and must not run its rollback Updates. The socket waiter in `_await_versioned_transition` waits at most 10 seconds, then answers with the pending snapshot while the shielded task continues. A Workspace submission (`_controller_update`, `server/services/node_invocations.py:44-74`) answers "submitted, still pending: check the task list" when its deadline passes, and also when the server refuses the Update because 10 are already in flight (that error's code was not verified). Submitting again is safe: the run id derives from the caller's `submission_id` (`node_invocations.py:182`), and the controller deduplicates by run id and fingerprint (`workspace_tasks_workflow.py:89-94`).
- Versioning: none (client code).
- When: Phase 1b (1b.15). Effort S.
- Risk: a pending answer instead of a final one; the UI already handles pending control states, and the task list shows a Workspace submission that later succeeds.
2. **R2. Stable Update ids for execution control.** `execution_control.register` and `unregister` pass `id=f"exec:{workflow_id}:{first_execution_run_id}:register"` (or `:unregister`), built from the payload they already carry (`server/services/temporal/execution_control.py:194-205`). In `transition_generation`, `unregister_execution` uses `control:{revision}:unregister:{workflow_id}:{chain}` and the final controller release uses `control:{revision}:release-controller:{epoch}`; the epoch is needed because the loop can send that call more than once for one revision. Retries then attach to the admitted Update instead of queueing a new one.
- Versioning: none (activity body and client code).
- When: P2, the controller-hardening lane. It is not urgent: the handlers are idempotent and the server suggests continue-as-new near 4,096 events, so the 2,000-Update cap is never approached; the cost is extra events and in-flight slots on rare retries. Effort S.
- Risk: a stable id makes the server return the first outcome to every retry in the run. That is safe here: the handlers repeat their validators' checks (`workflow_control_workflow.py:206-253`), and a failure after acceptance (a closing controller, a conflicting chain) is final anyway.
3. **R3. Validators for request-independent checks.** Move Workspace's first `_check_admission` pass and the reset request-id check into validators, and Pi's argument checks into validators. Keep the in-lock re-checks in the handler, and keep random submit Update ids: those checks can still fail after acceptance, and a stable id would make the server return that cached transient failure to every retry. With validators, most refused submits leave no history.
- Versioning: none; validators do not run on replay, and handlers keep their checks for histories that recorded the accepted-then-failed form.
- When: P3. Effort S. Risk: none at run time.
4. **R4. Small acknowledgements from controller Updates.** Return `state`, `revision`, `participant_state`, `participant_revision`, `membership_epoch` and `producers_held`, and leave `live_roots` to the `status` Query. The Resume path must then query `status` before `count_generation_actions` walks the roots, or the in-flight count drops every descendant.
- Versioning: `workflow.patched("controller-slim-update-result-v1")` inside the handlers; whether replay compares Update results was not verified, so assume it might.
- When: P2, the controller-hardening lane, with a fixture recorded under the marker. Effort S.
- Risk: any other caller that reads `live_roots` from an Update result breaks; the Resume count is the only one found.
5. **R5. Harden `register_trigger`.** Validate the spec shape in the handler and drop a malformed spec with an error log line. 1b.32 (§17.8) first adds a snapshot refresh for known listeners to the same handler; the shape check covers that path too.
- Versioning: none: valid specs take the same path, and only histories already stuck in a Workflow Task failure change, which is the docs' remedy ("until the exception is fixed, for example by a code change").
- When: P2, the controller-hardening lane. Effort S. Risk: a dropped spec leaves its trigger unregistered until the next re-arm; the error line is the only signal until RFC-0005's attention items exist.
6. **R6. Remove the dead Signal paths.** Delete Pi `pause` and `resume` together with the legacy pause and resume fan-out (`handlers.py:915-950`), and MachinaWorkflow's unused `on_event` (`workflow.py:427-447`).
- Versioning: removing `on_event` changes no commands (an unhandled Signal is buffered). Removing the Pi handlers does change commands (no more `coding.pi_child_control` after those Signals); it is replay-safe only because no recorded history has such a Signal reaching a Pi run, and after T5's fresh start no unversioned generation can send one.
- When: P3, or Phase 10 with the legacy path. Effort S. Risk: none; no live generation reaches these paths.
7. **R7. Owner actions through intent rows and a deliverer** (T10). Start, Stop, Resume, Apply, approval decisions and Interrupt write a database intent and answer at once; a background deliverer sends the Update by workflow id, with a stable id from the intent (`intent:{id}:{revision}`) and a per-attempt deadline, until the Update is applied, refused or its target is gone; the UI shows "applying" until a broadcast settles it (sketch). For Start and Apply the deliverer drives their steps from the intent row (§10.2).
- Handlers stay idempotent by revision, because Update-id dedupe is per run and the supervisor continues as new daily and is rebuilt in a new run after a loss (T20).
- Long handlers, such as the supervisor's Stop barrier that fans out to every live turn, use `start_update(wait_for_stage=ACCEPTED)` and fetch `result()` later by id.
- T12: Stop completes while an approval draft waits. A decision taken while the employee is stopped is delivered at once, so the owner sees it recorded, and its send waits for Resume. Today that means the send is not started while the generation is paused (1b.5), and the Stop checkpoint stops counting approval waits; §9.6 carries both changes.
- Today's code moves toward it without versioning: Stop and Resume answer after the bounded wait of R1, and a periodic sweep re-drives stale `pausing` and `resuming` rows instead of waiting for the next status read; an approval decision no longer reverts when Temporal is down, because `recover_sends` restarts sends that never started, and since 1b.5 (which absorbed 1b.6) it runs every round of the chat watchdog instead of once per process.
- Versioning: none (client code, and the new supervisor and ApprovalWorkflow types).
- When: P2, RFC-0005 Phases 4 (supervisor, Interrupt) and 7 (ApprovalWorkflow for the gate and delivery; it arrived for held tool calls with RFC-0007 AP.4a), plus the no-versioning steps above with §9.6. Effort M.
- Risk: the owner sees "applying" for as long as Temporal or the target is down; the deliverer must surface a stuck intent as an attention item.
**Remarks:**
- Long handlers are allowed but hold in-flight slots: `wait_for_checkpoint` waits for admitted actions to drain, and "A checkpoint can remain open on admitted work for months" (`handlers.py:1601-1602`). Workspace `reset` waits for child cancellation and a 120-second activity while holding the admission lock (`workspace_tasks_workflow.py:166-191`); a submit that arrives during a reset fails its first admission check at once, and only submits already waiting on the lock when the reset began stay in flight, then fail the re-check.
- If a handler is stuck in a Workflow Task failure, Updates not yet accepted get `FAILED_PRECONDITION` and accepted ones stay durable (Python message-passing guide); the deliverer of R7 keeps retrying both.
- Replay coverage: `controller_stop_resume`, `controller_push`, `controller_poll`, `controller_queue_read`, `controller_spill_rollover`, `workspace_task`, the `pi_*` and `paused_*` fixtures exist. R4 needs a new fixture with its marker; R1 to R3 and R5 change no commands.
**Sketch** (SDK 1.30.0):
```python
# server/services/temporal/update_calls.py (new)
class UpdatePending(Exception):
"""Our deadline passed. The Update may still apply: keep the intent and retry."""
async def execute_update_bounded(handle, update: str, arg: Any, *, update_id: str | None, seconds: float) -> Any:
try:
async with asyncio.timeout(seconds) as scope: # bounds the send loop and the outcome poll
return await handle.execute_update(update, arg, id=update_id)
except WorkflowUpdateRPCTimeoutOrCancelledError as exc:
if scope.expired(): # our deadline, not the caller's cancellation
raise UpdatePending(update) from exc
raise
except TimeoutError as exc:
raise UpdatePending(update) from exc
# T10: the only sender of owner Updates (Stop, Resume, decide, interrupt_turn)
async def deliver_due_intents(database, client) -> None:
for intent in await database.claim_due_intents(limit=50): # leased, oldest first
handle = client.get_workflow_handle(intent.target_workflow_id) # by id, never a run id
try:
result = await execute_update_bounded(
handle, intent.update_name, intent.payload,
update_id=f"intent:{intent.id}:{intent.revision}", seconds=30)
except UpdatePending:
continue # still "applying"; the next pass retries
except WorkflowUpdateFailedError as exc: # the validator or the handler refused it
await database.settle_intent(intent, "refused", error=str(exc.cause))
except RPCError as exc:
if exc.status != RPCStatusCode.NOT_FOUND:
raise
await database.settle_intent(intent, "target_gone")
else:
await database.settle_intent(intent, "applied", result=result)
await broadcast_intent_settled(intent) # the UI leaves "applying"
```
### 7.4 Event Accumulator
**Docs:** https://docs.temporal.io/design-patterns/event-accumulator. One workflow per group key, with a deterministic id ("optionally" with the accumulation period, for example `accumulator-order-123-2026-05-14`); producers always use Signal-With-Start; Signal handlers only append to a buffer.
- Main loop: wait with a sliding inactivity timeout (`workflow.wait_condition(..., timeout=timedelta(seconds=10))`), drain and dedupe by a stable item key, run one batch activity on timeout or `flush`, then complete. When `is_continue_as_new_suggested()` and the buffer is empty, continue as new with the items and the sorted seen keys.
- Best practices: a dedupe key in every payload; "Size the inactivity timeout to your domain's quiet period"; carry items and seen keys into continue-as-new ("If you omit the seen-keys list, any signal delivered to both the old and new run during the Continue-As-New handoff will be processed again"); "Keep signal handlers fast and side-effect free"; a flush Signal "for testing and operational runbooks"; "Keep producer signal rate below 5/sec per workflow instance"; the 10,000-Signal limit per run (stated "on Temporal Cloud"; self-hosted servers have the same default).
- Pitfalls: random ids; "Calling start followed by a separate signal"; no dedupe key; a timeout too short; forgetting state at continue-as-new; activities in handlers; "Assuming workflow completion means all events were captured"; "Signal rate too high to allow Continue-As-New to complete", which needs "a brief window (~100 ms) with no unhandled signals"; "Not draining the signal queue before calling Continue-As-New".
- The page sets no maximum on the window: with arrivals faster than the timeout, a pure sliding window never flushes. Its only bounds are the explicit flush and a period encoded in the id.
- **The Python sample is wrong for the Python SDK.** It reads `wait_condition` as returning a bool (`timed_out = not await workflow.wait_condition(...)`). In 1.30.0 and 1.34.0 it returns `None` and raises `asyncio.TimeoutError` on timeout (`temporalio/workflow/_context.py:920-943`; 1.34.0 `:1016-1049`), and an `asyncio.TimeoutError` that escapes `run` fails the workflow (`temporalio/worker/_workflow_instance.py:1824-1839`). As written, the sample flushes on the first wake-up and fails on the first quiet window. Catch `asyncio.TimeoutError` instead.
**SDK:** all in 1.30.0: `wait_condition` with a timeout, `is_continue_as_new_suggested`, `continue_as_new(args=...)`, `all_handlers_finished`. The 1.34.0 docstring adds that switching a timeout between `None` and a positive value is a nondeterministic change and that `0` "should not be used"; under CPython 3.12 a timeout of 0 or less raises at once without a timer.
**OpenCompany today:**
- **WorkflowControlWorkflow is the accumulator, keyed by deployment generation.**
- *Follows the docs.* `on_event` only buffers and dedupes on `listener_id:event_id` (`server/services/temporal/workflow_control_workflow.py:336-359`). The last 4,096 seen keys are kept (`:32`, `:535-539`), and queued keys stay deduplicated after they age out (`:72-76`, `:353-354`, `:556-566`). Continue-as-new carries the queue head (256 events or 512 KB), the seen keys and the triggers' poll baselines (`:684-704`). Larger queues spill to the database, deduplicated by `sha256(controller:listener:event_id)` (`server/services/temporal/controller_queue.py:9-40`), and are read back idempotently by request id (`:43-71`). Before continuing as new, the v1 path re-spills new arrivals, waits for `all_handlers_finished` and re-checks with no await before the close (`workflow_control_workflow.py:647-665`): the page's drain-and-recheck guard. A paused controller keeps accepting and rolls over with its queue (`:355-359`). Duplicate child starts count as accepted (`:468-472`).
- *Not a batch accumulator.* The main loop takes one event at a time and starts one child run per event (`:431-483`). For each event, the controller's own history records the trigger filter as an activity when the trigger has filter parameters (`server/services/temporal/trigger_listener_workflow.py:337-343`, `:176-178`), two broadcast activities (`:425-434`, `:503-512`) and the child start (`:494-497`): about 40 events per fired event, and about 10 per event the filter rejects.
- *Coarse group key.* Telegram and WhatsApp events are unscoped (`server/nodes/telegram/_events.py:160`; `server/nodes/whatsapp/_events.py:193`) and reach every controller subscribed to the type (`server/services/events/dispatch.py:175-184`); webhook events are scoped to the workflows that declare the path (`server/nodes/trigger/webhook_trigger/_routes.py:95-96`). Every message on a busy account becomes a Signal, plus a filter activity, in every subscribed employee's controller, against the 5-per-second guidance.
- *A poison event blocks the queue.* Any spawn failure other than `WorkflowAlreadyStartedError` puts the event back at the head and sleeps 5 seconds, forever (`workflow_control_workflow.py:464-480`; RFC-0005 §4.2). A filter activity that fails deterministically qualifies.
- *Carry budget.* Seen keys are `<slug>-<label>:<event id>` strings of about 60-85 bytes with JSON quoting, so 4,096 keys take 0.25-0.35 MB of the 1.9 MB carry limit (`:38`, `:695`). Above the limit, `ControllerContinuationTooLarge` fails the controller (`:705-711`).
- *Legacy gaps.* Runs without execution control or without the accumulator marker never fence handlers before continue-as-new (`:605-609`, `:682-683`), and Signals that arrive during the final spill can be marked seen but neither spilled nor carried (`:666-669`). The legacy listener resets its seen ids at continue-as-new and carries only the newest 256 events (`trigger_listener_workflow.py:123-127`, `:291-303`). No generation started on current code takes these paths, and T5's fresh start fails every older one.
- **RFC-0005 (planned).** Debounce with a sliding window plus an absolute cap: owner chat 0.4 s / 1.5 s, customer chats 2 s / 8 s, groups 3 s / 10 s, others none, with "`workflow.wait_condition` with a timeout" driving it (RFC-0005 §11). Relay window of 32 inputs or 512 KB, batches of 16 inputs or 256 KB, 4 Signal-With-Start calls per entity per second (RFC-0005 §9.4). Entity continue-as-new "only between turns, when suggested or past 4,000 events or 4,000 Signals", carrying `applied_seq`, `turn_no` and at most 256 KB of pending inputs, "any excess goes back to the database as `hold(window)`" (RFC-0005 §10.2). Partitioning by conversation stream (RFC-0005 §12), matching and filters in the database before any Temporal call (RFC-0005 §9.3), dedupe in `bus_dedupe` for 7 days (RFC-0005 §9.2).
**Verdict:**
- Today: Adopted with deviations. Buffering, dedupe, carry and drain match the docs, and the spill goes beyond them. It does not batch, it filters inside history, it has no bound on a poison event, and its group key, a whole employee, is coarser than the docs' "partition by a finer-grained key".
- RFC-0005: Planned in RFC-0005 (follows the docs). It improves on the page twice: the absolute cap fixes the starvation of a pure sliding window, and a gap-free `seq` high-water mark is a bounded dedupe key. The gaps are A3 to A6 below.
**Recommendation:**
1. **A1. Dead-letter poison events in today's controller.** Built with §11.6 rec 2 in 64fe733b (2026-11-05), dormant for one release (§18.9 rec 2): the dead row names the exception's type in `_dead_reason` and is chunked like the rollover spill past 512 KB, and Reset deletes the generation's dead rows with any chunk rows a stopped spill left (`controller_queue.clear_dead_letters`); deletion already did. The design follows. §11.6 rec 2 owns this change: the dead letter ships in one patch with its spawn backoff (5 s, 30 s, 2 min, 10 min). The dead-letter details: count spawn attempts per event, carried with the event. After 5 failures that are not duplicate starts, write the event to dead-letter rows and move on; the dead-letter write can reuse `controller.queue.spill` with `controller_id=f"{controller_id}:dead"`, which `controller.queue.read` never reads back, so no schema change is needed (sketch). Reset and workflow deletion also clear those rows; RFC-0005 Phase 8 adds the attention item.
- Versioning: `workflow.patched("controller-spawn-backoff-v1")`, the one marker for the backoff and the dead letter, specified in §11.6 rec 2.
- When: P2, the controller-hardening lane, next to C1, H8 and H10. T5 does not remove the need: generations started after the first pinned release and before Phase 4 run on this controller until Phase 10. Effort M.
- Risk: an event that would have succeeded on a sixth try is dead-lettered; the dead row keeps it for replay.
2. **A2. Carry seen keys as digests.** Use the first 16 hex characters of `sha256(listener_id:event_id)` as the dedupe key everywhere, and convert carried raw keys when `_seed_carried_state` loads them: about 75% less of the carry budget. Keep carrying keys for spilled events too: the spill table forgets an event once `controller.queue.read` deletes its rows, after which only the carried key stops a producer retry.
- Versioning: none. Dedupe decisions and commands stay the same, and continue-as-new inputs are not compared on replay; prove it by replaying `controller_spill_rollover`.
- When: P2, the controller-hardening lane, folded into H10's slim carry. Effort S.
3. **A3. Specify the entity's close sequence** (RFC-0005 §10.2, §11 Passivation): `wait_condition(all_handlers_finished)`, re-check the pending buffer, no `await`, then complete or continue as new. Returning inputs to the database needs an activity, which is an `await`, so loop as the controller does (`workflow_control_workflow.py:651-665`). Versioning: none (a new type). When: P2, RFC-0005 Phase 4 (design text). Effort S. Risk: under a sustained flood the loop may not converge; the relay's rate cap is what lets it.
4. **A4. Carry cursors, not payloads.** When inputs above the 256 KB carry go back to the database, the carried `applied_seq` must be the highest seq actually retained; otherwise the re-delivered excess has `seq <= applied_seq` and the new run drops it as a duplicate. Simpler and closer to I1 ("The inbox and its cursors are the truth"): carry `consumed_seq`, `turn_no` and the debounce anchors only, set `applied_seq` to the consumed cursor, and have the relay rewind `delivered_seq` to it with one activity before the close. Versioning: none (a new type). When: P2, RFC-0005 Phase 4 (design text). Effort S. Risk: inputs delivered but not consumed are sent twice; the entity's `seq` check keeps that harmless.
5. **A5. Drive debounce by an absolute deadline.** `deadline = min(last_arrival + window, first_arrival + cap)`, then `wait_condition(new_input, timeout=deadline - workflow.now())` in a loop, returning when the remaining time is zero or less (sketch). `wait_condition` fixes its duration when called, and the Updatable Timer page says to work from absolute timestamps. In the entity, this deadline is one of the named deadlines that §8.4's shared helper keeps on one live timer. Carry `first_arrival` (RFC-0005's "debounce anchors"). Versioning: none (a new type). When: P2, RFC-0005 Phase 4. Effort S. Risk: none known.
6. **A6. No debounce for owner chat** (T9). An owner message starts its turn at once; messages sent while it runs merge at the agent's next step (B1), or start the next turn at once when the turn has no next step (RFC-0005 §11 step 6). The customer windows stay. The timers page says a Timer may be "as short as one second" and that workflows "should not rely on sub-second accuracy for Timers" (https://docs.temporal.io/workflow-execution/timers-delays); the SDK accepts shorter delays, so this is guidance, not an enforced floor. Of RFC-0005's owner values only the 0.4 s window was below one second; T9 removes both. Versioning: none. When: P2, RFC-0005 Phase 4 (RFC-0005 §11 table, §22 (c) and EG5). Effort S. Risk: two owner messages a second apart can make two turns; the owner accepted that for lower latency.
7. **A7. Add a flush message.** `wake(flush=True)` ends a debounce early, for tests and runbooks, as the docs recommend, so EG5 is testable without waiting real time. Versioning: none. When: P3. Effort S.
8. **A8. Keep each controller below 5 Signals per second in Phase 2.** The relay that delivers to today's controllers sends several events per Signal through a new `on_events` handler that takes a list, or limits each controller's rate. Versioning: none (adding a Signal handler changes no commands). When: P2, RFC-0005 Phase 2. Effort S. Risk: none known.
**Remarks:**
- Rate: the docs' 5 per second is per workflow. The relay's 4 calls per second per entity, with up to 16 inputs each, satisfies it; today's controllers do not, because one controller absorbs a whole employee's traffic.
- The continue-as-new quiet window (about 100 ms with no unhandled Signals) is protected in the entity by between-turn continue-as-new and the relay's rate cap. In today's controller the spill-and-recheck loop converges only if arrivals pause briefly; under a sustained flood each pass adds spill activities to history.
- "Assuming workflow completion means all events were captured": with passivation and ALLOW_DUPLICATE, a late `deliver` starts the next run, as intended. Without the cursor input (§7.2 recommendation 2) the new run cannot tell consumed inputs from new ones.
- Testing: windows of seconds need the time-skipping environment (`WorkflowEnvironment.start_time_skipping()`, RFC-0004 §14). Sub-second windows are not meaningfully testable with time skipping, one more reason for A6.
**Sketch** (SDK 1.30.0):
```python
# (A1) WorkflowControlWorkflow.run, the spawn-failure branch (workflow_control_workflow.py:464-480).
# The dead-letter arm of §11.6 rec 2's patch; §11.6's sketch shows the backoff arm.
SPAWN_BACKOFF_PATCH = "controller-spawn-backoff-v1"
_MAX_SPAWN_ATTEMPTS = 5
except Exception as exc: # noqa: BLE001 - per-event isolation
if accumulator_v1 and isinstance(exc, WorkflowAlreadyStartedError):
accepted = True
elif queue_v2 and workflow.patched(SPAWN_BACKOFF_PATCH):
attempts = int(event.get("_spawn_attempts", 0)) + 1
if attempts >= _MAX_SPAWN_ATTEMPTS:
await workflow.execute_activity(
"controller.queue.spill",
{"controller_id": f"{workflow.info().workflow_id}:dead", # never read back
"events": [[listener_id, {**event, "_dead_reason": type(exc).__name__}]]},
start_to_close_timeout=timedelta(seconds=60), retry_policy=QUEUE_PAGING_RETRY)
accepted = True # the dead row is the record; the queue moves on
else:
event = {**event, "_spawn_attempts": attempts}
... # §11.6 rec 2: behind the rest of the queue, after its backoff
elif queue_v2 and not accepted: # recorded histories: today's head-of-line loop
self._events.insert(0, (listener_id, event))
self._durable_prefix += 1
await workflow.sleep(timedelta(seconds=5))
# (A5) ConversationWorkflow: sliding window with an absolute cap; deliver() bumps self._arrivals
async def _debounce(self, window: timedelta, cap: timedelta) -> None:
first = self._first_arrival_at # carried across continue-as-new
while True:
seen = self._arrivals
remaining = min(self._last_arrival_at + window, first + cap) - workflow.now()
if remaining <= timedelta(0): # a timeout of 0 raises at once and arms no timer
return
try:
await workflow.wait_condition(lambda: self._arrivals != seen, timeout=remaining)
except asyncio.TimeoutError: # wait_condition returns None; a timeout raises
return
```
## 8. Entity and lifecycle
Three patterns keep long-lived workflows healthy: one workflow per business entity, continue-as-new to bound its history, and timers whose deadline can move. OpenCompany uses the first two today (the deployment controller, the Workspace controller, `AgentWorkflow`), and RFC-0005 adds more entities (supervisor, conversation, approval). The third exists only in RFC-0005's plan.
### 8.1 Entity and lifecycle patterns (category overview)
**Docs:** https://docs.temporal.io/design-patterns/entity-lifecycle-patterns. A selection guide with no API content of its own. It maps three needs to three patterns:
- "You are modeling something with an ongoing lifecycle": use an Entity Workflow "as the single source of truth for that entity".
- "Your Workflow runs long enough to grow a large history": apply Continue-As-New "to reset history while preserving state".
- "You need a wait that responds to new information": use an Updatable Timer "instead of a fixed sleep".
It points to the messaging patterns for the Signals and Updates that drive state changes, and to task orchestration to "decompose a large entity into child Workflows".
**OpenCompany today:**
| Need | Today | RFC-0005 |
|---|---|---|
| An entity with a lifecycle | `WorkflowControlWorkflow`, one per deployment generation; `WorkspaceTaskControllerWorkflow`, one per workflow; legacy `TriggerListenerWorkflow` and `PollingTriggerWorkflow`, one per (deployment, trigger) | `EmployeeSupervisorWorkflow` per employee generation, `ConversationWorkflow` per conversation stream, `ApprovalWorkflow` per approval (RFC-0005 §10.1) |
| A history that grows | the controller, `AgentWorkflow` and Pi, the Workspace controller, legacy listeners | supervisor, conversation entity, `AgentWorkflow` (RFC-0005 §10.2) |
| A wait whose deadline moves | none in workflow code | alarms, debounce, idle passivation, slow-turn alerts (T19), the no-progress backstop (a default, §24.1), restart backoff, budget period end (RFC-0005 §10.2, §11, §15) |
| Splitting an entity into children | controller to `MachinaWorkflow` (ABANDON) to `AgentWorkflow`; Workspace controller to `NodeInvocationWorkflow` (REQUEST_CANCEL) | entity to turn child (REQUEST_CANCEL) to `AgentWorkflow` |
**Verdict:** Not applicable (selection guide). One structural deviation is deliberate and should be stated once: the app database, not workflow state, is the source of truth (RFC-0005 I10). Chat runs, approvals, employee jobs and budgets are database rows changed by compare-and-swap, which is the page's "Database + Locks" row. That is justified. A Temporal database can be replaced by an empty one (RFC-0005 E16, confirmed by T5). Closed histories are kept 24 hours on today's dev server and 7 days in RFC-0005's plan (RFC-0005 §25.1), so history cannot be the audit trail.
**Recommendation:**
1. Add two rules to §5 (rules every workflow follows):
- Any counter whose value names something outside the run (turn numbers, sequence cursors, any id that feeds an idempotency or spend key) lives in the app database, or is derived from it. It is never kept only in workflow state carried by continue-as-new.
- Every entity start (first start, restart after passivation, rebuild after a failure) seeds its state from the database.
Versioning: none (text). When: with this RFC. Effort S.
**Remarks:** The docs' "single source of truth" also means "audit trail in history". OpenCompany keeps its audit trail in the database instead: `approval_decisions` today, and `employee_timeline` from RFC-0005 Phase 8 (RFC-0005 §18.4). §8.2 gap a shows what breaks when the first rule is missed, and item 2 fixes it.
### 8.2 Entity Workflow
**Docs:** https://docs.temporal.io/design-patterns/entity-workflow. One workflow per long-lived business entity (an account, a device, an order), "using the entity ID as the Workflow ID", holding the entity's state in workflow variables for its whole life. Rules that matter:
- Use Updates (synchronous messages the workflow validates and answers) for operations "that require validation and a return value", Signals (asynchronous messages) "for asynchronous notifications that do not need responses", and Queries to "Expose state for monitoring and debugging".
- Handlers only change state. "All SDK docs explicitly warn: do not call Continue-As-New from Update or Signal handlers." The main method decides, after `workflow.all_handlers_finished()`.
- "Losing Update ID deduplication across Continue-As-New ... Carry processed IDs in the Continue-As-New input": the server deduplicates Update ids within one run only.
- "Keep state minimal. Store large data externally and reference it in the Workflow."
- "Handle deletion. Implement an explicit deletion or decommission Signal."
- "Set timeouts. Use Workflow execution timeout as a safety net." "Version carefully. Use Worker versioning."
- Fit and cost: not for "high-frequency updates (more than 100 per second per entity)"; "One Workflow per entity means you should consider costs at scale"; "The first operation after an idle period may have latency".
- Do not copy the page's Python sample: it calls `datetime.utcnow()` in workflow code (use `workflow.now()`), and after the handler fence it continues as new without re-checking `deleted`, so a delete that lands during the fence is lost.
**SDK:** Everything is in 1.30.0:
- `@workflow.init`, `@workflow.update` with `.validator`, `@workflow.signal`, `@workflow.query`, `workflow.wait_condition`, `workflow.all_handlers_finished`, `workflow.info().is_continue_as_new_suggested()`, `workflow.continue_as_new`, `workflow.current_update_info()`.
- Signal-With-Start (start the workflow if needed, then deliver a Signal, atomically) through `Client.start_workflow(start_signal=...)`; Update-With-Start through `execute_update_with_start_workflow`.
- `QueryRejectCondition.NOT_OPEN` (`temporalio/common.py:338-346`) and `WorkflowQueryRejectedError`, for a Query that must not answer from a closed run.
- Order inside one activation: the worker applies patches, then Signals and Updates, then everything else, and builds the workflow object on the first pass (`temporalio/worker/_workflow_instance.py:441-472`, `:2532-2540`). So handlers can run before `run()`, and state they need belongs in `@workflow.init`.
- 1.34.0 adds one rule: continue-as-new from an Update handler fails the workflow task (1.34.0 `worker/_workflow_instance.py:777-783`). OpenCompany never does this: all six `workflow.continue_as_new` calls sit in main loops.
**OpenCompany today:**
- **Deployment generation, `WorkflowControlWorkflow`.**
- Id `workflow-control-<workflow_id>-g<generation>` (`server/services/deployment/control.py:147`). Started at Start with the default id reuse policy and an `EventWorkflowId` search attribute (`server/services/deployment/handlers.py:614-661`), and addressed by workflow id only (`:664-680`).
- Messages (`server/services/temporal/workflow_control_workflow.py:94-388`): Signals `pause` and `resume` (legacy), `reset`, `register_trigger`, `on_event`; Updates `set_control_state`, `wait_for_checkpoint`, `register_execution`, `unregister_execution` and `replace_graph` with validators, and `pause_admissions` without one; Queries `status` and `execution_control_status`. The `reset` Signal closes it (`:274-280`, `:398`, `:438-439`).
- A vanished controller. Only unversioned (legacy) generations are paused and later rebuilt with `WorkflowIDConflictPolicy.USE_EXISTING` (`handlers.py:788-825`). Every generation started on current code has `execution_control_version=1` (`control.py:160`). A vanished v1 controller fails its generation with `controller_execution_missing` (`handlers.py:826-840`, `:1913-1914`), and only Reset, which starts a new id, recovers it. That is what E16 prescribes. For RFC-0005's supervisor, decision T20 rebuilds a lost one in place and asks for Reset only after 3 rebuilds within an hour (item 5). CLAUDE.md's "Resume rebuilds the controller" is stale for v1 generations.
- **A workflow's Workspace, `WorkspaceTaskControllerWorkflow`.** Id `workspace-tasks-<sha256(workflow_id)>` (`server/services/node_invocations.py:27-28`), started by Update-With-Start with `USE_EXISTING` (`:44-74`). Updates `submit` and `reset`, Query `describe` (`server/services/temporal/workspace_tasks_workflow.py:55-193`). Its `run()` only ever continues as new (`:207-215`); there is no exit.
- **What follows the docs:**
- The entity id is the workflow id.
- Handlers set state or request a rollover; the main loop alone continues as new: "Wake the main run loop, which alone owns CAN and the handler completion fence, rather than continuing as new from an Update handler" (`workflow_control_workflow.py:588-593`). Handler fences come before continue-as-new (`:608-609`, `:661`, `:683`; the Workspace controller at `workspace_tasks_workflow.py:195-205`).
- Carried state is seeded in `@workflow.init`: `workflow_control_workflow.py:62-92` (v1 controllers), `workspace_tasks_workflow.py:39-53` ("Update-With-Start can invoke a handler before run()"), `server/services/temporal/agent_workflow.py:746-757`.
- Update-id deduplication across rollover is done in workflow code, as the docs require. Controller Updates carry revisions (`server/services/temporal/execution_control.py:84-106`) and deterministic ids of the form `control:{revision}:admission`, `:checkpoint` and `:release` (`server/services/deployment/execution_control.py:80-81`, `:120-121`, `:150-152`, `:181-183`). The Workspace controller carries its last 128 submissions and resets (`workspace_tasks_workflow.py:45-46`, `:210-215`).
- Run-chain identity is checked by `first_execution_run_id` (`execution_control.py:71-73`), never by a cached run id.
- The controller's queue overflow lives in the database and is read back in bounded pages (`server/services/temporal/controller_queue.py`).
- **Exception to id-only addressing.** The legacy pause and resume fan-out for unversioned generations signals run-pinned handles (`handlers.py:915-940`), so a target that continued as new between listing and signalling misses the Signal. v1 generations do not use this path.
- **Deviation 1: no decommission for the Workspace controller.**
- Deleting a workflow runs Reset (`server/services/workflow_storage/deletion.py:50-60`). Reset calls `_reset_workspace_tasks` (`handlers.py:1145-1159`), which sends the controller's `reset` Update through Update-With-Start whenever the graph or its frozen snapshot has a Browser agent, Mobile Agent or Android tool node (`node_invocations.py:77-106`), or a controller already answers.
- Nothing closes the controller afterwards. Reset's execution sweep (`handlers.py:843-900`) does not match it, and no workflow-deleted hook touches it.
- So every deleted workflow whose deletion ran Reset and whose graph has such a node leaks one Running `workspace-tasks-*` execution, even if no Workspace task ever ran: Reset itself starts it.
- The leaked run is idle: no timer, no workflow tasks, no history growth. The cost is Visibility and persistence clutter, one execution per deleted employee, not load.
- **Deviation 2: one hot entity per deployment.** Every push event, poll cycle and run registration of a generation passes through its controller: about 40 controller history events per fired push run, and four Updates (about 20 events) per cron firing (fixtures `controller_push.json` and `cron_schedule.json` in `server/tests/temporal/fixtures/0.3.1/`). The event-accumulator page asks for Signals "below 5/sec per workflow instance". RFC-0005 removes this for new generations: one entity per conversation, at most 4 Signal-With-Starts per entity per second (RFC-0005 §9.4).
- **Deviation 3: duplicated state.** Every trigger spec carries the whole graph and parameter snapshot (`server/services/deployment/manager.py:896-913`), rewritten by `replace_graph` (`workflow_control_workflow.py:303-306`), against "Keep state minimal" (RFC-0004 H10; §8.3 item 1).
- **Deviation 4: legacy entities restore control state in `run()`.** `TriggerListenerWorkflow` and `PollingTriggerWorkflow` have no `@workflow.init` and set their pause flag at the top of `run()` (`server/services/temporal/trigger_listener_workflow.py:131-135`, `:254`; `server/services/temporal/polling_trigger_workflow.py:95-98`, `:139`). Unversioned controllers reset their state there too (`workflow_control_workflow.py:90-92`, `:393-394`). A `pause` or `resume` Signal applied in the first activation of a continued run is overwritten. After T5's fresh start no unversioned generation survives, so only generation-less deployments can still run these types.
- **Justified deviations.** No workflow execution timeout: it spans the whole continue-as-new chain and would end months-long, possibly paused entities (RFC-0005 §27). No Worker Versioning: a one-process desktop cannot keep old builds polling, so patches and per-generation flags are used instead (RFC-0005 §10.4).
**RFC-0005 plan, before the amendments in §20 (RFC-0005 §10.1, §10.2, §11):** Ids, message types, "signal handlers that only buffer" and "continues as new only from the main loop, after `workflow.all_handlers_finished()`" follow the docs. Gaps:
- **a. Nothing seeds the entity's counters after passivation.** The entity completes after an idle period (RFC-0005 §11) and the relay restarts it with Signal-With-Start and `ALLOW_DUPLICATE` (RFC-0005 §9.4), passing no workflow argument. `applied_seq` and `turn_no` are only "carried across continue-as-new" (RFC-0005 §9.5 step 4, §10.2), so a restarted run starts both at zero.
- Turn ids `turn-<conv id>-<n>` (RFC-0005 §10.1) then repeat, and so do the agent child ids built from them (`<root workflow id>-agent-<node id>`, `server/services/temporal/workflow.py:170-172`) and the agent's activity ids (`llm-step-{iteration + 1}`, `agent_workflow.py:3187`).
- Spend reservations are keyed `{workflow_id}:{activity_id}` under a UNIQUE constraint (RFC-0005 §14), and send keys come from workflow and activity ids (RFC-0005 §21.3; RFC-0004 §9.4). A repeated turn id makes a new turn's reservations look like old ones, and can make a provider drop a new send as a duplicate.
- **b. Supervisor Updates must survive its rollover.** The server deduplicates Update ids per run, the supervisor rolls over daily (RFC-0005 §10.2), and T10's deliverer resends an Update with the same id until it is applied. RFC-0005 §10.2 gives `set_limits` and `resume_after_budget` no payload shape; a relative change ("add $10") applied by two runs is applied twice.
- **c. Cost per employee.** A supervisor that never passivates and wakes every 5 minutes for upkeep (RFC-0005 §10.2) wakes 288 times a day. A desktop's sticky cache (the worker's cache of recently run workflows) holds 50 workflows (`server/services/temporal/worker.py:76-88`; `company deploy` VMs hold 500 and Docker 100). With 100 employees on a desktop, most wakes replay the run's whole history. The docs: "consider costs at scale"; RFC-0005 principle 1: "Sleep by default; wake on an edge". (§8.4 item 5.)
- **d. A lost supervisor can only be Reset.** The supervisor starts with `REJECT_DUPLICATE` (RFC-0005 §10.1), which suits Start, because each generation has a new id, but leaves a lost supervisor no way back under its id; the signal-with-start page recommends `ALLOW_DUPLICATE_FAILED_ONLY` for entities. That matches E16 and today's v1 behaviour (a failed controller fails its generation with `controller_execution_missing` and needs Reset, which starts a new id), but Reset clears the generation's chat and Context. Decision T20 keeps `REJECT_DUPLICATE` for Start and rebuilds a lost supervisor in place with `ALLOW_DUPLICATE_FAILED_ONLY`; after 3 rebuilds within an hour an attention item asks for Reset (item 5).
**Verdict:** Adopted with deviations (today). Planned in RFC-0005 (deviates in a, c and d, misses b).
**Recommendation:**
1. **Decommission the Workspace controller.**
- In `controller_status` (`server/services/node_invocations.py:31-41`), query with `reject_condition=QueryRejectCondition.NOT_OPEN` and treat `WorkflowQueryRejectedError` as "no controller". A Query also answers from a closed run, so without this a retried deletion would see the cancelled controller as present.
- In `_reset_workspace_tasks` (`server/services/deployment/handlers.py:1145-1159`), when no controller is running, run the plugin runtime cleanup in process (the body of `workspace_tasks.reset_runtime`, `server/services/temporal/workspace_task_activities.py:7-47`) instead of starting a controller through Update-With-Start.
- In `stop_workflow_for_deletion` (`server/services/workflow_storage/deletion.py:55-60`), after a successful Reset, cancel `workspace-tasks-<hash>` by id. Reset has already cancelled the children and the run waits in `wait_condition`, so the cancel ends it.
Versioning: none (client side only). When: P2, with RFC-0005 Phase 4, which adds entity cancellation to Reset (RFC-0005 §23); it needs no versioning and may land earlier. Effort S. Risk: a Workspace submit racing the deletion starts a child; the child is cancelled through REQUEST_CANCEL when the controller closes.
2. **The entity-start rule (RFC-0005 Phase 4 design).**
- The relay's Signal-With-Start passes the stream cursor (`consumed_seq`) as the workflow argument. The entity seeds `applied_seq` from it in `@workflow.init` on every start, a restart after passivation included. The server ignores the argument when a run is open.
- The turn number is allocated in the admission transaction (`admission.acquire`, RFC-0005 §13.3, which already writes one `sched_leases` row per turn) and returned with the grant. No restart, rebuild or rollover can reuse it, so turn ids, agent child ids and spend keys stay unique after passivation.
- The continue-as-new carry holds cursors, not payloads (§7.4 A4).
- Not chosen: keeping `turn_no` on the `bus_streams` row and seeding it the same way.
Versioning: new type, none. When: P2, Phase 4 (design text now). Effort S. Risk: none beyond the design.
3. **Absolute, revisioned supervisor Updates.** Every supervisor Update (`set_control_state`, `set_limits`, `resume_after_budget`) carries absolute values and the database row's revision. The validator rejects a revision at or below the applied one, which writes nothing to history. Required by T10, whose deliverer resends with a stable id. Versioning: new type. When: P2, Phases 4-5. Effort S.
4. **Legacy entities.** Stop starting generation-less deployments instead of fixing deviation 4 (default, §24): refuse the `deploy_workflow` WebSocket command without a generation (`server/services/deployment/handlers.py:163`, registered at `:2817`), remove the client's unused `deployWorkflowAsync` (`client/src/contexts/WebSocketContext.tsx:2766-2773`, exposed as `deployWorkflow`; it has no caller), and keep the two listener types for replay only. Versioning: none. When: P1, Phase 1b (1b.46). Effort S. Risk: a script or an old client that still calls `deploy_workflow`.
5. **Rebuild a lost supervisor in place** (decision T20; amends RFC-0005 §10.1).
- Start keeps `id_reuse_policy=WorkflowIDReusePolicy.REJECT_DUPLICATE`, because each generation has a new id. Every rebuild uses `ALLOW_DUPLICATE_FAILED_ONLY`, which admits a new run under `emp-<workflow_id>-g<gen>` only when the last one ended terminated, cancelled, timed out or failed, never after a completed one. Both pass `id_conflict_policy=WorkflowIDConflictPolicy.USE_EXISTING`, which returns the open run instead of failing, so a duplicate Start or rebuild attaches to it.
- Whatever reaches the supervisor by known id (RFC-0005 I6) and finds it closed rebuilds it: boot re-arm, the T10 deliverer, or the upkeep sweep (§8.4 item 5). It does so only while the database shows the generation live, because the rebuild's reuse policy would also admit a run after a Reset that cancelled or terminated the old one.
- The generation stays, so it keeps its chat and Context, which Reset would clear. The new run seeds its control state, alarms and budget from the database (§8.1 rule 2).
- At most 3 rebuilds within an hour, counted in the database. The next loss stops rebuilding and raises an attention item that asks the owner to Reset.
Versioning: new type (start options only). When: P2, RFC-0005 Phase 4 (attention items from Phase 8). Effort S. Risk: a cause the rebuild does not clear, such as an activity that fails for good, fails each new run; the limit turns that loop into one attention item.
**Remarks:**
- Limits per run: 10 Updates in flight and 2,000 in total (the server suggests continue-as-new at 90%), 10,000 Signals, 51,200 history events (https://docs.temporal.io/workflow-execution/event; https://docs.temporal.io/design-patterns/request-response-via-updates). The Workspace controller rolls over every 256 operations (`workspace_tasks_workflow.py:23`, `:202`), far inside the Update cap.
- The controller stays the hot path for every generation started before RFC-0005 Phase 4, until Phase 10 upgrades them.
- Item 1 changes no workflow code. Item 2 needs a test that passivates an entity and checks that the next turn's child and activity ids differ from the previous turn's. Item 5 needs a test that terminates a supervisor and checks that the rebuilt run keeps the generation, its chat and Context, and that a fourth loss within an hour raises the attention item instead of a rebuild.
- A rebuilt supervisor (item 5) is a new run, so the server's Update-id deduplication does not carry over from the lost run; item 3's revisions keep a resent Update from applying twice. The limit of 3 rebuilds an hour has the restart-intensity shape of RFC-0005 §18.2 (OTP MaxR/MaxT).
- Replay: the corpus (`server/tests/temporal/fixtures/0.3.1/`) covers controller push, poll, queue read, stop and resume, a spill rollover and one Workspace task. It has no Workspace controller rollover or decommission.
**Sketch:** (items 1 and 2)
```python
# Item 1: server/services/node_invocations.py
from temporalio.client import WorkflowQueryRejectedError
from temporalio.common import QueryRejectCondition
from temporalio.service import RPCError, RPCStatusCode
async def controller_status(workflow_id: str, *, client=None) -> dict | None:
handle = (client or temporal_client()).get_workflow_handle(controller_id(workflow_id))
try:
# A Query also answers from a closed run: a closed controller is absent.
return await handle.query(WorkspaceTaskControllerWorkflow.describe,
reject_condition=QueryRejectCondition.NOT_OPEN)
except WorkflowQueryRejectedError:
return None
except RPCError as exc:
if exc.status == RPCStatusCode.NOT_FOUND:
return None
raise
async def close_controller(client, workflow_id: str) -> None:
"""stop_workflow_for_deletion calls this after Reset succeeded."""
try:
await client.get_workflow_handle(controller_id(workflow_id)).cancel()
except RPCError as exc:
if exc.status != RPCStatusCode.NOT_FOUND:
raise
# Item 2: ConversationWorkflow (RFC-0005 Phase 4), sandboxed (§17.18)
@workflow.defn(name="ConversationWorkflow")
class ConversationWorkflow:
@workflow.init
def __init__(self, cursor: dict) -> None:
# The relay passes the stream cursor; "deliver" can run before run().
self._applied_seq = int(cursor.get("consumed_seq") or 0)
async def _start_turn(self, admission_request: dict) -> str:
grant = await workflow.execute_activity(
"admission.acquire", admission_request, start_to_close_timeout=timedelta(seconds=30))
# The database allocates the number with the lease: never reused.
return f"turn-{workflow.info().workflow_id}-{grant['turn_no']}"
```
### 8.3 Continue-As-New
**Docs:** https://docs.temporal.io/design-patterns/continue-as-new. Continue-as-new "completes the current Workflow execution and atomically starts a new one with the same Workflow ID"; state travels as arguments, and the new run (one execution in the chain of a workflow id) starts with an empty history. It keeps a workflow below the hard limits: terminated above 51,200 events, 2,000 Updates or 10,000 Signals (https://docs.temporal.io/workflow-execution/event). Rules that matter:
- Decide with `workflow.info().is_continue_as_new_suggested()`: "more reliable than a fixed iteration count". The page also says "Continue as new every 100-1000 iterations", which contradicts its own pitfall; a count is a backstop, not the trigger.
- "Pass minimal state": arguments "are serialized into the first event of the new execution", and the 2 MB payload limit applies.
- Call it from the main method only, after handlers finish: "don't call Continue-as-New from the handlers. Instead, wait for your handlers to finish" (https://docs.temporal.io/develop/python/workflows/continue-as-new).
- Pitfalls: "Forgetting to drain Signals before continuing ... carry pending Signals forward as arguments"; "Not accounting for Child Workflows ... By default, children are terminated. If children must survive, set ParentClosePolicy to ABANDON and pass their Workflow IDs"; "Caching Run IDs ... Always use Workflow ID without a Run ID"; "Catching the Continue-As-New exception"; "Not versioning state arguments".
- Best practices: log transitions, test state transfer, and "For fixed Schedules, use Temporal Schedules". The concept page adds: "you may also want to Continue-as-New periodically, depending on how often you deploy. This makes sure you're running only a couple of versions" (https://docs.temporal.io/workflow-execution/continue-as-new).
**SDK:**
- 1.30.0 has `workflow.continue_as_new(...)` (memo and search attributes default to the current run's), `info.get_current_history_length()`, `info.get_current_history_size()` and `info.is_continue_as_new_suggested()` (`temporalio/workflow/_context.py:166-205`).
- `continue_as_new` raises `ContinueAsNewError`, a `BaseException`, so `except Exception` cannot swallow it. The four `except BaseException` blocks in `AgentWorkflow` all re-raise and none wraps a rollover (`agent_workflow.py:1728`, `:1762`, `:2030`, `:2268`).
- `workflow.patched()` (the patch marker that versions workflow code) is memoized per run (`temporalio/worker/_workflow_instance.py:1358-1372`). A run that first evaluates a marker while replaying a history without it keeps the old branch for the rest of that run; only a later run of the chain takes the new branch. A marker first reached outside replay returns True.
- The server suggests continue-as-new at 4,096 events or 4 MiB of history by default (server 1.32 dynamic config `limit.historyCount.suggestContinueAsNew` and `limit.historySize.suggestContinueAsNew`). Neither the app nor the dev server overrides them.
- 1.34.0 adds only the Update-handler rule of §8.2.
**OpenCompany today:**
- **`WorkflowControlWorkflow`.**
- Trigger: `_history_pressure` is "suggested, or more than 10,000 events" (`workflow_control_workflow.py:27`, `:568-576`; helper `trigger_listener_workflow.py:89-102`), checked after events, spawns, poll cycles and Updates (`:359`, `:483`, `:588-593`, `:853`, `:865`, `:908`), and acted on only by the main loop (`:440-443`). The suggestion is read first, so controllers normally roll over near 4,096 events.
- Fences: it waits for the execution-control checkpoint and for all handlers (`:608-609`), spills any tail that arrived meanwhile and fences again ("There is no await between this final fence and CAN", `:647-665`).
- Carry (`:684-704`): state, revision, triggers with their provider `seen_ids`, the first 256 events or 512 KB of the queue (the rest goes to the database), the last 4,096 dedupe keys, execution-control state, live roots and the membership epoch. It measures the argument and raises the non-retryable `ControllerContinuationTooLarge` above 1,900,000 bytes, for v1 controllers only (`:697-711`).
- Children: graph runs start with ABANDON and live roots are carried by id. Poll tasks are cancelled and awaited (`:612-615`). The transition is logged (`:712-715`). Fixture `controller_spill_rollover`.
- **`AgentWorkflow`.** Rolls over at an iteration boundary when pressure says so and no delegation is live (`agent_workflow.py:2982-2996`). The durable path waits until resumed, fences handlers and carries `_agent_resume` (iteration, transcript, usage, prepared payload, tools, thinking text, browser artifacts), with a 1.9 MB check that raises `AgentContinuationTooLarge` (`:2997-3031`). The legacy path has no fence and drops the transcript to the opening prompt above 1 MB (`:3032-3069`). Iterations continue across runs (`:1151-1159`), so activity ids stay unique in the chain. Fixture `agent_rollover`.
- **`WorkspaceTaskControllerWorkflow`.** Rolls over only when idle, after 256 operations, a suggestion or 10,000 events (`workspace_tasks_workflow.py:195-205`). It carries its epoch and the bounded dedupe maps (`:207-215`). It logs nothing and has no rollover fixture.
- **Others.** `PiCodingAgentWorkflow` defers its rollover while Task Manager children are live (`server/services/temporal/pi_workflow.py:382-387`). The legacy listeners roll over on pressure (fixtures `legacy_trigger_listener`, `legacy_polling_trigger`). `CronTriggerWorkflow` handles one firing, because cron is a Temporal Schedule (`server/services/temporal/schedules.py:139-250`), as the page advises.
- **Tests.** Fixtures were recorded on a server with `limit.historyCount.suggestContinueAsNew=40`, so four scenarios record a rollover (`server/tests/temporal/record_fixtures.py:69`, `:1215-1220`): the page's "Test state transfer", done well.
- **Deviation 1: carry weight.** Each controller trigger embeds the graph (RFC-0004 H10). The agent's durable carry holds the stored seed conversation inside `prepared_payload` (up to 1 MB), the tools twice and unbounded thinking text (`server/services/temporal/agent_activities.py:1825-1830`; `agent_workflow.py:3014-3016`). Over the limit the entity fails, explicitly; a failed v1 controller then fails its generation (§8.2).
- **Deviation 2: idle controllers never roll over.** A controller with no events, poll cycles, firings or Updates never reaches a suggestion. When a later release adds a patch marker, such a run replays without it and keeps the old branch for its whole life (markers read at run start, like `controller-durable-queue-v2` at `:396` and `controller-event-accumulator-v1` at `:549`, are memoized per run). The old branch can then never be deleted, and RFC-0005 §24.3 cannot deprecate the marker. Busy controllers are fine: a v1 cron firing sends the controller four Updates and Update handlers request a rollover (`:588-593`), so a cron-only controller rolls over after roughly 200 firings.
- **Deviation 3: a swallowed cancellation can stall the rollover (RFC-0004 H8).**
- `_continue_as_new` cancels the poll tasks and awaits them (`:612-615`). A cancelled poll activity reaches the loop as `ActivityError`, which `except Exception` catches (`:851`, `:863`); the loop continues and the `gather` never returns.
- Unversioned controllers are fully exposed, and v1 controllers are exposed too. The checkpoint at `:608` means no poll is in flight when it resolves, but `begin_action` does not refuse new actions during a rollover. A poll task that becomes ready before the cancel at `:612` (the first cycle of a trigger registered during the `:609` fence, or a loop released from its pause wait in the same pass) can be cancelled mid-activity.
- A `register_trigger` Signal, or a `replace_graph` that revives a retired poll, during the spill phase creates a poll task after the cancel (`:301-302`, `:331-332`). It is not cancelled before the rollover.
- **Deviation 4: legacy listeners (P3).** A paused `TriggerListenerWorkflow` checks pressure only after processing an event (`trigger_listener_workflow.py:208-225`, `:261-264`). Queued events beyond 256 are dropped at rollover (`:291-303`), against "carry pending Signals forward". `PollingTriggerWorkflow` skips the check on failed cycles (`polling_trigger_workflow.py:186-198`), so a permanently failing poll reaches termination in about 3.4 days at the 60 s default (11 events per cycle, fixture `legacy_polling_trigger.json`). The legacy `AgentWorkflow` path loses a pause at rollover.
- **Minor.** The durable `AgentWorkflow` rollover and the Workspace controller do not log the transition.
**Verdict:** Adopted with deviations. Main-loop-only rollover, handler fences, pending events carried or spilled, dedupe ids carried, ABANDON children (or waiting for none), id-only addressing and chain identity all follow the docs. The deviations are carry weight, idle controllers that never pick up new code, and a cancellation path that can stall a rollover.
**Recommendation:**
1. **Slim the controller carry** (RFC-0004 D7 and H10; already in RFC-0005 §24.2's controller lane).
- Step 1: carry each distinct graph once, in a `graphs` map keyed by content hash; give each trigger a `graph_ref`, and rebuild `listener_args` in `_seed_carried_state`. Deduplicate by hash, not "one graph": a trigger retired by `replace_graph` keeps its old snapshot (`workflow_control_workflow.py:293-300`).
- Step 2: carry a generation reference and load the graph by activity.
Versioning: write the new shape behind `workflow.patched("controller-slim-carry-v1")`, evaluated in `_continue_as_new`; read both shapes in `_seed_carried_state`. When: P2, controller lane after Phase 1. Effort S, then M.
2. **Fix cancellation in the poll loops, for v1 controllers too** (RFC-0004 H8). In `_poll_trigger` (`:849-866`), re-raise when `temporalio.exceptions.is_cancelled_exception(exc)`. In `_continue_as_new`, cancel and await any poll task created after the first cancel, immediately before `workflow.continue_as_new`. Versioning: `workflow.patched("controller-poll-cancel-v1")`, because a history that recorded a swallowed cancel must keep its old branch (RFC-0004 H8). When: P2, controller lane. Effort S.
3. **Move idle controllers onto new code.**
- Add a `request_rollover` Signal that sets `_can_requested`; the main loop already wakes on that flag (`:431-443`).
- At the first boot of each release, boot reconcile sends it to every live generation's controller, by id from the app database (RFC-0005 I6). Default (§24).
- A new handler adds no command until a history contains its Signal, so it needs no patch. The old run leaves through its own memoized code, and the next run takes the newest branches. This is the "periodic rollover" that RFC-0005 §24.3 needs to retire markers.
Versioning: none (new handler). When: P2, controller lane, after items 1 and 2. Effort S. Risk: forcing a rollover of a controller whose carry is too large fails it; hence item 1 first.
4. **Trim the agent carry and carry unmerged inputs.** The trim is 1b.19 (§16.8 rec 1): drop `conversation`, `memory_content` and `parameter_snapshot` from the carried `prepared_payload`, and keep `tools`, which the resumed run reads (done, 675375dc). Also cap the carried thinking text (not built in 1b.19). Under RFC-0005 merging (T9's merge point B1), carry offered-but-unmerged input references in `_agent_resume`, or merge them at B1 before rolling over ("Drain your Signal channel and carry pending Signals forward"). Versioning: none for the trim and the cap, because the continue-as-new input is not compared on replay; the new carry under the frozen `merge_version=1` flag. When: the trim in Phase 1b (1b.19); the rest P2, RFC-0005 Phase 4. Effort S, then M. T6 keeps transcripts in history, which makes this trim the main lever on carry size.
5. **Byte backstop.** Add `info.get_current_history_size()` to `_history_pressure` only if a namespace ever raises the server's 4 MiB suggestion. Versioning: patch. When: P3. Effort S.
6. **Legacy listeners.** Request a rollover from `on_event` while paused, spill instead of dropping, and check pressure on failed poll cycles; or stop starting generation-less deployments (the default: §8.2 item 4, 1b.46). When: P3. Effort S.
7. **Log every rollover** (`workflow.logger` is replay-aware) and record a Workspace controller rollover fixture. Versioning: none. When: P3. Effort S.
**Remarks:**
- Order matters: items 1 and 2 before item 3, or forced rollovers fail large controllers or stall.
- Signal rate: above about 5 Signals per second, the controller's multi-step rollover (checkpoint wait, spill activities, fences) can be rewound again and again ("Continue-As-New requires a brief window (~100 ms) with no unhandled signals", event-accumulator page). RFC-0005's cap of 4 per entity per second keeps new generations out of this.
- The carry lands in the new run's first event, so a large carry also starts every run large and brings the next size-based suggestion closer.
- RFC-0005's plan before the amendments in §20 (the entity rolls over "only between turns, when suggested or past 4,000 events or 4,000 Signals"; the supervisor "when suggested, or daily", RFC-0005 §10.2) follows the docs, periodic rollover included. It needs §8.2 item 2, and the excess inputs returned to the database at rollover must not be dropped by `applied_seq` when they are delivered again (§7.4).
**Sketch:** (items 1 and 3)
```python
# server/services/temporal/workflow_control_workflow.py (controller lane)
class WorkflowControlWorkflow:
@workflow.signal
async def request_rollover(self) -> None:
# Recorded in history, so replay is deterministic. The main loop owns
# continue-as-new and already wakes on _can_requested.
if not self._closed:
self._can_requested = True
def _slim_triggers(self) -> tuple[dict, dict]:
"""Carry each distinct graph once; retired triggers keep their own."""
graphs, triggers = {}, {}
for listener_id, spec in self._triggers.items():
args = dict(spec["listener_args"])
graph = {key: args.pop(key, None) for key in ("nodes", "edges", "parameter_snapshot")}
ref = hashlib.sha256(json.dumps(graph, sort_keys=True, default=str).encode()).hexdigest()
graphs[ref] = graph
triggers[listener_id] = {**spec, "listener_args": {**args, "graph_ref": ref}}
return graphs, triggers
# _continue_as_new writes {"graphs", "triggers"} only when
# workflow.patched("controller-slim-carry-v1"); _seed_carried_state reads both shapes.
# Boot reconcile, once per release (the app database records the last booted version):
if await release_changed_since_last_boot(database):
for control in await database.list_live_workflow_controls(): # known ids (I6)
await client.get_workflow_handle(control.controller_workflow_id).signal("request_rollover")
```
### 8.4 Updatable Timer
**Docs:** https://docs.temporal.io/design-patterns/updatable-timer. A wait "with both a time limit and an update condition": when a message moves the deadline, the workflow recomputes the remaining time and waits again. In Python, `workflow.wait_condition(fn, timeout=duration)` "returns normally when the condition is met, or raises `asyncio.TimeoutError` on timeout"; the page's helper keeps an absolute wake time and loops. Rules that matter:
- "Use absolute timestamps"; "Validate updates. Ensure new deadlines are in the future"; "Consider max extensions"; "Add Queries"; "Log changes"; "Reuse the timer helper"; "Combine with conditions".
- Pitfalls: "A wait without a timeout does not create a timer"; "The timer duration is set once when the wait is called ... This is why the timer helper loops and recalculates"; "Accepting a deadline in the past causes the timer to expire immediately".
- Fit: approval deadlines, SLAs, escalations, grace periods. Not for "fixed timeouts that never change (use a fixed sleep)" or "complex scheduling (use Temporal Schedules)".
- Timers are accurate only from one second up. The timers page: "your Workflow might specify a value as short as one second or as long as several years ... your Workflows should not rely on sub-second accuracy for Timers" (https://docs.temporal.io/workflow-execution/timers-delays). This is a documented range and an accuracy caveat, not an enforced minimum.
**SDK:**
- 1.30.0: `wait_condition(fn, *, timeout, timeout_summary)` (`temporalio/workflow/_context.py:920-943`) is `asyncio.wait_for` over a condition future. A positive timeout starts a timer; when the condition wins first, the SDK cancels it, so each move of a deadline costs a TimerStarted and a TimerCanceled event (`temporalio/worker/_workflow_instance.py:1790-1815`, `:2768-2799`).
- Under CPython 3.12, a timeout of 0 or less raises `TimeoutError` at once without creating a timer, even if the condition is already true.
- `workflow.sleep(duration, *, summary=)` and `timeout_summary` name the timer in the Web UI.
- 1.34.0 documents two traps: switching a wait between `timeout=None` and a positive timeout "constitutes a nondeterministic change", and a timeout of `0` "should not be used".
**OpenCompany today:**
- No workflow deadline moves. Fixed timers: `ApprovedToolCallWorkflow` sleeps until an absolute `grace_until` and skips the sleep when it has passed (`server/services/temporal/approved_tool_call_workflow.py:41-46`), then claims the row at its revision, so an Undo, a newer Send or a cancel ends it with nothing sent (`:48-55`); the controller's 5-second head-of-queue retry (`workflow_control_workflow.py:480`); poll intervals (`:823`).
- Human waits use no workflow timer: the approval gate re-reads its row inside an activity (`server/nodes/workflow/approval_gate/__init__.py:235-277`); approval expiry (168 hours by default) is enforced by that loop or by the next decision (`server/services/approvals/decisions.py:65-81`); reviewed delivery waits inside 31-day activities (§9.6).
**RFC-0005 plan, before the amendments in §20:** the supervisor's "updatable alarm timer", "budget period end (midnight in the owner's time zone)" and "5-minute upkeep" (RFC-0005 §10.2, §15); the entity's debounce, 60-second admission retry, turn deadline per class (a "slow turn" alert under T19), idle passivation and restart backoff (RFC-0005 §10.2, §11, §18.2); `ApprovalWorkflow`'s expiry, then the Undo window (RFC-0005 §10.2, §17.5); recurring alarms as Schedules (RFC-0005 §15).
**Verdict:** Planned in RFC-0005 (follows the docs, misses details). Today's code has no moving deadline, and its fixed Undo wait on an absolute instant is what the page prescribes for a fixed wait. RFC-0005 uses absolute instants, caps extensions (debounce caps, 500 alarms per employee), fires late alarms with a lateness note (better than rejecting a past deadline, for alarms), treats Signals as hints reconciled from the database, and uses Schedules for recurring alarms. It misses the details below.
**Recommendation:**
1. **One live timer per workflow.**
- The supervisor, the entity and `ApprovalWorkflow` keep named absolute deadlines and wait on one `wait_condition(lambda: changed or work_ready, timeout=earliest - now)`, recomputed after every wake.
- One shared helper in `server/services/temporal/` (sketch below), with time-skipping tests (RFC-0004 §14).
- The deadlines cross continue-as-new as instants; timers never do.
- Deadlines that move: the per-class "slow turn" alert (T19) does not count time parked or paused, and it is recomputed when priority inheritance changes the turn's class (RFC-0005 §11 step 5). The no-progress backstop (a default, §24.1; specified in §6.4) cancels a turn after 24 hours without a completed step; each completed step restarts it, and it never counts time parked, paused, or waiting for the owner (an approval draft, the owner's browser). Parking ships with RFC-0005 Phase 5, for generations with `scheduler_version=1` (T18, §11.8); today's chat runs keep the 24-hour cancel (1b.14, §6.4).
Versioning: new types. When: P2, Phases 4-7. Effort M.
2. **Never wait with a remaining time of 0 or less.** Check `remaining <= 0` and act first (fire, expire, flush), as the page's Go sample does. Late alarms after a laptop sleep (RFC-0005 §22 scenario d) take the same path. Add the check to the determinism lint (1b.40, §17.18; RFC-0004 §8.1). Versioning: new types. When: P2, Phase 4. Effort S.
3. **Compute "midnight in the owner's time zone" outside workflow code**, with the same function that keys the ledger's day, and carry the UTC instant to the supervisor. Otherwise the timer and the ledger can disagree on daylight-saving days, and a tzdata update changes what replayed workflow code computes. Versioning: `scheduler_version=1`, because the supervisor already runs from Phase 4 without a period timer. When: P2, Phase 5. Effort S.
4. **No sub-second windows.** T9 removes the owner-chat debounce (§7.4); its window (0.4 s, RFC-0005 §11) was the only one below one second. Keep every remaining window at one second or more, and document that timers are accurate only from one second up. When: P2, Phase 4. Effort S (text).
5. **No per-employee upkeep timer** (default, §24). Remove the supervisor's 5-minute upkeep. One instance-wide sweep in the backend process, every 5 minutes beside the chat watchdog (`server/services/chat/watchdog.py`), reads the database for overdue work and signals, by known id, only the supervisors and entities concerned:
- leases whose turn has closed (confirmed by describing the turn by id);
- turns suspended or parked past the 7-day limit (RFC-0005 §10.2; T19 then dead-letters a parked turn's input with an attention item);
- alarms that did not fire;
- parked turns whose cause has cleared, such as an allowance topped up (T11): they get `released{reason}` (§11.8). Outage parks probe on their own 15-minute timer. Parking exists from RFC-0005 Phase 5, with `scheduler_version=1` (T18).
Versioning: new types (the supervisor is built without an upkeep timer in Phase 4; the sweep itself is backend code). When: P2, Phase 4, and the sweep gains parked turns in Phase 5. Effort S. Saves 288 wakes per employee per day (28,800 a day at 100 employees).
6. **Name timers and log moves.** Pass `summary` or `timeout_summary` (`alarm`, `period`, `debounce`, `passivate`, `slow-turn`, `no-progress`, `expiry`, `undo`), and log every deadline change ("Log changes"). When: P3. Effort S.
**Remarks:**
- Adding a timeout to an existing wait (for example a passivation timeout in an existing type) is a command change: it needs a patch or a frozen flag. New types are free.
- Each debounce extension costs about 6 events (a Signal, a workflow task of 3 events, TimerCanceled, TimerStarted). The 8-10 second caps and the relay's limit of 4 Signals per second bound it.
- Timers keep running through a cooperative pause. A timer that fires while the employee is stopped must check control state before it acts; alarms become inputs that the relay holds (RFC-0005 §14).
- `alarms_changed` as a Signal matches the docs and RFC-0005 I1: the timer re-reads the earliest due time from the database, and `alarms.fire_due` by compare-and-swap makes a stale timer harmless ("Check if the timer already expired before updating").
**Sketch:** (items 1 and 2)
```python
# server/services/temporal/deadlines.py: one live timer over named absolute deadlines
import asyncio
from datetime import datetime
from typing import Callable
from temporalio import workflow
class Deadlines:
def __init__(self, carried: dict[str, str] | None = None) -> None:
self._due = {name: datetime.fromisoformat(at) for name, at in (carried or {}).items()}
self._changed = False
def set(self, name: str, at: datetime | None) -> None: # handlers call this
if at is None:
self._due.pop(name, None)
else:
self._due[name] = at
self._changed = True # wakes wait(); it re-arms on the new minimum
def carry(self) -> dict[str, str]: # continue-as-new carries instants, never timers
return {name: at.isoformat() for name, at in self._due.items()}
async def wait(self, ready: Callable[[], bool]) -> list[str]:
"""Return the names now due, or [] once ready() is true."""
while True:
now = workflow.now()
due = sorted(name for name, at in self._due.items() if at <= now)
if due or ready():
return due # act first: never wait with a timeout <= 0
self._changed = False
earliest = min(self._due.values(), default=None)
try:
await workflow.wait_condition(
lambda: self._changed or ready(),
timeout=None if earliest is None else earliest - now,
timeout_summary="next deadline")
except asyncio.TimeoutError:
pass # the next pass returns what is now due
```
## 9. External interaction
Five patterns cover waiting on the world outside a workflow: polling, long activities, delayed starts, HTTP callbacks and human decisions. OpenCompany has all five needs. Two rules of the category are broken today: a delay (the Timer node) and multi-day human waits (approvals) run inside activities.
### 9.1 External interaction patterns (category overview)
**Docs:** https://docs.temporal.io/design-patterns/external-interaction-patterns. A selection guide for waiting on the world:
- "The external system offers no notification": poll, and "tune the interval to the expected latency".
- "One Activity runs for a long time": heartbeats "so failures resume instead of restarting".
- "A person or external system must decide": Approval, "block on a Signal".
- "Execution should begin later": Delayed Start.
- "You send or receive HTTP callbacks": Delayed Callback.
**OpenCompany today:**
| Wait | Today | Pattern |
|---|---|---|
| Gmail, Outlook, IMAP triggers | a sleep-plus-activity loop inside the controller | Polling (§9.2) |
| LLM steps, coding and browser agents, Pi driver, Apify, permit waits | heartbeating activities | Long-running activity (§9.3) |
| Drafts (approvalGate), reviewed team-job delivery | a database poll inside an activity | Approval (§9.6) |
| Held tool calls (Ask first) | a database row, then a send workflow | Approval plus Delayed Callback pattern 2 |
| Browser "Needs you" | an in-memory wait inside the browser activity | Approval (bounded, deliberate) |
| The Undo window before a send | `workflow.sleep`, then one activity | Delayed Callback pattern 2 |
| Timer node; cron node on a canvas Run | `asyncio.sleep` inside an activity | Delayed Callback pattern 2, misused (§9.5) |
| Generic and signed webhooks | `emit`: a Visibility query, then a plain Signal | Delayed Callback pattern 1 (§9.5) |
| An external job that completes later | none | Delayed Callback pattern 3 |
**Verdict:** Not applicable (selection guide).
**Recommendation:**
1. Add a "where each wait goes" rule to §5 and to `docs-internal/TEMPORAL_ARCHITECTURE.md`:
- human waits never sit inside an activity; they are an Update plus a durable timer (RFC-0004 D13);
- delays are workflow timers; an activity may sleep only within one attempt;
- a poll is one retried activity per trigger (T7, §9.2);
- inbound events reach workflows by known id from the app database (RFC-0005 I6), and no request path calls Temporal (I2);
- timers are accurate only from one second up (T9).
Versioning: none (text). When: with this RFC; the TEMPORAL_ARCHITECTURE.md text with Phase 3. Effort S.
**Remarks:** The index summarizes Approval as "block on a Signal", but the Approval page suggests "an Update with a validator" when the approver needs immediate confirmation. OpenCompany's card needs that (`too_late`, `already_decided`, `approval_conflict`). Under I2 and T10 the database compare-and-swap gives the immediate answer, and the Update carries the decided state into the workflow (§9.6).
### 9.2 Polling External Services
**Docs:** https://docs.temporal.io/design-patterns/polling. Three strategies, chosen by interval:
1. **Frequent** (1 s or faster): a loop inside one activity that heartbeats on every iteration, with `heartbeat_timeout` shorter than `start_to_close_timeout` (the sample uses 2 s and 60 s); for operations of "minutes, not hours".
2. **Infrequent** (1 minute or slower): the activity polls once and raises when the result is not ready, and `RetryPolicy(backoff_coefficient=1, initial_interval=<interval>)` sets the cadence. "Retries do not add events to the Workflow history, keeping it small."
3. **Periodic sequence**: a child workflow loops an activity and `workflow.sleep` up to N times, then continues as new.
Also on the page:
- The comparison table rates history impact "Medium" (frequent), "Minimal" (infrequent), "Low (with CAN)" (periodic), and "Workflow Timer Loop | Any | High | Medium | Avoid this approach."
- Best practices: "Add exponential backoff. Use backoff for error cases (not normal polling)"; "Implement circuit breakers"; "Add jitter. Prevent thundering herd when many Workflows poll simultaneously"; "Distinguish transient from permanent failures"; "Monitor polling metrics"; "Respect rate limits"; "Consider webhooks".
- Pitfalls: missing heartbeats, unbounded history, tight loops, "No timeout", "Ignoring errors", "Workflow timer loops ... bloats history".
- The page never mentions Schedules.
- The delayed-retry page adds: "`nextRetryDelay` overrides the interval for a single retry; the RetryPolicy still governs maximum attempts" (https://docs.temporal.io/design-patterns/delayed-retry).
**SDK:** All in 1.30.0 and unchanged in 1.34.0: `activity.heartbeat`; `RetryPolicy`; `ApplicationError(..., non_retryable=, next_retry_delay=, category=ApplicationErrorCategory.BENIGN)` (`temporalio/exceptions.py:111-175`); a BENIGN failure logs at DEBUG and records no failure metric (`temporalio/worker/_activity.py:457-472`); `workflow.execute_activity(..., summary=)`; `WorkflowIDConflictPolicy.USE_EXISTING`.
**OpenCompany today:**
- **Controlled generations.** Polls run inside `WorkflowControlWorkflow._poll_trigger` (`server/services/temporal/workflow_control_workflow.py:792-908`):
- `workflow.sleep(poll_interval)` (`:823`), then one `poll.<type>.v<n>` activity per cycle, with `start_to_close = max(30, 4 x interval)` seconds and no heartbeat timeout (`:808-809`, `:834-838`, `:856-860`);
- `DEFAULT_ACTIVITY_RETRY`: 3 attempts, 1-30 s apart, only `NodeUserError` and `InvalidEvent` non-retryable (`server/services/temporal/_retry_policies.py:52-55`, `:62-67`);
- each cycle is an admitted action under execution control (`:833`), so Stop and rollover wait for it;
- the trigger arrives by a `register_trigger` Signal that carries the full graph (`server/services/deployment/manager.py:965-975`).
- **Generation-less deployments** run the same loop in the standalone `PollingTriggerWorkflow` (`server/services/temporal/polling_trigger_workflow.py:157-245`).
- **The cycle activity**, `PollingTriggerNode.as_poll_activity` (`server/services/plugin/polling.py:204-272`):
- fetches the visible ids, then each new id's details, skipping (and so losing) any that fail (`:236-246`);
- runs `post_emit`, for example mark-as-read, before returning (`:256-259`);
- returns full payloads and the whole visible-id set (`:270`), with no byte bound (1b.18 adds one, §12.3); the count is capped by the provider's page (Gmail 20, Outlook 25, IMAP 50);
- never heartbeats.
- **Canvas Runs** loop `asyncio.sleep(poll_interval)` inside the trigger's node activity until the first new item (for example `server/nodes/google/gmail_receive/__init__.py:149-193`), kept alive by the plugin beat loop. That is the frequent shape, acceptable for an interactive run.
- **Measured against the page:**
- Interval: a 30-second floor in both workflow paths (`workflow_control_workflow.py:37`, `:808`; `polling_trigger_workflow.py:80`) and a 60-second default. Gmail's parameters allow 10 s (`gmail_receive/__init__.py:39`), which is silently raised.
- No jitter: every loop sleeps exactly the interval, and after a boot re-arm or a rollover all of an instance's polls restart together.
- No failure classification and no backoff: a cycle that exhausts its attempts is logged and the loop continues at the same interval (`workflow_control_workflow.py:851-866`; `polling_trigger_workflow.py:193-198`). A disconnected Google account raises a retryable `ValueError` (`server/nodes/google/_auth_helper.py:68`): three attempts and an ERROR line every cycle, forever. Nothing escalates.
- Cancellation is swallowed (RFC-0004 H8; §8.3 deviation 3).
- No heartbeat: a crashed worker or a hung IMAP call is noticed only at start-to-close (2 minutes at 30 s, 4 minutes at 60 s, 4 hours at the 3,600 s maximum), and a cancel cannot reach the activity. Because Stop waits for an admitted poll, a hung poll can hold Stop for up to three start-to-close timeouts (12 minutes at the 60-second default).
- History: about 8-11 events per quiet cycle (fixture `legacy_polling_trigger.json`: 11). With the server's 4,096-event suggestion, a controller with one 30-second poll rolls over about every 3 hours (est.), carrying every trigger spec with a full graph copy.
- RFC-0004 M3: mark-as-read runs before the event is recorded, and an id whose detail fetch failed counts as seen.
**RFC-0005 plan, before the amendments in §20:** one Schedule per poll trigger (`poll-<workflow_id>-g<gen>-<node_id>`, overlap SKIP, RFC-0005 §15), whose short workflow fetches, appends to the inbox and saves the cursor in one transaction, then runs `post_emit` (fixing M3). That is not one of the page's strategies, and it costs one workflow execution per tick: 1,440 a day per 60-second trigger, about 10,080 retained per trigger at 7-day retention, about 1M at 100 employees with one poll each (est.). Owner decision T7 replaces it.
**Verdict:** Adopted with deviations. Today's loop is the page's periodic sequence, hosted inside the one workflow meant to live for months, without jitter, backoff, failure classification, a heartbeat timeout or working cancellation. T7 adopts the page's infrequent strategy for RFC-0005, which then follows the docs.
**Recommendation:**
1. **Polls as retried activities** (decision T7; amends RFC-0005 §3.1 E5 and §15).
- A new `PollWorkflow`, id `poll-<workflow_id>-g<gen>-<node_id>`, is started at Start and by boot re-arm with `WorkflowIDConflictPolicy.USE_EXISTING` and `EventWorkflowId`.
- It runs one activity that never completes: `RetryPolicy(initial_interval=<interval>, backoff_coefficient=1, maximum_attempts=0)`, no `schedule_to_close_timeout`, `start_to_close` 3 minutes, `heartbeat_timeout` 60 s.
- Each attempt reads the subscription and the cursor from the app database (I10). When the generation has ended it raises a non-retryable `PollRetired`, which ends the workflow. While the employee is stopped or held by its budget, it skips the fetch, so Stop needs no Temporal call.
- Otherwise it fetches, appends at most one page of new items (50 items or 512 KB, §12.3) and saves the cursor, advanced only over appended items, in one transaction, heartbeating after the list call and after each detail, and runs `post_emit` after the commit (fixes M3).
- It ends with `ApplicationError("polled", category=BENIGN, next_retry_delay=<interval plus jitter, or the error backoff>)`.
- Effects: the history holds about five events per generation and trigger (est.); bodies never enter Temporal (I7); Reset also terminates the workflow through `EventWorkflowId`.
Versioning: new type. When: P2, RFC-0005 Phase 3. Effort M. Risk: operators see an activity at attempt 50,000; give it a `summary=` such as "poll Gmail every 60 s" and document it.
2. **Errors and pacing, for the new path and today's loops** (defaults, §24):
- jitter of up to 10% of the interval, at most 30 s;
- credential and configuration failures: in today's loops they raise `NodeUserError` (after 1b.17), which `DEFAULT_ACTIVITY_RETRY` does not retry within a cycle; in the new path they are counted and backed off, and never end the poll (the owner may reconnect later);
- consecutive failures back the interval off, doubling up to 1 hour;
- 5 consecutive failed cycles raise the "driver down" attention item of RFC-0005 §18.3;
- poll metrics: attempts, durations, items found.
In the new path this lives in the activity and the database. In today's controller, jitter and backoff only change timer durations, which replay never compares except from or to zero; derive the jitter from a hash of the listener id and cycle number, so `workflow.random()` stays untouched. Versioning: none. When: P2, Phase 3 and the controller lane. Effort S.
3. **Today's loops, options and activity body only.** Built narrower in 1b.17 (c6c55451): no heartbeats; both deadlines become `min(180 s, today's formula)` (`POLL_ACTIVITY_MAX_S`); a credential error annotated at its source becomes `NodeUserError`. The design follows: set `heartbeat_timeout=60 s` and a fixed `start_to_close` of 3 minutes instead of 4 x the interval (`workflow_control_workflow.py:808-809`, `:834-838`, `:856-860`, and the same options in `polling_trigger_workflow.py`); heartbeat after the list call and after each detail in `as_poll_activity`; raise `NodeUserError` for credential failures. Activity options are not compared on replay (RFC-0004 §8.5). Versioning: none. When: P1, Phase 1b (1b.17). Effort S. The replay corpus must still pass `controller_poll.json` and `legacy_polling_trigger.json`.
4. **Retried-by-design activities must not log as failures.** `ActivityObservabilityInterceptor` logs a WARN `activity_retry` on every attempt after the first and a WARN `activity_end` on every failure (`server/services/temporal/_interceptors.py`): under item 1 that is two WARN lines per poll cycle. Log BENIGN failures, and attempts of poll-cycle activities, at DEBUG. Poll attempts count toward no retry alert (1b.30 excludes T7's poll attempts, §11.7); BENIGN LLM retries still do. This refines 1b.30's quieter retry log. The BENIGN half is built: since 48aebf17 a BENIGN failure's `activity_end` line logs at DEBUG. Versioning: none. When: P2, Phase 3 for the poll half. Effort S.
**Remarks:**
- Why not Schedules: every tick becomes an execution with its own events and Visibility row, kept for the namespace retention; the infrequent strategy records nothing per quiet cycle. T7 also keeps Stop and Resume free of Temporal calls.
- A never-completing poll activity must not count as a business action, or a Stop checkpoint never comes. `PollWorkflow` lives outside the controller and the supervisor, so nothing counts it.
- `next_retry_delay` overrides the retry interval for one attempt, and `maximum_attempts=0` keeps attempts unlimited. Never add `schedule_to_close_timeout`: it would end the poll.
- Under T8 (workers in a second process on hosted tier S), poll activities read credentials. By default the worker builder initializes the encryption key in the worker process too (§15.2 rec 3), so reading credentials does not bind polls to the API process.
- Generations that keep today's controller until RFC-0005 Phase 10 keep its loop; items 2 and 3 cover them.
**Sketch:** (item 1)
```python
# server/services/temporal/poll_workflow.py: new type (RFC-0005 Phase 3, decision T7)
@workflow.defn(name="PollWorkflow") # sandboxed (§17.18)
class PollWorkflow:
@workflow.run
async def run(self, ref: dict) -> dict:
# ref: workflow_id, generation, node_id, activity ("poll.<type>.v<n>"), interval_s
try:
await workflow.execute_activity(
ref["activity"], ref, activity_id="poll",
start_to_close_timeout=timedelta(minutes=3),
heartbeat_timeout=timedelta(seconds=60),
retry_policy=RetryPolicy(initial_interval=timedelta(seconds=ref["interval_s"]),
backoff_coefficient=1.0, maximum_attempts=0),
summary=f"poll every {ref['interval_s']} s")
except ActivityError as exc:
if getattr(exc.cause, "type", None) != "PollRetired":
raise
return {"retired": True}
# The activity body, in an activities module outside the sandbox: one cycle per attempt.
# It never returns normally.
async def poll_cycle(ref: dict) -> None:
sub = await poll_subscription(ref) # generation state and cursor, from the database
if sub is None: # generation reset, or the trigger was removed
raise ApplicationError("retired", type="PollRetired", non_retryable=True)
delay = sub.interval + timedelta(seconds=random.uniform(0, sub.jitter_s))
if sub.running: # stopped or budget-held: skip the fetch
try:
await poll_once(sub, heartbeat=activity.heartbeat) # items and cursor in one tx; post_emit after
await poll_succeeded(sub)
except Exception as exc: # counted in the database; 5 in a row: driver down
failures = await poll_failed(sub, exc)
delay = min(sub.interval * 2 ** failures, timedelta(hours=1))
raise ApplicationError("polled", type="PollCycleDone", next_retry_delay=delay,
category=ApplicationErrorCategory.BENIGN)
```
### 9.3 Long-Running Activity
**Docs:** https://docs.temporal.io/design-patterns/long-running-activity. An activity that runs for minutes to hours reports progress with heartbeats (periodic calls that prove liveness and can carry checkpoint details), resumes from its last checkpoint after a failure, and handles cancellation. Rules that matter:
- Heartbeat details "are persisted and available to retry attempts": read `activity.info().heartbeat_details` on a retry and resume from them.
- "Heartbeat timeouts detect stuck Activities faster than execution timeouts." Set the heartbeat timeout to "2-3x the expected heartbeat interval"; heartbeat "every 10-30 seconds"; keep details small ("IDs, offsets, counts"); make reprocessing the last checkpoint idempotent; "Test resumption"; avoid heartbeat spam.
- Cancellation is delivered on a heartbeat, and in Python it arrives as `asyncio.CancelledError`: clean up, then re-raise.
- Pitfalls: "Missing HeartbeatTimeout"; "Heartbeating too infrequently. Cancellation is only delivered on the next heartbeat"; not resuming from details; catching the wrong exception.
- Throttling (https://docs.temporal.io/encyclopedia/detecting-activity-failures#activity-heartbeat): heartbeats reach the server at most every `min(0.8 x heartbeat timeout, 60 s)`, or every 30 s without a heartbeat timeout. "Activities that don't Heartbeat can't receive a Cancellation."
**SDK:** 1.30.0 has:
- `activity.heartbeat(*details)` and `Info.heartbeat_details`;
- `Info.scheduled_time` (the proto's "When was this task first scheduled", the first attempt's schedule time, timezone-aware UTC), `Info.started_time` and `Info.start_to_close_timeout`;
- `activity.cancellation_details()` with `worker_shutdown`, `cancel_requested`, `paused`, `reset` and `timed_out`; `activity.is_worker_shutdown()`;
- worker throttles of 60 s maximum and 30 s default.
An activity is recorded Cancelled only when it raises the cancellation. Nothing these recommendations use changes in 1.34.0.
**OpenCompany today:**
- **Liveness heartbeats are well covered:**
| Activity | Beat every | Heartbeat timeout | Evidence |
|---|---|---|---|
| Per-type node wrapper (when start-to-close exceeds the heartbeat timeout) | 30 s | 2 min (24 h start-to-close in graphs) | `server/services/plugin/base.py:1248-1264`; `server/services/temporal/workflow.py:167`, `:845-861` |
| LLM step | 20 s while the provider call runs | 1 min | `server/services/temporal/agent_activities.py:378-399`; `agent_workflow.py:99` |
| Permit waits (small dict details) | 1 s | 10 s | `agent_activities.py:2288-2316`; `agent_workflow.py:569-576` |
| Pi driver | 30 s | 90 s | `server/services/temporal/pi_activities.py:125-128`; `pi_workflow.py:184` |
| Workspace cleanup | 5 s | 30 s | `server/services/temporal/workspace_task_activities.py:15-18` |
| Reviewed delivery | 30 s | 2 min | `server/services/temporal/employee_job_workflow.py:19-22`, `:135` |
After throttling the server sees a heartbeat every 60 s (2-minute timeout), 48 s (1-minute timeout) or 8 s (10-second timeout), all inside the docs' guidance.
- **Resumption uses durable rows keyed by the activity's identity, not heartbeat details** (`heartbeat_details` is not used anywhere in server code): the approval gate's key `t:{workflow_id}:{run_id}:{activity_id}` (`server/nodes/workflow/approval_gate/__init__.py:110-123`), reviewed delivery's claim, lease and `inflight` mark (`server/services/employees/jobs.py:315-386`), and permit lease ids per attempt. Equivalent, and it survives a Temporal rebuild (I10).
- **Done well:** permit acquisition compensates, then re-raises on cancel (`agent_activities.py:2317-2339`); the LLM step re-raises (`:575-578`); `NodeWaitInterrupted` turns a worker-shutdown interruption into a retryable failure that re-attaches (`server/services/plugin/base.py:125-143`, `:1355-1366`).
- **Gap 1: node activities swallow cancellation (RFC-0004 H2).** `NodeExecutor` turns `CancelledError` into `{success: False, error: "Cancelled"}` for every node type except Workspace-task nodes (`server/services/node_executor.py:231-239`), so the activity completes normally after a cancel request. An approved send cancelled mid-call is recorded as `not_sent` (`server/services/temporal/approved_tool_call_workflow.py:66-69`), so the owner's Retry asks no confirmation (`server/services/approvals/decisions.py:167-168`), although the message may have gone out. No RFC-0005 phase schedules H2.
- **Gap 2:** the Timer node turns cancellation into `RuntimeError("Timer cancelled")` (`server/nodes/scheduler/timer/__init__.py:81-84`); see §9.5.
- **Gap 3:** poll activities never heartbeat (§9.2).
- **Gap 4: long external jobs start over.**
- The Apify node calls `actor(...).call(...)` and waits up to the actor timeout of 1-3,600 s (`server/nodes/scraper/apify_actor/__init__.py:221`, `:287-291`) without recording the Apify run id.
- As an AI tool, its attempt runs under the plugin's 10-minute start-to-close (`server/services/plugin/scaling.py:93`).
- On new generations (3 attempts), a long actor run times out and up to three paid runs start.
- On the legacy tool path, which manual canvas runs use and which sets no retry policy (`agent_workflow.py:114-143`), a new paid Apify run starts every 10 minutes with no limit.
- In a graph, the 24-hour attempt covers the 1-hour maximum.
- **Gap 5: cancellation latency** is up to about 60 s of throttling plus the 30-second beat. The chat path works around it deliberately by polling a database stop flag (`agent_activities.py:419-444`: "a heartbeat-borne cancel arrives too late to stop a stream mid-sentence").
- **Gap 6: the beat loop runs whether or not the body progresses** (`base.py:1248-1264`), so a hung body is caught only at start-to-close: 24 hours in graphs, 30 minutes for AI nodes, 10 minutes for actions.
**Verdict:** Adopted with deviations. Heartbeat cadence and timeouts are good and resumption is durable, but node activities swallow cancellation, polls never heartbeat, and long external jobs restart from scratch.
**Recommendation:**
1. **Node activities re-raise cancellation** (RFC-0004 D9 and H2; 1b.2). In `NodeExecutor` (`server/services/node_executor.py:231-239`), on the Temporal path, re-raise `CancelledError` after cleanup. When `activity.cancellation_details()` reports `worker_shutdown`, raise `ApplicationError(type="NodeWaitInterrupted")` instead, which the wrapper passes on as a retryable failure (`base.py:1398-1400`). A cancel during a send then becomes `unknown`: `ApprovedToolCallWorkflow` sees an `ActivityError` and records `broke_off`, whose outcome is `unknown` (`server/services/approvals/execution.py:148-155`), so the owner's Retry asks for confirmation; a retried graph or tool send meets the redo guard (1b.1, §11.2). Versioning: none (activity body; old histories keep their recorded results). When: P1, Phase 1b (1b.2). Effort S-M. Test: RFC-0004 gate AG10, and a time-skipping `ApprovedToolCallWorkflow` run whose send is cancelled records `unknown`. Risk: agent tool calls use `WAIT_CANCELLATION_COMPLETED` (`agent_workflow.py:145-156`) and will see a Cancelled `ActivityError` where they now see a failed result; check the tool-result path (`agent_workflow.py:2574-2628`) and the pause-on-failure streak.
2. **Long waits fit one attempt.** Dropped on 2026-10-25 (1b.4, §22.2). A shared helper `attempt_deadline()` in `server/services/plugin/` returns `info.started_time + info.start_to_close_timeout` minus a 60-second margin, or None outside an activity. The Timer node (§9.5 item 1) and the Apify node refuse, with `NodeUserError`, a wait or actor timeout that would end after it. It applies on every path, because the helper reads the attempt's own timeout; no branch on the path. Together with 1b.3 (§11.2: the legacy tool path honours each plugin's declared retry policy and timeouts), a long actor run as a tool costs at most one paid run. Versioning: none. When: P1, Phase 1b (1b.4; the Timer part is 1b.8). Effort S.
3. **Poll attempt deadlines**: §9.2 item 3 (P1, Phase 1b, 1b.17, built without heartbeats).
4. **Checkpoint external job ids** (§5 rule 20). Start the Apify run with `actor(...).start()`, save its run id in a database row keyed by the activity's identity and in the heartbeat details before waiting, and on a retry wait on that run (`run(run_id).wait_for_finish()`) instead of starting another. The row is the durable record: heartbeat details alone are lost when the activity is scheduled again after a rollover, and rows survive a Temporal rebuild (I10). The plugin beat loop, which today overwrites the details every 30 s with a status string, must repeat the run id instead. This makes the retry idempotent, as T13 requires for redoing a side-effecting call (1b.1, §11.2). Versioning: none. When: P2, RFC-0005 Phase 7. Effort M.
5. **Progress heartbeats and resumption tests.** Plugins that can report progress heartbeat on progress instead of the fixed 30-second beat, so a hung body is caught by the heartbeat timeout; add `ActivityEnvironment` tests for resume after a crash. When: P3. Effort M.
**Remarks:**
- The embedded worker shares uvicorn's event loop (RFC-0004 M1). A blocked loop stops heartbeats and causes false heartbeat timeouts and duplicate attempts. Sends refuse a second attempt (`server/services/approvals/tool_calls.py:367-379`) only for plugins with an approval spec; T13 extends the rule to every side-effecting call, graph nodes included (1b.1, §11.2).
- Keep the LLM step's beat at 20 s or less: its 1-minute timeout throttles heartbeats to every 48 s, a 12-second margin.
- Tests exist for the shutdown re-attach (`server/tests/services/test_node_wait_interrupted.py`); none for resuming an external job after a crash.
**Sketch:** (items 1 and 2)
```python
# server/services/plugin/base.py (helper shared by Timer and Apify)
def attempt_deadline(margin_s: float = 60.0) -> datetime | None:
"""When this attempt's start-to-close timeout fires, minus a margin."""
from temporalio import activity
if not activity.in_activity():
return None
info = activity.info()
if info.start_to_close_timeout is None:
return None
return info.started_time + info.start_to_close_timeout - timedelta(seconds=margin_s)
# server/services/node_executor.py:231-239 (RFC-0004 H2)
except asyncio.CancelledError:
from temporalio import activity
if activity.in_activity():
details = activity.cancellation_details()
if details is not None and details.worker_shutdown:
# Not a failure: the next attempt re-attaches (NodeWaitInterrupted contract).
raise ApplicationError("Worker shutting down", type=NODE_WAIT_INTERRUPTED) from None
raise # recorded Cancelled, as Temporal expects
return ExecutionResult(False, node_id, node_type, error="Cancelled",
execution_id=execution_id,
execution_time=time.time() - start_time).to_dict()
```
### 9.4 Delayed Start
**Docs:** https://docs.temporal.io/design-patterns/delayed-start. `start_delay` creates the execution and reserves its id at once, but no workflow code runs and no worker is used until the delay passes. Rules that matter:
- "If the client sends a Signal-With-Start or Update-With-Start during the delay, the remaining delay is bypassed ... Regular Signals do not interrupt the delay." Handlers do not run during the delay, and Queries "have no state to return".
- "You cannot dynamically adjust the delay after creation (use the Updatable Timer pattern for that)."
- One-time delays only (recurring work belongs on Schedules); not for sub-second delays ("Treat the configured duration as a minimum, not an exact value").
- To cancel before it runs, cancel the execution or use Signal-With-Start.
**SDK:** `Client.start_workflow(..., start_delay=timedelta)` in 1.30.0 (`temporalio/client/_client.py:414`) and 1.34.0. Child workflows have no equivalent.
**OpenCompany today:**
- Not used: `start_delay` appears nowhere in server code.
- The only workflow whose first operation is a sleep is `ApprovedToolCallWorkflow`. It sleeps to `grace_until`, 5 s after Send (`server/services/temporal/approved_tool_call_workflow.py:41-46`), and has no handlers.
- Every other delay is an in-workflow timer or a misused `asyncio.sleep` (§9.5).
**Verdict:** Not applicable.
**Recommendation:**
1. **No adoption.** `start_delay` would save `ApprovedToolCallWorkflow` one timer per send, but RFC-0005 Phase 7 replaces it with `ApprovalWorkflow`, whose Undo window must accept an Update. A delayed start runs no handler, and Update-With-Start would bypass the delay. This corrects RFC-0004 §4.2, which lists `start_delay` for the Undo window. When: never. Effort none.
2. **A rule for §5:** never use `start_delay` on an id that the relay addresses with Signal-With-Start (`conv-...`, RFC-0005 §9.4), because any delivery would bypass it. Debounce, quiet hours and restart backoff stay in-workflow timers, as RFC-0005 §11 and §18.2 already have them. Versioning: none (text). When: with this RFC. Effort S.
**Remarks:**
- The page's "Cancellable delayed execution" sample relies on a cancel Signal that cannot run during the delay; the page admits this afterwards.
- `ApprovedToolCallWorkflow` decides whether to sleep from `workflow.now()`, which replays deterministically, so skipping a past deadline never turns into a timer change from or to zero on replay.
### 9.5 Delayed Callback (Webhooks)
**Docs:** https://docs.temporal.io/design-patterns/delayed-callback. Three patterns that make HTTP callbacks durable:
- **Pattern 1, inbound:** the HTTP handler calls Signal-With-Start; "The handler can return an HTTP 200 immediately after this call; Temporal takes responsibility for delivery." Workflow ids are business ids, "deterministic and stable across webhook deliveries". "Sending a plain Signal to a Workflow that does not exist causes an error."
- **Pattern 2, outbound after a delay:** `workflow.sleep(delay)`, then an activity whose retries cover HTTP failures. "Using `time.sleep()` (non-durable) ... A process sleep disappears on restart; only Temporal's timer is durable." Not for delays under one second.
- **Pattern 3, async completion:** the activity stores `activity.info().task_token` (an opaque handle for completing the activity from outside) "in a transactional write alongside the job submission", calls `activity.raise_complete_async()`, and the callback handler completes it with `client.get_async_activity_handle(task_token=...)`. For a human review that can take a week, the activity-execution page prefers a short activity plus a Signal (https://docs.temporal.io/activity-execution#asynchronous-activity-completion).
**SDK:** All in 1.30.0 and 1.34.0: Signal-With-Start through `start_signal` (`temporalio/client/_client.py:415`), `activity.raise_complete_async()`, `Client.get_async_activity_handle`, and `AsyncActivityHandle.heartbeat`, `complete`, `fail` and `report_cancellation`.
**OpenCompany today:**
- **Pattern 1.**
- `server/routers/webhook.py:59-117` hands the request to `deliver_webhook`, which builds a route table from the generation snapshots and answers 404, 405 or 401 (`server/nodes/trigger/webhook_trigger/_routes.py:59-101`), then calls `emit` with `broadcast=False` (`server/nodes/trigger/webhook_trigger/_events.py:43-48`).
- `emit` runs a Visibility query, then sends a plain `on_event` Signal (`server/services/events/dispatch.py:155-165`, `:251-258`).
- The route answers 200 whether or not any controller received the event: `emit` logs and returns when the client is missing, the query fails, nothing matches or a Signal fails (`dispatch.py:118-130`, `:155-165`, `:186-191`, `:196-199`). Signed providers (Stripe, Dodo Payments, WhatsApp Business) feed the same `emit`. Delivery is at most once (RFC-0004 H1).
- RFC-0005 Phase 2 replaces this. Ingress appends to a database inbox and commits before the 2xx; a relay sends Signal-With-Start to the business-stable entity id `conv-<workflow_id>-g<gen>-<lane>-<thread>` (RFC-0005 §8, §9.4).
- **Pattern 2, adopted correctly in `ApprovedToolCallWorkflow`** (`server/services/temporal/approved_tool_call_workflow.py:33-80`): it sleeps to `grace_until`, claims the row at its revision, sends once with `SEND_ONCE = RetryPolicy(maximum_attempts=1)` (`:30`, `:57-72`), and records `sent`, `not_sent` or `unknown`. The reconciler restarts sends that never started and ends stale `sending` rows as `unknown` (`server/services/approvals/reconcile.py:76-110`).
- **Pattern 2 misused, the Timer node** (`server/nodes/scheduler/timer/__init__.py`): `asyncio.sleep(wait_seconds)` inside an activity (`:81-84`); durations of 1-86,400 in seconds, minutes or hours (`:18`), while the tool text says 1-3,600 (`:38`); on `machina-default` (20 slots per worker, 10 in local mode, `server/services/temporal/worker.py:413-423`, `:477`); usable as an AI tool (`:45`). Consequences:
- As a tool on new generations: the plugin's 10-minute start-to-close and 3 attempts (`server/services/plugin/scaling.py:58`, `:93`; `agent_workflow.py:145-156`). A 20-minute wait fails after about 30 minutes, and the model receives an `ActivityError`.
- As a tool on the legacy path (manual canvas runs): no retry policy (`agent_workflow.py:114-143`), so the wait times out every 10 minutes and is retried without limit. The call never ends.
- In a graph: 24 hours per attempt (`server/services/temporal/workflow.py:167`, `:845-861`). A crash restarts the full delay on the next attempt, and a delay over 24 hours fails after three attempts (about 72 hours).
- A graceful shutdown that outlasts the grace period cancels the activity. The node turns the cancel into `RuntimeError`, the plugin returns it as a failed result (`server/services/plugin/base.py:766-768`), and MachinaWorkflow records the node as failed (`workflow.py:893-908`): the node and the run fail, with no retry. Only a crash restarts the delay.
- A user's "wait 2 hours, then call this URL" graph is exactly pattern 2, and it is not durable. The cron node's canvas Run does the same for up to 24 hours (`server/nodes/scheduler/cron_scheduler/__init__.py:274`, `:312-315`); deployed cron uses Schedules.
- **Pattern 3:** not used. No node submits a job that calls back; Apify waits through its SDK's own polling.
**Verdict:** Misused. The Timer node implements pattern 2 with a process sleep, and inbound webhooks answer 200 without any delivery until RFC-0005 Phase 2. `ApprovedToolCallWorkflow` adopts pattern 2 correctly, with no retries by decision E12, and pattern 3 is not applicable. RFC-0005's inbound design deviates from pattern 1 on purpose: the inbox commit comes before Temporal (I2, I10), which survives Temporal outages and gives deduplication and 429/503 backpressure.
**Recommendation:**
1. **Timer node, interim fix** (activity body only):
- Anchor the deadline to `activity.info().scheduled_time` (the first attempt's schedule time), so a retry after a crash or a restart waits only the remainder.
- Refuse, with `NodeUserError`, a wait that does not fit one attempt (§9.3 item 2), on every path including the legacy tool path. For a tool call this caps waits near 9 minutes; in a graph, near 24 hours.
- On a worker-shutdown cancellation raise `NodeWaitInterrupted` (retryable), and otherwise re-raise the cancellation.
It still holds a worker slot for the whole wait. Versioning: none. When: P1, Phase 1b (1b.8). Effort S.
2. **Durable Timer.** MachinaWorkflow schedules no activity for a `timer` node and awaits `workflow.sleep` instead, for runs whose frozen input has `durable_timer_version=1`: set in `resource_manifest` by `begin_generation` (`server/services/deployment/control.py`) and in the executor's run input for manual runs (`server/services/temporal/executor.py:140-152`). The parameters come from `parameter_snapshot`, or from one small activity when a manual run has none. The node's output shape stays the same, and status broadcasts use the existing maintenance activities. Sleeps are not business work for a Stop checkpoint (`docs-internal/temporal-workflow-control.md:117-119`): Stop completes during the wait, the delay keeps counting, and the next node waits for Resume. Versioning: per-generation (and per-run) input flag, so no patch. When: P2, RFC-0005 Phase 6 (§22.3); it may land earlier, in the controller lane after Phase 1. Effort M.
3. **As an AI tool, keep the one-attempt cap.** Agents that need to wait longer use RFC-0005's alarms (`set_alarm`, Phase 6) (default, §24). Versioning: none. When: Phase 6 documents it in the tool text. Effort S.
4. **Pattern 1: RFC-0005 Phase 2 as designed.** Keep entity ids business-stable and tie the 2xx to the commit (EG9). When: P2, Phase 2.
5. **Pattern 3, if ever needed.** Where a provider can call back (Apify run webhooks, batch APIs), the callback becomes an inbox row delivered to the entity, and the submitting activity completes with the job id. A callback handler that completes a task token would call Temporal from a request path, against I2. Use task tokens only for an external process that must heartbeat or receive cancellation. When: P3.
**Remarks:**
- `webhookResponse`'s "responseNode" mode is not wired: `_pending_responses` is never filled (`server/routers/webhook.py:41`, `:51`). If it is built, it cannot wait on Temporal from the request path (I2).
- The page's pattern 3 sample opens a new `Client` per callback; reuse the app's client.
- The 5-second Undo window satisfies "not under one second".
**Sketch:** (items 1 and 2)
```python
# Item 1 (Phase 1b): server/nodes/scheduler/timer/__init__.py, TimerNode.wait
wait_seconds = _seconds(params) # unchanged unit arithmetic
deadline = datetime.now(timezone.utc) + timedelta(seconds=wait_seconds)
if activity.in_activity():
# Anchor to the first attempt's schedule time: a retry waits only the remainder.
deadline = activity.info().scheduled_time + timedelta(seconds=wait_seconds)
limit = attempt_deadline() # §9.3 item 2
if limit is not None and deadline > limit:
raise NodeUserError(f"A wait of {params.duration} {params.unit} does not fit in "
"one attempt. Use a shorter wait, or an alarm.")
try:
await asyncio.sleep(max(0.0, (deadline - datetime.now(timezone.utc)).total_seconds()))
except asyncio.CancelledError:
if activity.in_activity() and activity.is_worker_shutdown():
raise NodeWaitInterrupted("worker shutting down during a Timer wait") from None
raise
# Item 2 (P2): server/services/temporal/workflow.py, MachinaWorkflow dispatch
if node_type == "timer" and workflow_data.get("durable_timer_version") == 1:
running[node_id] = asyncio.create_task(self._durable_timer(node_id, timer_params))
continue # no activity, no admitted action
async def _durable_timer(self, node_id: str, params: dict) -> dict:
started = workflow.now()
await workflow.sleep(timedelta(seconds=timer_seconds(params)), summary=f"Timer {node_id}")
return {"success": True, "result": {
"timestamp": workflow.now().isoformat(), "duration": params.get("duration"),
"unit": params.get("unit"),
"elapsed_ms": int((workflow.now() - started).total_seconds() * 1000)}}
```
### 9.6 Approval
**Docs:** https://docs.temporal.io/design-patterns/approval. A workflow blocks until a person or an external system decides, with a timeout. Rules that matter:
- Block with a timeout: `await workflow.wait_condition(lambda: self.approval_data is not None, timeout=...)`, which raises `asyncio.TimeoutError`.
- The decision is a rich object (approver, decision `APPROVED` / `REJECTED` / `ESCALATED`, comments, timestamp). It arrives by Signal, or, "If the approver's interface needs immediate confirmation that the approval was accepted and valid, consider using an Update with a validator instead of a Signal."
- A Query for status; multi-level chains; escalation (on timeout, notify, then wait longer).
- Best practices: reasonable timeouts; "Validate Signal data. Verify approver permissions"; log every decision with timestamps; "Support cancellation"; "Ensure idempotency ... Signals may be duplicated"; expose approval history.
- Pitfalls: no timeout; missing validation; lost context; no audit trail; tight timeouts; boolean-only Signals; no status Query; no duplicate handling; "Design your timeout path to account for late-arriving Signals"; "No escalation path".
- The page's comparison table accepts a polling activity for external approval systems, at high complexity: "Polling Activity | Yes | Yes | Yes | High | External approval systems". The activity-execution page prefers a short activity plus a Signal for a review that "could take up to a week" (https://docs.temporal.io/activity-execution#asynchronous-activity-completion).
**SDK:** 1.30.0 has `@workflow.update` with `@<update>.validator`, `workflow.wait_condition(timeout=)`, `@workflow.query` and `workflow.all_handlers_finished`. Update ids are deduplicated per run. 1.34.0 changes nothing used here (the Update-handler rule of §8.2 does not apply: an approval never continues as new).
**OpenCompany today:**
1. **Decision semantics** (`server/services/approvals/`):
- Rows in the app database. Each decision is a compare-and-swap on `revision`, with a `decision_key` and the actor recorded in `approval_decisions` (`server/services/approvals/decisions.py:116-196`).
- Owner-only (`server/services/approvals/handlers.py:172-175`). Expiry is applied before any decision, so a late Send gets `expired` (`decisions.py:65-81`, `:135`).
- Default expiry 168 hours (`server/services/approvals/contract.py:25`); the gate allows 1-720 hours (`server/nodes/workflow/approval_gate/__init__.py:69`); Undo and Restore 5 s (`contract.py:35-39`).
- Status comes from the list handlers and the identity-only `approval_lifecycle` broadcast; the employee reads the outcome as an `[update]` note.
- Not found: a comment or reason field, and any reminder or escalation before expiry.
2. **Held tool calls** (Ask first) wait on nothing: the tool answers that it is waiting for the owner and the agent goes on (`server/services/approvals/tool_calls.py:178-224`). Send starts `ApprovedToolCallWorkflow` from the request handler, and reverts the decision with `send_unavailable` when Temporal is down (`handlers.py:139-159`, `:199-209`). Neither `start_send` nor `claim_send` checks whether the employee is stopped, and `will_send_on_resume` is reported for gates only (`server/services/approvals/execution.py:58-70`; `handlers.py:215-216`): a held call approved while the employee is stopped goes out after the Undo window anyway (by code reading).
3. **The approvalGate node in graphs** waits inside its activity:
- It re-reads its row at least every 10 s, woken earlier by a process-local notifier (`approval_gate/__init__.py:47-48`, `:235-277`; `server/services/approvals/waiter.py:1-6`).
- Each attempt runs up to 24 hours with unlimited retries (`:181-185`), re-attaching to the same row by the activity's identity (`:110-123`). On worker shutdown it raises `NodeWaitInterrupted` (`:274-275`).
- Each waiting draft holds one `triggers-event` slot: 100 per worker, 50 in local mode, shared with every other event-trigger activity (`server/services/temporal/worker.py:413-423`, `:477`).
4. **Reviewed team-job delivery** waits in three 31-day activities, each with 3 attempts:
- `employee.job.deliver` in `EmployeeJobWorkflow` (`server/services/temporal/employee_job_workflow.py:132-139`);
- `employee-reviewed-delivery` in `AgentWorkflow` (`agent_workflow.py:3133-3138`, `AGENT_ACTIVITY_RETRY = DEFAULT_ACTIVITY_RETRY` at `:184`);
- `employee.job.resolve_delivery` in `EmployeeJobDeliveryWorkflow` (`employee_job_workflow.py:155-157`).
The gate runs inline (`server/services/employees/jobs.py:376-381`), on `machina-tasks`, whose 100 slots it shares with LLM steps. Defects (by code reading):
- A shutdown during a pending draft makes the gate raise `NodeWaitInterrupted`, which `deliver_job` turns into `RuntimeError` (`jobs.py:385-386`): a failed attempt. If the next attempt starts after the 2-minute delivery lease has expired (set and renewed every 30 s, `jobs.py:319-320`, `:358-366`), it re-attaches to the draft and uses one attempt; the third interruption fails the delivery activity and the workflow. The job row stays `delivering`, because the delivery call sits outside the code that runs `employee.job.failed` (`employee_job_workflow.py:114-126`).
- If the next attempt starts within the lease (a quick restart, or a crash whose heartbeat timeout fires before the lease ends), it returns `{"delivered": False, "state": "delivering"}` (`jobs.py:273-275`) and completes. The workflow ends successfully with the draft still pending and nothing waiting on it.
- Nothing expires such a stranded gate row while the generation lives: `expire_due` ends only `tool_call` rows (`server/services/approvals/reconcile.py:113-125`), and live generations keep their gate rows (`:53-63`). `reconcile_jobs.py` repairs only jobs whose lead finished without assignments.
5. **Stop and pending drafts.** Every node activity and the reviewed-delivery activities are admitted business actions (`workflow.py:865-868`; `employee_job_workflow.py:132-136`; `agent_workflow.py:3134`). Stop's checkpoint waits until admitted actions reach zero, with no deadline (`server/services/temporal/execution_control.py:114-119`; `server/services/deployment/execution_control.py:144-169`). So an employee with a pending draft stays "pausing" until the owner decides or the draft expires. Crash recovery's boot pause (`server/services/deployment/handlers.py:1511-1541`) goes through the same path and stays "pausing" too. `handlers.py:215-216` already models "will send on resume". No test covers it: `server/tests/services/approvals/` has no pause or checkpoint test.
6. **Browser "Needs you"** waits in memory in the browser activity, 600 s by default (60-1,800 s); each tool call waits up to 480 s and then answers `still_waiting` (`server/nodes/browser/browser/__init__.py:104-105`, `:210`, `:556-564`). Deliberate: Chrome dies with the process.
**RFC-0005 plan, before the amendments in §20 (RFC-0005 §10.1, §10.2, §17.5; RFC-0004 D13):** `ApprovalWorkflow` with id `approval-<id>`; an Update `decide{decision, revision, key}` and a `status` Query; timers for expiry, then the Undo window; one send activity, then `approval.decided` appended to the origin thread; it replaces `ApprovedToolCallWorkflow`, with `expire_due` as a backstop. It names held sends only: not the gate node, and none of the three delivery waits.
**Verdict:** Misused (the wait). The verdict rests on the owner's decision (RFC-0004 Q10, D13: "Human waits become an Update plus a durable timer") and on the measured costs: a worker slot per waiting draft, Stop held in "pausing" for up to the draft's expiry (168 hours by default, 720 at most), and a reviewed delivery that a restart can strand. The page's comparison accepts a polling activity for external approval systems, at high complexity, so the verdict does not rest on the page alone. The decision semantics are adopted with deviations (no comment field, no reminder or escalation). Keeping the record in the database rather than in history is correct here: history is kept 24 hours today and 7 days in RFC-0005, and T5 makes Temporal disposable. RFC-0005's `ApprovalWorkflow` follows the docs (an Update with a validator plus a durable timer) but misses the gate node, the three delivery waits, and the details in item 4.
**Recommendation:**
1. **Reviewed delivery survives restarts** (1b.9; sketch part a):
- Give the three 31-day activities one dedicated policy, `REVIEWED_DELIVERY_RETRY = RetryPolicy(maximum_attempts=0, initial_interval=2 s, maximum_interval=1 min)` in `server/services/temporal/_retry_policies.py`, used at `employee_job_workflow.py:132-139`, `:155-157` and `agent_workflow.py:3133-3138`.
- Let a retry re-attach to its own lease. `deliver_job` records `lease_holder` = `<workflow_id>:<activity_id>` (from `activity.info()`, stable across the activity's retries) with the lease, and treats an unexpired lease held by the same holder as its own instead of answering `delivering`.
- When the gate's result is a wait interruption, `deliver_job` raises `NodeWaitInterrupted` instead of `RuntimeError`, so the attempt fails retryably and nothing records a failure.
- A clean failure clears `inflight`. When a sink answers `success: False`, `deliver_job` clears the `inflight` mark and the lease, records the job `failed`, and returns that state instead of raising. Today it raises first (`jobs.py:385-386`, before `:393` clears the mark), so a clean rejection is shown as "Check whether it arrived" (`jobs.py:412-418`). Returning also keeps the unlimited policy from repeating a send the sink refused.
A send that broke off is not retried forever: the `inflight` write-ahead mark turns a retry into `delivery_needs_review` (`jobs.py:276-281`). Versioning: none; retry policies are not compared on replay. When: P1, Phase 1b (1b.9). Effort S.
2. **Expire stranded drafts.** `expire_due` (`reconcile.py:113-125`) also expires gate rows past `expires_at`. A live gate would expire its own row at the same moment, and compare-and-swap keeps one result, so this is only a backstop for rows that nothing waits on. Versioning: none. When: P1, Phase 1b (1b.10). Effort S.
3. **T12 before Phase 7.** Two interim fixes. Reviewed delivery keeps holding Stop until item 4, because its gate and its send share one activity (default, §24).
- **Approved sends wait while the employee is stopped** (1b.5). `claim_send` (`server/services/approvals/execution.py:101-128`) answers `{"claimed": false, "reason": "stopped"}` and leaves the row `approved` while the row's generation is not running; `recover_sends` (`server/services/approvals/reconcile.py:76`), which the chat watchdog runs every 30 s (1b.6, merged here), skips such rows while the generation is stopped and starts them once it runs; a row whose generation ended is cancelled and the employee told; Undo works until the claim. Under RFC-0007 this covers drafts only (generations without `approval_wait_version=1`, runs that answer no chat run, CLI and in-process calls) and Retry; a row whose turn waits for it is left to that turn. `will_send_on_resume` is then reported for these rows too, so the card says the message goes when the employee resumes. Rows without a deployment keep today's behaviour. Versioning: none (activity bodies and the Resume handler). When: P1, Phase 1b (1b.5). Effort S. Test: a send approved while paused is not sent and its row stays `approved`; the sweep skips it until the generation runs, then sends it once.
- **Stop completes while a graph draft waits** (graph gates). For runs whose frozen input has `approval_wait_version=1` (set in `resource_manifest` by `begin_generation`), MachinaWorkflow still schedules an `approvalGate` activity only while the employee is running, as today, but no longer counts it in the checkpoint. Stop and the boot pause then complete while the draft waits. A decision taken while stopped is consumed by the run, and the next node, the send, waits for Resume through its own admission (`begin_action`). Versioning: per-generation input flag. When: Phase 1b, as RFC-0007 AP.5; RFC-0007 AP.7 starts writing the flag one release later. RFC-0007 widens `approval_wait_version=1` to the agent tool calls of chat runs, which wait in the turn (RFC-0007 §4.6). Effort S. The test comes first: it should show today's "pausing" and then the fix.
4. **`ApprovalWorkflow` for all four waits** (RFC-0004 D13, T10, T12; sketch part b). Amended by RFC-0007 on 2026-10-25:
- **Held tool calls get it first,** in generations with `approval_wait_version=1`. This applies to the agent that answers a chat run.
- **Its form** is nudge and re-read: no `decide` Update, no validator and no Undo window (D4).
- **`ApprovedToolCallWorkflow` stays** for drafts and for Retry. Drafts are older generations, runs that answer no chat run, and CLI and in-process calls.
- **Phase 7** moves the gate and reviewed delivery onto the same type (`approval_wait_version=2`).
- The bullets below and sketch (b) show the design before RFC-0007:
- **Coverage.** The approvalGate node (MachinaWorkflow runs a short `approvals.open` activity, then starts a child `ApprovalWorkflow` and awaits it, gated by the generation flag `approval_wait_version=2`; the `child_workflow` dispatch kind already exists, `workflow.py:801-830`); the gate step of reviewed delivery in `EmployeeJobWorkflow` and `AgentWorkflow` (new generations); the resolution path in `EmployeeJobDeliveryWorkflow`; held tool calls, replacing `ApprovedToolCallWorkflow`. Child-result waits are excluded from Stop checkpoints, which removes the Stop block and the slot cost.
- **Database first (T10).** `decisions.apply` stays the record and the immediate answer; the T10 deliverer then sends `decide` with the decided row and `id=decision_key` until it is applied, and the UI shows "applying". Deciding still works while the worker or Temporal is down.
- **Validator.** The state allows the decision and the revision is newer; a stale or final request is rejected without writing history.
- **Absolute deadlines** (`expires_at`, the reminder, `grace_until`, `restore_until`) in one timer loop (§8.4 item 1), so Undo returns to the original expiry instead of starting a fresh 168-hour wait.
- **Late decisions.** When a deadline fires, an activity re-reads the row, and a decision committed just before the timer wins (the page's late-arriving Signal). A decision delivered after the workflow closed is answered with the row's final state (`expired`).
- **T12 for held tool calls.** The send step keeps item 3's rule (1b.5): it claims the row only while the generation runs. The supervisor's Stop and Resume fan-out also reaches open `ApprovalWorkflow`s, whose ids come from the approval table (RFC-0005 I6), so a send decided while stopped waits without holding a slot and goes out once on Resume.
- **Reminder** (default, §24): one attention item 24 hours before expiry (kind `approval_expiring`), from the reminder deadline; drafts whose whole lifetime is 24 hours or less get none.
Versioning: new type; the gate child behind `approval_wait_version=2`, set by `begin_generation` from Phase 7. When: P2, RFC-0005 Phase 7 (attention items from Phase 8). Effort L. Risk: the Undo window spans Update and timer races; test them with time skipping (RFC-0004 §14).
5. **Comments and escalation.** A reason field on decisions and escalation beyond the reminder are left to the owner (open item, §24). When: P3.
**Remarks:**
- Signal or Update: the card needs immediate answers (`too_late`, `already_decided`, `approval_conflict`). Under I2 they come from the database compare-and-swap; the Update only carries the decided state into the workflow, which is why a decision works while the engine is down.
- Duplicate decisions are deduplicated twice: by `decision_key` in the database and by the Update id within the run. Update ids do not survive continue-as-new, but an approval never continues as new.
- Audit: do not rely on "All approval data is recorded in the Workflow history". Retention and T5 delete history, so `approval_decisions` stays the audit trail.
- Payload size: drafts are at most 20,000 characters (`approval_gate/__init__.py:49`), fine in an Update; pass a reference where possible (I7).
- Under T8 (workers in a second process), a decision made in the API process does not wake a gate waiting in a worker process, because the notifier is process-local (`waiter.py:1-6`). The gate then sees the decision at its next 10-second re-read. Item 4 removes the re-read.
- Browser help: a workflow-level wait would save repeated model steps, but the in-memory wait is bounded and deliberate. P3.
- Turn deadlines. Under item 4, a turn that waits on an approval child (the gate node, the delivery gate) completes no step while it waits. The 24-hour no-progress backstop (a default, §24.1; §6.4) never counts time waiting for the owner, as it never counts time parked or paused, so a draft left for more than a day does not cancel its turn.
**Sketch:** (part a: item 1, Phase 1b; part b: item 4, Phase 7)
```python
# (a) server/services/employees/jobs.py, deliver_job: a retry keeps its own lease
info = activity.info() if activity.in_activity() else None
holder = f"{info.workflow_id}:{info.activity_id}" if info else None # same on every retry
lease = job.delivery.get("lease_until")
if lease and datetime.fromisoformat(lease) > _now() and not (holder and job.delivery.get("lease_holder") == holder):
return {"delivered": False, "state": "delivering", "job_id": job.id}
# The claim then writes "lease_holder": holder beside "lease_until".
# After the sink call, in place of the RuntimeError at jobs.py:385-386:
if not result.get("success"):
if result.get("error_type") == NODE_WAIT_INTERRUPTED:
raise NodeWaitInterrupted("reviewed delivery wait interrupted") # retryable; same lease
await record_refusal(database, job.id, claim) # clears inflight and the lease; state "failed"
return {"delivered": False, "state": "failed", "job_id": job.id}
# (b) server/services/temporal/approval_workflow.py: database first, one timer; sandboxed (§17.18)
with workflow.unsafe.imports_passed_through(): # app modules
from services.temporal._retry_policies import BOOKKEEPING_RETRY
from services.temporal.deadlines import Deadlines # §8.4 helper
@workflow.defn(name="ApprovalWorkflow")
class ApprovalWorkflow:
@workflow.init
def __init__(self, ref: dict) -> None:
self._deadlines = Deadlines() # §8.4 helper
self._arm(dict(ref["row"])) # status, revision, deadlines in UTC
def _arm(self, row: dict) -> None: # Undo returns to the original expiry
self._row = row
for name, at in deadlines_of(row).items():
self._deadlines.set(name, at)
@workflow.update
async def decide(self, row: dict) -> dict: # the row decisions.apply committed (T10)
self._arm(row)
return {"status": row["status"], "revision": row["revision"]}
@decide.validator
def _check(self, row: dict) -> None: # rejected requests write no history
if self._row["status"] in FINAL or row["revision"] <= self._row["revision"]:
raise ApplicationError("stale_or_final")
@workflow.run
async def run(self, ref: dict) -> dict:
while self._row["status"] not in FINAL:
due = await self._deadlines.wait(lambda: self._row["status"] in FINAL)
if due: # the database says what a deadline means now; a late decision wins
self._arm(await workflow.execute_activity(
"approvals.on_deadline", {"id": ref["id"], "due": due},
start_to_close_timeout=timedelta(seconds=30), retry_policy=BOOKKEEPING_RETRY))
return {"status": self._row["status"]} # a held tool call's SEND_ONCE step is omitted
```
## 10. Distributed transactions
### 10.1 Distributed Transaction Patterns (category overview)
**Docs:** https://docs.temporal.io/design-patterns/distributed-transaction-patterns. A selection guide for work that spans services "that each own their own data, with no shared database transaction to roll back".
- "You need to undo completed steps when a later step fails": use the Saga Pattern and "define a compensation for every step that has an external effect".
- "You need to respond to the caller before the transaction finishes": use Early Return.
- Retry patterns "control how each step retries before a compensation triggers"; Update-with-Start (one call that starts a workflow and sends it an Update, a synchronous message that returns a result) is the mechanism behind Early Return.
**SDK:** no dedicated API. A saga is a list of callables in Python (there is no Java-style `Saga` helper). Early Return uses `Client.execute_update_with_start_workflow` or `start_update_with_start_workflow` with `WithStartWorkflowOperation`. All of it exists in 1.30.0 and in 1.34.0.
**OpenCompany today:** multi-step flows use four techniques, roughly in order of preference.
- **One local transaction plus an outbox** (a row that a background loop turns into the next step):
- Hire writes the graph, parameters, employee row, grants and an activation intent `hire:<id>` in one reserved transaction (`server/services/employees/store.py:140-200`, intent at `:192`). The start runs from the intent, in the background and from a 10-second recovery loop (`server/services/employees/hire.py:292-297`; `server/services/employees/activation.py:13-75`).
- Chat admission writes the owner's message and its run in one reserved transaction (`server/services/chat/ledger.py:252-331`).
- A team job writes its `EmployeeJob` row first, then starts with `REJECT_DUPLICATE` and the row id; the same loop re-drives it (`server/services/employees/jobs.py:128-187`, `:208-252`).
- Context archive receipts are written with the graph save and drained idempotently later (`server/services/workflow_storage/handlers.py:90-135`).
- **Irreversible step last:** deleting a workflow runs Reset first and deletes only after Reset succeeds (`server/services/workflow_storage/deletion.py:26-84`). An approved send waits out its Undo window, claims its row at a revision, sends once and records `sent`, `not_sent` or `unknown` (`server/services/temporal/approved_tool_call_workflow.py:41-79`).
- **Forward recovery with a write-ahead mark:** reviewed job delivery marks a sink `inflight` before calling it; an expired mark becomes `delivery_needs_review` (`server/services/employees/jobs.py:276-281`, `:351-357`).
- **Compensating sagas:** Start (`server/services/deployment/handlers.py:2157-2186`), Apply's rollback (`server/services/employees/safe_apply.py:246-279`), adding an MCP connector (`server/nodes/mcp/_store.py:175-183`; `server/nodes/mcp/_handlers.py:91-106`), an approval decision (`server/services/approvals/handlers.py:139-159`) and `DelegatedTaskWorkflow` (`server/services/temporal/agent_workflow.py:535-728`).
**Verdict:** Adopted with deviations. The technique chosen for each flow is sound; the deviations are in how the compensating sagas are written (§10.2).
**Recommendation:**
1. Write OpenCompany's rule into §5 and `docs-internal/DESIGN.md`:
- prefer one local transaction plus an outbox;
- put irreversible effects last;
- use a saga only where several systems change;
- guard an effect that has no compensation (a sent message, a refund) before the pivot with an Undo window and a claim at a revision, and after it report `unknown` (E12, extended to every side-effecting tool by T13), never a compensating "unsend";
- owner actions write an intent, answer, and let a deliverer apply them (T10).
- Versioning: none (docs). When: §5 now; `DESIGN.md` in Phase 1b (1b.45). Effort S. Risk: none.
**Remarks:**
- Most OpenCompany effects cannot be compensated. The Saga page lists "processes where compensations cannot be defined" as not a good fit; Undo windows, claims at a revision and `unknown` are the right answer.
- Idempotency keys: the Fixed Count page asks for "a stable identifier derived from the Workflow and Activity IDs". RFC-0004 §9.4 forbids the run id, which changes across continue-as-new (ending a run and starting a fresh one with the same workflow id), and RFC-0005 §7.2 and §21.3 follow that. Every key this RFC adds uses the workflow id and the activity id.
### 10.2 Saga
**Docs:** https://docs.temporal.io/design-patterns/saga-pattern. A saga runs a distributed transaction as a sequence of local steps. Each step with an external effect has a compensation, and on failure the compensations run "in reverse order". Rules:
- Register each compensation before its step runs ("recommended for safety"), so it also covers a step that failed after a partial effect. "All compensations must be idempotent and able to handle cases where the forward Activity never executed."
- Run compensations last in, first out, then "Re-throw the original error after compensating".
- "Log compensation failures but continue"; "In production, alert for manual intervention on persistent compensation failures."
- "Set a `StartToCloseTimeout` but avoid `ScheduleToCloseTimeout` on compensations. Do not set Workflow-level timeouts — let compensations retry until they succeed." Pitfall: "Compensations that can fail permanently ... Design compensations with generous retry policies."
- "Use idempotency keys for forward Activities"; "Keep compensation payloads small" (the 2 MB payload limit).
- "Temporal's durable execution guarantees that compensations will execute even after Worker failures." Not after termination: terminating a workflow runs none of its code.
- Not a good fit for "single-service transactions that can use a local database transaction".
- Python: a list plus `reversed()`. The page catches `Exception`, which does not include `asyncio.CancelledError`; a Python workflow that must clean up after cancellation catches it separately and shields the cleanup.
**SDK:** plain workflow code: `workflow.execute_activity`, and `asyncio.shield` for cleanup after cancellation. Nothing to add in 1.30.0. Release notes for SDK 1.28 and 1.31 list fixes to shielded cancellation (unverified here), so the 1.34.0 upgrade's replay corpus should include a cancelled delegation and a cancelled team job. Both are in `fixtures/0.3.1-cancel/` (1b.11), and they replay on 1.30.0 and 1.34.0:
- `task_runner_paused_cancel`: the runner's cleanup after its assignee ended cancelled;
- `employee_job_cancel`: the job's shielded `employee.job.failed`.
**OpenCompany today:**
- **`DelegatedTaskWorkflow` is a textbook saga** (`server/services/temporal/agent_workflow.py:535-728`). Forward steps register the task, acquire a permit, begin, start the agent and finish (`:553-623`). On cancellation it cancels the task whether or not `begin` ran, because "Queue/acquire/begin may have completed even when their result lost the cancellation race" (`:647-651`), and it releases the permit in `finally` (`:703-721`). Deviations:
- cleanup uses `DELEGATION_CLEANUP_RETRY`, 10 attempts at 1-30 s (`server/services/temporal/_retry_policies.py:112-116`). It gives up after about 2.5 to 7.5 minutes and logs; the task row stays non-terminal and the permit stays held until the 3-hour age reclaim (`server/core/database.py:4153-4183`);
- the `except Exception` arm calls `finish_delegation` unguarded (`:692-701`), so its failure replaces the original error.
- `AgentWorkflow`'s `_cleanup_cancelled_delegations` cancels every owned task, then releases every permit, logging and continuing per step (`agent_workflow.py:1525-1566`): the right shape, with the same capped policy.
- `EmployeeJobWorkflow` runs a shielded `employee.job.failed` on any exception with `RetryPolicy(maximum_attempts=3)` (`server/services/temporal/employee_job_workflow.py:121-130`). A short database outage leaves the job `running`; Apply then treats it as live work (`server/services/employees/safe_apply.py:14-31`) until the owner picks Stop work.
- `workflow_runs.record_completion` in `NodeInvocationWorkflow` is capped at 3 attempts (`server/services/temporal/node_invocation.py:85-99`).
- **Reset terminates, so none of these compensations runs.** Reset's sweep calls `terminate` (`server/services/deployment/handlers.py:974-1017`; strict at `:2591`). Reset runs database hooks instead (`reset_execution_state`, `server/services/deployment/runtime_state.py:39-56`), implemented for the browser, browser agent, chat reply, chat trigger, context, memory, mobile and approval gate nodes, and for no team, team task, employee job or permit. Those rows stay non-terminal, and the next generation's Apply answers "waiting", because `safe_apply._active` reads them by workflow id (`safe_apply.py:14-31`). RFC-0004 H12 found this; D9 plans cancel, a bounded grace period, then terminate.
- **Start compensates in the API process** (`handlers.py:2122-2186`). On failure it fails the control row, signals the controller to reset, terminates the generation's workflows, cancels the local deployment, broadcasts and re-raises. It is not in reverse order. If `service.fail` loses its compare-and-swap, the rest is skipped. What to undo is found through Visibility (Temporal's search index of executions, `handlers.py:844-900`), which is eventually consistent, although every resource has a known id (the controller is `workflow-control-<workflow_id>-g<generation>`).
- **Reset is idempotent but not resumed.** A latched `resetting` state and strict steps let a repeated Reset continue where it stopped, but boot reconcile leaves `resetting` rows as they are (`_reconcile_control`, `handlers.py:1223-1260`); the owner must press Reset again.
- **Apply is the largest saga** (`safe_apply.py:34-289`): a durable intent with the previous snapshot, a renewed lease, forward steps, a rollback (`:246-279`) and the 10-second sweeper. Undo data is captured before each step. But the rollback is not the reverse of the forward order, and every rollback step re-checks the lease (`:162-167`), so a lost lease aborts the rollback half-way; the sweeper then re-runs the forward path from the top (`:285-289`).
- Adding an MCP connector writes two rows to the same credentials database and removes the first if the second fails (`server/nodes/mcp/_store.py:175-183`). One local transaction would do. An orphaned URL row is harmless, because `exists` reads the sign-in row (`:157-158`).
- An approval decision moves the row, then starts the send; a failed start reverts the row (`server/services/approvals/handlers.py:139-159`). An ambiguous start is harmless: the send claims only at the revision it was started for (`server/services/approvals/execution.py:101-128`).
- `ApprovedToolCallWorkflow` has no `finally` around the send. A cancelled or terminated send leaves the row `sending` until `recover_sends` marks it `unknown` after 30 minutes, and that sweep runs once per process, on the first draft listing (`server/services/approvals/reconcile.py:76-110`, `:131-141`; `server/services/approvals/handlers.py:84`). Reset never reaches this workflow, because it carries no `EventWorkflowId` (`server/services/approvals/execution.py:58-70`).
- Dead code with a recovery hole: `restart_with_latest_graph` (`server/services/deployment/restart.py:72-101`) runs Reset and then Start with no intent row. Apply and Turn on Talk call `apply_saved_changes` instead (`server/services/employees/handlers.py:222-223`, `:232-233`, `:276-277`). Only its test, `resources/builder_documentation.json`, CLAUDE.md's Normal Mode row and `docs-internal/normal_mode.md:839` still mention it.
**Verdict:** Adopted with deviations. `DelegatedTaskWorkflow` follows the page. The service-side sagas are not last in, first out and stop half-way, compensations give up after 3-10 attempts, and Reset terminates, so workflow compensations never run. Phase 1b closed the first three: Start's undo runs last in, first out (1b.22), compensations retry until they land (1b.20), and Reset closes the open work itself (1b.21).
**Recommendation:**
1. **Compensations retry until they land.** Built in 1b.20 (2dcd53f9) without two parts: `record_completion` stays best-effort, and `finish_delegation` gets no guard. The design follows. Make `DELEGATION_CLEANUP_RETRY` unlimited (1 s to 30 s). Give both `employee.job.failed` calls and `workflow_runs.record_completion` an unlimited bookkeeping policy, the one `ApprovedToolCallWorkflow` already uses (`BOOKKEEPING_RETRY`, `approved_tool_call_workflow.py:26`), moved into `_retry_policies.py`. Guard `finish_delegation` in the `except Exception` arm (`agent_workflow.py:692-701`) so its failure is logged and the original error is re-raised.
- Versioning: none. Retry policies are activity options, which replay does not compare (RFC-0004 §8.5); the replay corpus must stay green.
- When: Phase 1b (1b.20). Effort S.
- Risk: a broken compensation retries until it is fixed, so it needs the retry alert (§11.7) and the rewording of RFC-0004's gate AG9 (§21), which today forbids an unlimited policy without `schedule_to_close_timeout` (a total time budget across all attempts). The Saga page says the opposite for compensations, and T4 says reliability wins.
2. **Reset closes what terminated workflows cannot.** Built differently in 1b.21 (a6f9af68): no plugin hook and no permit release. Reset calls `safe_apply.close_open_work`, shared with "Stop work and apply", which dissolves the teams, cancels their open tasks and cancels the open and queued employee requests; `dispatch_job` cancels a job of an ended generation. The design follows: add `reset_execution_state` to `TaskManagerNode` (`server/nodes/tool/task_manager/__init__.py`): it cancels the workflow's open team tasks (`cancel_team_task` is already idempotent and guards terminal states, `server/core/database.py:3845-3860`), dissolves its active teams and releases their subagent permits. Reset also fails the workflow's non-terminal `EmployeeJob` rows with reason `reset`, from `services/employees`, which creates them. With the rows closed, `safe_apply._active` stops reporting live work.
- Versioning: none (database only). When: Phase 1b (1b.21). Effort S-M.
- Risk: a hook that misses a row type; the test lists every non-terminal state.
- T14 makes this essential: a generation whose histories cannot replay is marked "needs Reset", its workflow compensations can never run, and these hooks are its only cleanup.
3. **Start's compensation runs in reverse order, by known ids, each step logged and continued:** cancel the local deployment, signal the controller reset, sweep the generation's workflows, then fail the row (only this generation's, while it still reads `starting`) and broadcast. A lost compare-and-swap on the row no longer skips the rest. The Start runs as a task of its own, so a Start whose socket drops finishes on its own instead of being abandoned halfway. As built (a95120ed, 986b8484), the local cancel and the sweep run only while the row is still this generation's `starting`, because Reset can take a starting row; a clean shutdown cancels a running Start without its undo, leaving `starting` for the boot reconcile.
- Versioning: none (API process). When: Phase 1b (1b.22). Effort S.
- Risk: mistakes appear only on failure paths; the test injects a failure at each step.
- New generations get the rest with RFC-0005 §9.3 (subscriptions compiled in the transaction that creates the `starting` generation, §7.2) and RFC-0005 §15 (Schedules addressed by id).
4. **Resume an interrupted Reset:** boot reconcile drives `resetting` rows through `handle_reset_workflow` at the row's revision instead of leaving them. Built in 1b.23 (9357689a) as one inline call per boot; a failure leaves the row `resetting`. Versioning: none. When: Phase 1b (1b.23). Effort S. Risk: a Reset that fails on every boot stays visible, as it is today.
5. **Run `recover_sends` from the chat watchdog's 30 s round** (`server/services/chat/watchdog.py`) instead of once per process, so a cancelled send always becomes `unknown` within about 30 minutes. Versioning: none. When: Phase 1b, merged into 1b.5. Effort S. It stays for drafts and Retry (RFC-0007 §4.10).
6. **Specify Apply for `event_os_version=1`.** RFC-0005 does not specify Apply: it names it only as an owner action (I2) and a Phase 4 item, and never mentions `replace_graph` or `pending_changes`. With subscriptions and the snapshot in the database, Apply becomes one local transaction (snapshot, subscriptions, revision), the supervisor's hold and release Updates, and Schedule updates by id: a much smaller saga. Following T10, the owner's Apply writes an intent, answers "applying", and the deliverer drives the steps until they are applied.
- Versioning: new workflow types. When: RFC-0005 Phase 4 (amendment, §20). Effort M.
- Risk: two Apply paths until old generations are retired (R_k+1).
7. **Cleanups:** add the MCP connector in one `AuthService` transaction; delete `restart_with_latest_graph` and its test; fix CLAUDE.md's Normal Mode row and `docs-internal/normal_mode.md:839`. Versioning: none. When: later. Effort S.
**Remarks:**
- Do not add `schedule_to_close_timeout` or a workflow timeout to compensations (the page; RFC-0005 §27 keeps workflow timeouts unset). Unlimited compensation retries are bounded in practice by Reset, which under RFC-0004 D9 cancels, waits a grace period and then terminates.
- The shielded finalizers (`employee_job_workflow.py:124`; `node_invocation.py:84`, `:102`) fit the cancellation guide's "execute any cleanup logic" after a cancel.
- Under T10 and T12 the approval decision stops being a saga: the decision is an intent the deliverer sends, and a decision taken while the employee is stopped is recorded at once while its send waits for Resume (T12, 1b.5). The revert-on-failed-start branch then goes away (§9.6; RFC-0005 §17.5).
- Under T10, Start and Apply run from intent rows the way a hire's activation already does; the compensation runs inside the deliverer, which retries until the row reaches a terminal state.
- Compensation payloads are small: ids and short lifecycle dicts. Apply keeps its snapshots in the database.
**Sketch** (recommendation 2; helper names other than `cancel_team_task` and `update_team_status` are illustrative):
```python
# server/nodes/tool/task_manager/__init__.py
class TaskManagerNode(ToolNode):
type = "taskManager"
@classmethod
async def reset_execution_state(cls, *, node_id, workflow_id, execution_id,
generation, graph, database) -> dict:
"""Reset terminates the generation's workflows, so DelegatedTaskWorkflow
and AgentWorkflow never run their own cancel-and-release. Close what
they would have closed. Idempotent: a second Reset finds nothing open."""
cancelled = released = 0
for team in await database.list_active_teams(workflow_id): # status == "active"
for task in await database.get_team_tasks(team["id"]):
if await database.cancel_team_task(team["id"], task["id"], reason="reset"):
cancelled += 1 # terminal rows are kept
released += await database.release_subagent_permits(
root_execution_id=team["root_execution_id"])
await database.update_team_status(team["id"], "dissolved")
return {"reset": bool(cancelled or released),
"cancelled_tasks": cancelled, "released_permits": released}
```
### 10.3 Early Return
**Docs:** https://docs.temporal.io/design-patterns/early-return. Answer the caller as soon as a fast initialization succeeds while the slow phase continues in the background, using Update-with-Start. Rules:
- The Update handler only waits (`await workflow.wait_condition(lambda: self.init_done)`); `run()` does the initialization, ideally as a local activity "under 5 seconds", and then the slow phase.
- "A `WorkflowIdConflictPolicy` must be specified." Use `FAIL` for one workflow per request; "Use `USE_EXISTING` only for lazy initialization patterns."
- "Update-with-Start is **not atomic**": the workflow may start without the Update. "Set a timeout on the Update result"; "Provide a unique Update ID ... so retried requests attach to the same Update"; "The default `maxInFlightUpdates` is 10 per Workflow"; "Avoid Workflow timeouts".
- Pitfalls: "Blocking too long in the Update handler"; using it for fire-and-forget work ("use Signals").
**SDK:** `Client.execute_update_with_start_workflow` and `start_update_with_start_workflow`, `WithStartWorkflowOperation(..., id_conflict_policy=...)` and `rpc_timeout=` all exist in 1.30.0 (`temporalio/client/_client.py:930-1043`) and keep the same parameters in 1.34.0. `rpc_timeout` bounds each request, not the call: the client repeats its request until the Update is accepted, then polls for the outcome without a deadline (1.30.0 `temporalio/client/_impl.py:973-1003`). Only `asyncio.timeout` around the whole call bounds the wait (§7.3).
**OpenCompany today:**
- **Workspace tasks use the page's lazy-initialization form:** Update-with-Start on a long-lived controller per workflow, with `USE_EXISTING` (`server/services/node_invocations.py:44-74`).
- Its state is set in `@workflow.init` because "Update-With-Start can invoke a handler before run()" (`server/services/temporal/workspace_tasks_workflow.py:39-53`).
- Admission is a projection activity plus the child's start acknowledgement (`:97-125`); a duplicate is caught by run id and fingerprint (`:89-94`); at most 32 tasks run at once (`:20`, `:95-96`).
- The Update id has a random suffix on purpose, so a refused submission is not cached (`node_invocations.py:249-254`). The controller's own dedup gives the idempotency the page wants from a stable Update id.
- Deviations:
- **no deadline:** neither `_controller_update` nor the HTTP routes set one (`server/routers/browser_tasks.py:40-50`; `server/nodes/mobile/_router.py:251-255`), so with no worker polling, a submission hangs;
- the handler runs an activity and a child start under a lock instead of only waiting, so concurrent submissions queue as in-flight Updates (limit 10), and only `WorkflowUpdateFailedError` is mapped to a user message (`node_invocations.py:60-74`);
- refusals are raised inside the handler, not in a validator, so a refused submission still costs history (RFC-0004 L3);
- the controller rolls over only when idle (`workspace_tasks_workflow.py:195-206`).
- **Chat, hire and team jobs answer after a database commit** and start their Temporal work in the background (`server/services/chat/handlers.py:233-341`; `server/services/employees/hire.py:292-297`; `server/services/employees/jobs.py:208-252`). This is a database-first early return. RFC-0005 keeps it and rejects ingress straight to Temporal (RFC-0005 §9.7), and T10 extends it to owner actions. It is the right choice for E4 and I10: an Update-with-Start would refuse or hang whenever Temporal or a worker is down.
- **Start is synchronous:** the controller starts empty and triggers arrive as Signals (one-way messages to a running workflow) afterwards (`server/services/deployment/manager.py:971`), so a crash in between leaves an empty controller that boot converges (`handlers.py:1708-1764`). RFC-0005 §9.3 removes the gap.
**Verdict:** Adopted with deviations. The one Update-with-Start caller follows the page's lazy-initialization form but never bounds its wait; the database-first answers elsewhere are the better fit for OpenCompany (E4, I10, T10).
**Recommendation:**
1. **Bound the Workspace submission.** §7.3 owns the fix (1b.15): every client Update call runs under `asyncio.timeout`, the Workspace Update-with-Start included (about 15 s). On this route, an expired deadline, or the server's refusal when 10 Updates are already in flight, answers "submitted, still pending: check the task list". The stable `submission_id` makes a resubmission safe, because the controller dedups by run id and fingerprint.
- Versioning: none (client). When: Phase 1b (1b.15). Effort S.
- Risk: a "pending" answer for a submission that later succeeds; the task list shows it.
2. **With T10's deliverer, put submissions behind it:** write the `workspace_tasks` record as the intent, answer "applying", and let the deliverer send the Update-with-Start. It keeps a random Update id per attempt, because the submit checks can fail after acceptance (§7.3 R3); the controller's dedup by run id and fingerprint makes a resend safe. A refusal (queue full, resetting) then ends the intent visibly.
- Versioning: none (client). When: with T10. Effort M.
**Remarks:**
- A Workspace controller that never idles cannot roll over: its children use `REQUEST_CANCEL` (`:114-121`) so that Reset can cancel them and wait, and rollover waits until no task is active. It grows by about 10 events per submission (est.), and `_OPERATIONS_PER_RUN = 256` and the 10,000-event soft cap apply only when it is idle. Add a fixture with one long task and many submissions (P3). Abandoning children, as Sliding Window does (§12.4), would give up Reset's barrier.
- Moving the controller's refusals into an Update validator (RFC-0004 L3) belongs to §7.3.
- Running `workspace_tasks.admit_record` as a local activity, as the page advises for the init phase, is not worth it: RFC-0004 D17 lists local activities as not adopted.
- RFC-0005's relay uses Signal-With-Start (RFC-0005 §9.4), the page's fire-and-forget case; the ApprovalWorkflow `decide` Update (RFC-0005 §17.5) is request-response on a running workflow, not Early Return.
## 11. Error handling and retries
### 11.1 Error Handling & Retry Patterns (category overview)
**Docs:** https://docs.temporal.io/design-patterns/error-handling-patterns. A decision tree for a failing activity:
- each attempt costs money or quota: Fixed Count of Retries;
- the error will never succeed by itself: Resumable Activity if a human can correct it, otherwise Non-Retryable Errors;
- the downstream has a predictable unavailability window: Delayed Retry;
- it "Must resolve within a time budget": Fixed Wall-Time Retries (`ScheduleToCloseTimeout`);
- recover fast, then wait patiently: Fast/Slow Retries;
- "For any long-running retry scenario, add **Retry Alerting via Metrics**."
- Defaults: "Temporal's default `RetryPolicy` retries Activities indefinitely with exponential backoff": `MaximumAttempts` 0 (unlimited), `InitialInterval` 1 s, `BackoffCoefficient` 2.0, `MaximumInterval` 100 times the initial interval, no non-retryable types. "`ScheduleToCloseTimeout` is set on the Activity call options, not in `RetryPolicy`."
- The tree calls Delayed Retry "Fixed interval backoff"; the Delayed Retry page itself is about the per-error `next_retry_delay`. Follow the page.
**SDK:** `RetryPolicy` is identical in 1.30.0 and 1.34.0 (AST comparison of `temporalio/common.py`). `workflow.execute_activity` takes the same keyword arguments, except that 1.34.0 adds the experimental `event_groups`.
**OpenCompany today:**
- **Where Temporal sees an error.**
- Plugin code returns every exception as a result envelope: `PermissionError`, `NodeUserError`, `NodeWaitInterrupted` and any other `Exception` (`server/services/plugin/base.py:692-768`). `NodeExecutor` turns anything else into a failure dict, including `asyncio.CancelledError` for every class except `workspace_task` ones (`server/services/node_executor.py:231-244`).
- So for graph nodes and agent tools, Temporal retries only a lost worker (after the 2-minute heartbeat timeout, the longest a running activity may go without reporting that it is alive), an expired start-to-close timeout, `NodeWaitInterrupted` (re-raised as a retryable error, `base.py:1355-1366`) and a crash in the activity wrapper (`:1398-1410`). A plugin's `retry_policy` and its `non_retryable_error_types` apply to nothing else; since RFC-0004 Q9 was answered (2026-11-04), they also govern the retry of a transient failure of a call that is safe to repeat and is not a send (§11.4 item 3).
- Framework activities raise typed `ApplicationError`s: the LLM step (`server/services/temporal/agent_activities.py:246-303`), Pi (`server/services/temporal/pi_activities.py:27-47`), the controller's Update validators and compaction.
- `MachinaWorkflow` and `AgentWorkflow` turn an activity or child failure into `{"success": False, ...}`. From the exception they read only `requires_user_action` (behind a patch in `MachinaWorkflow`), never `retry_state` or the timeout type (`server/services/temporal/workflow.py:1080-1133`; `agent_workflow.py:1258-1340`). Runs end Completed with `success: False` (`workflow.py:913-930`), although the owner chose Temporal Failed (RFC-0004 Q4).
- **Retry and timeout inventory:**
| Activity | Attempt timeout | Retry policy | Total budget |
|---|---|---|---|
| graph node `node.{type}.v{n}` (`workflow.py:837-869`) | 24 h, heartbeat 2 min | the plugin's, default 3 attempts at 1-60 s (`server/services/plugin/scaling.py:58`) | none |
| agent tool, frozen path (`agent_workflow.py:117-122`, `:143-155`) | plugin value | frozen plugin policy | none |
| agent tool, legacy path: manual canvas runs and generations without execution control (`agent_workflow.py:113-140`) | 10 min | none: the server default, unlimited | none |
| `agent.execute_llm_step` (`agent_workflow.py:3183-3188`) | 10 min, heartbeat 1 min | `LLM_STEP_RETRY`: unlimited, 5 s to 5 min (`_retry_policies.py:94-100`) | none |
| `agent.compact_context`, a paid LLM call (`agent_workflow.py:2829-2835`) | 5 min, no heartbeat (the step's attempt timeout and heartbeat since 1b.27) | 3 attempts (`LLM_STEP_RETRY` since 1b.27) | none |
| `agent.acquire_subagent_permit` (`agent_workflow.py:569-578`) | 1 h, heartbeat 10 s | `PERMIT_WAIT_RETRY`: unlimited | none |
| approved send (`approved_tool_call_workflow.py:59-65`) | 10 min | `SEND_ONCE`: 1 attempt | none |
| approval bookkeeping; `workspace_tasks.update_record` | 30 s | unlimited | none |
| `employee.job.deliver`, `employee.job.resolve_delivery` (`employee_job_workflow.py:132-139`, `:155-157`) | 31 days | 3 attempts (unlimited `REVIEWED_DELIVERY_RETRY` since 1b.9) | none |
| controller queue read and spill (`workflow_control_workflow.py:402-406`, `:629`) | 60 s | `DEFAULT_ACTIVITY_RETRY`: 3 attempts (unlimited `QUEUE_PAGING_RETRY` since 1b.16) | none |
| `store_node_output_activity`; graph load (`workflow.py:618-626`; `trigger_listener_workflow.py:360-364`) | 10 s | none: unlimited | none |
Outside any policy, the controller puts a failed spawn back at the head of its queue and sleeps 5 s, without limit (`workflow_control_workflow.py:454-480`).
- **No metric leaves the process:** both Temporal runtimes configure logging only (`server/services/temporal/worker.py:91-102`; `server/services/temporal/client.py:76-83`).
- The lint against inline policies covers five files (`server/tests/test_retry_policies.py:89-95`); five other workflow modules build policies inline: `approved_tool_call_workflow.py`, `employee_job