AGENTS.md
Rama main · réplica de lectura del repositorio original.
# AGENTS.md
This file provides guidance to coding agents working with code in this repository.
Claude Code compatibility is kept through `CLAUDE.md`, which imports this file.
## Build and Test Commands
This project uses [uv](https://docs.astral.sh/uv/) for Python project management.
### Building
```bash
uv run maturin develop # Build Rust code and install Python package (required after Rust changes)
```
### Testing
```bash
cargo test # Run Rust tests
uv run mypy # Type check Python code
uv run pytest python/ # Run Python tests (use after both Rust and Python changes)
```
### Code Formatting and Linting
Python linting and formatting go through `prek`, which runs the `ruff-check` and
`ruff-format` hooks from `.pre-commit-config.yaml` with the exact ruff version
and file set CI uses. A `ruff` from `uv run`, `uvx`, or Homebrew is usually a
different version than the hook pins, so run the hooks rather than `ruff` itself.
```bash
uv tool install prek # one-time; some hooks also need `protoc` (brew install protobuf)
prek run ruff-check ruff-format --all-files # Lint (applying safe fixes), then format Python code (same as CI)
cargo fmt # Format Rust code
./dev/run_cargo_clippy.sh # Lint Rust code other than the Rust SDK and its CLI (same as CI)
./dev/run_cargo_clippy_sdk.sh # Lint the Rust SDK and its CLI, `rust/cli` (same as the daily CI)
```
### Pre-submission Checks
CI enforces the pre-commit hook suite from `.pre-commit-config.yaml`, which covers
more than the commands above — notably regenerating the CLI docs
(`docs/src/content/docs/cli.mdx`) whenever `cli.py` changes. The Rust SDK hooks,
described below, are the exception. Before pushing, run the hooks whose inputs
your branch changed, as PR CI does:
```bash
prek run --from-ref origin/main --to-ref HEAD
```
`prek run --all-files` runs every hook on every file, the Rust SDK hooks
(`cargo-test-sdk`, `cargo-clippy-sdk`, `cargo-test-rust-externs`) included. The
Rust SDK isn't launched yet, so those hooks run only for a change under
`rust/sdk/`, `rust/cli/`, `examples/rust/` or `benchmarks/*/rust/`, not for a change to the
code the SDK builds on. CI runs them on no pull request and in no merge group,
even for a change to the SDK, because they are slow, so the local run is the only
check before such a change lands. The `Rust SDK Daily` workflow runs them on
`main` every day and files an issue when they fail;
`gh workflow run rust_sdk_daily.yml --ref <branch>` runs them on a branch.
### Pull Requests and CI
Open a pull request as a draft and mark it ready once the checks above pass:
drafts run no CI. When `main` has a merge queue, a ready pull request runs only
the fast checks, and the queue runs the full suite on the commit that lands.
For a full run on a branch before then, use `gh workflow run CI.yml --ref <branch>`.
The CI tiers, the merge queue and stacked pull requests are described in
[CI.md](https://github.com/cocoindex-io/.github/blob/main/CI.md).
A pull request to a release branch (`release/X.Y`, see [Versioning](#versioning))
has no merge queue: its own run is the full tier, and it is squash-merged once
`ci-gate` passes (`gh pr merge --squash`). Such a pull request is a backport of
a change that already landed on `main`; `dev/agent-skills/backport/SKILL.md`
says what qualifies and how to land one.
### Workflow Summary
| Change Type | Commands to Run |
|-------------|-----------------|
| Rust code only | `uv run maturin develop && cargo test` |
| Python code only | `uv run mypy && uv run pytest python/` |
| Both Rust and Python | Run all commands from both categories above |
| Rust SDK, Rust CLI (`rust/cli`), Rust examples or Rust benchmarks | `prek run cargo-test-sdk cargo-clippy-sdk update-rust-extern-locks cargo-test-rust-externs --all-files` |
| Python lint and format | `prek run ruff-check ruff-format --all-files` |
| Before pushing | `prek run --from-ref origin/main --to-ref HEAD` |
After changing a core engine API (`rust/core`), run `cargo check --workspace --all-targets` to catch call sites in every workspace crate — `rust/py`, the Rust SDK under `rust/sdk/` and the Rust CLI under `rust/cli/` call core directly, and the SDK and its CLI are easy to miss when grepping. The `examples/rust/*` and `benchmarks/*/rust` crates are standalone workspaces that consume the SDK; the `prek` suite's `cargo check (rust examples + benchmarks)` hook covers those. That hook and the other Rust SDK hooks don't run for a change to `rust/core` alone, so this check is the one that catches a broken SDK before the daily run does.
## Code Structure
```
cocoindex/
├── rust/ # Rust crates (workspace)
│ ├── core/ # Core engine crate
│ │ └── src/
│ │ ├── engine/ # Core engine
│ │ ├── state/ # States of the core engine
│ │ └── inspect/ # Database inspection utilities
│ ├── py/ # Python bindings (PyO3); calls core engine APIs
│ ├── py_utils/ # Python-Rust utility helpers (error, convert, future)
│ ├── utils/ # General utilities: error, batching, fingerprint, etc.
│ ├── code_ast/ # Tree-sitter foundation: language registry (links all grammars), CodeSource, positions
│ ├── ops_text/ # Text processing operations (splitter, language detection)
│ ├── code_match/ # Structural code pattern matching over tree-sitter ASTs
│ ├── sdk/
│ │ ├── cocoindex/ # Rust SDK (`cocoindex` crate); calls core engine APIs directly, like py/
│ │ └── cocoindex_macros/ # Proc macros for the Rust SDK (`#[function]`, `#[app]`, `#[lifespan]`, `#[main]`)
│ └── cli/ # `cocoindex` CLI for Rust projects: builds the user's crate, drives its binary (spec: dev/plans/rust-cli.md)
│
├── python/
│ ├── cocoindex/ # Python package
│ │ ├── __init__.py # Package entry point
│ │ ├── cli.py # CLI commands
│ │ ├── _internal/ # Internal implementation for the core engine
│ │ │ ├── api.py # Public API: mount, use_mount, mount_each, map, mount_target, App, fn, start/stop
│ │ │ ├── app.py # App base implementation
│ │ │ ├── context_keys.py # ContextKey and ContextProvider
│ │ │ ├── environment.py # Environment and lifespan handling
│ │ │ ├── function.py # @coco.fn decorator implementation
│ │ │ ├── component_ctx.py # ComponentContext and component_subpath
│ │ │ ├── target_state.py # Target state implementation
│ │ │ └── core.pyi # Type stubs for the Rust extension module (update when PyO3 APIs change)
│ │ ├── connectors/ # External system connectors (localfs, postgres, qdrant, lancedb, google_drive)
│ │ ├── connectorkits/ # Connector building utilities
│ │ ├── resources/ # Abstractions: file.py (FileLike), chunk.py (Chunk), schema.py
│ │ └── ops/ # Operations: text.py (RecursiveSplitter), sentence_transformers.py
│ └── tests/ # Python tests
│
├── examples/ # Example applications
├── docs/ # Documentation
└── dev/ # Development utilities
```
## Key Concepts
### Mental model: declarative data pipelines
CocoIndex uses a **declarative** programming model — you specify *what* your output should look like (target states), not *how* to incrementally update it. The engine handles change detection and applies minimal updates automatically.
Think of it like:
* **React**: declare UI as function of state → React re-renders what changed
* **Spreadsheets**: declare formulas → cells recompute when inputs change
* **CocoIndex**: declare target states as function of source → engine syncs what changed
### Core concepts
**App** — The top-level runnable unit. Bundles a main function with its arguments. When you call `app.update()`, the main function runs as the root processing component.
**Processing Component** — The unit of execution that owns a set of target states. Created by `mount()` or `use_mount()` at a specific component path. When a component finishes, its target states sync atomically to external systems.
**Component Path** — Stable identifier for a processing component across runs. Created via `coco.component_subpath("process", filename)`. CocoIndex uses component paths to:
* Match components to their previous runs for change detection
* Determine ownership of target states (if a path disappears, its target states are cleaned up)
**Target State** — What you want to exist in an external system (a file, a database row, a table). You *declare* target states; CocoIndex keeps them in sync — creating, updating, or removing as needed.
**Target** — The API object used to declare target states (e.g., `DirTarget`, `TableTarget`). Targets can be nested: a container target state (directory/table) provides a Target for declaring child target states (files/rows).
**Function** — A Python function decorated with `@coco.fn`. Use `memo=True` to enable memoization (skip execution when inputs and code are unchanged).
**Context** — React-style provider mechanism for sharing resources. Define keys with `ContextKey[T]`, provide values in lifespan via `builder.provide()`, use in functions via `coco.use_context(key)`.
### Key APIs
```python
# Mounting processing components (subpath auto-derived from fn.__name__)
await coco.mount(fn, *args, **kw) # child runs independently
result = await coco.use_mount(fn, *args, **kw) # returns value directly
# Explicit subpath (for multi-part paths or multiple mounts of same function)
await coco.mount(coco.component_subpath("process", filename), fn, *args, **kw)
result = await coco.use_mount(coco.component_subpath("name"), fn, *args, **kw)
# Component subpath composition
subpath = coco.component_subpath("process", filename) # multiple parts
subpath = coco.component_subpath("a") / "b" / "c" # chaining with /
# Using component_subpath as context manager (applies to all nested mount calls)
with coco.component_subpath("process"):
for f in files:
await coco.mount(coco.component_subpath(str(f.relative_path)), process_file, f, target)
# Declaring target states (typically via Target methods)
dir_target.declare_file(filename=name, content=data)
table_target.declare_row(row=MyRow(...))
# Using context values
db = coco.use_context(PG_DB) # retrieve value provided in lifespan
# Explicit context management (for ThreadPoolExecutor)
ctx = coco.get_component_context()
with ctx.attach():
# coco APIs work correctly in this thread
coco.mount(...)
```
**Mount handles:**
* `mount()` → `ComponentMountHandle`: call `await handle.ready()` to wait until target states are synced
* `use_mount()` → returns the result value directly (awaitable)
### How syncing works
When a processing component finishes, CocoIndex compares its declared target states with those from the previous run at the same component path:
* New target states → create (insert row, create file)
* Changed target states → update
* Missing target states → delete
Changes are applied atomically per component. If a source item is deleted (path no longer mounted), all its target states are cleaned up automatically.
### Example
```python
import pathlib
import cocoindex as coco
from cocoindex.connectors import localfs
from cocoindex.resources.file import FileLike, PatternFilePathMatcher
@coco.fn(memo=True)
async def process_file(file: FileLike, target: localfs.DirTarget) -> None:
html = _markdown_it.render(await file.read_text())
outname = "__".join(file.file_path.path.parts) + ".html"
target.declare_file(filename=outname, content=html)
@coco.fn
async def app_main(sourcedir: pathlib.Path, outdir: pathlib.Path) -> None:
target = await coco.use_mount(localfs.declare_dir_target, outdir)
files = localfs.walk_dir(
sourcedir,
recursive=True,
path_matcher=PatternFilePathMatcher(included_patterns=["**/*.md"]),
)
await coco.mount_each(process_file, files.items(), target)
app = coco.App(
coco.AppConfig(name="FilesTransform"),
app_main,
sourcedir=pathlib.Path("./docs"),
outdir=pathlib.Path("./out"),
)
app.update_blocking(report_to_stdout=True)
```
## Code Conventions
### Internal vs External Modules
We distinguish between **internal modules** (under packages with `_` prefix, e.g. `_internal.*` or `connectors.*._source`) and **external modules** (which users can directly import).
**External modules** (user-facing, e.g. `cocoindex/ops/sentence_transformers.py`):
* Be strict about not leaking implementation details
* Use `__all__` to explicitly list public exports
* Prefix ALL non-public symbols with `_`, including:
* Standard library imports: `import threading as _threading`, `import typing as _typing`
* Third-party imports: `import numpy as _np`, `from numpy.typing import NDArray as _NDArray`
* Internal package imports: `from cocoindex.resources import schema as _schema`
* Exception: `TYPE_CHECKING` imports for type hints don't need prefixing
**Internal modules** (e.g. `cocoindex/_internal/component_ctx.py`, `**/_target.py`):
* Less strict since users shouldn't import these directly
* Standard library and internal imports don't need underscore prefix
* Only prefix symbols that are truly private to the module itself (e.g. `_context_var` for a module-private ContextVar)
### Minimize API surface until deemed necessary
When adding a new public API (function, class, kwarg, configuration option), prefer the smallest surface that solves the concrete need in front of you. Do not pre-expose tunables, hooks, or alternatives "in case someone needs them." Adding a kwarg later is straightforward and backwards-compatible; removing or renaming one is disruptive. If a knob is currently a hardcoded constant inside the implementation, leave it there until a real use case demands it — at which point promoting it to a kwarg is mechanical. This applies equally to optional parameters, callback hooks, parser/transformer plug-points, and configuration keys.
### General principles (also covered by `/review-changes`)
- **Top-level imports.** Defer to in-function only for a real circular dependency or a heavy import that isn't always needed.
- **Specific types over `Any`.** Use concrete types — including concrete types from third-party libraries. `Any` only when the type is truly generic and no downstream code needs to downcast it.
- **Validate and exchange early.** When a value enters as a weaker form (`str`, `Any`, raw identifier), convert to the strong type at the earliest point. Don't propagate the weak form. Concrete example in this codebase: connector handlers receive a `ContextKey` string as part of a target-state key. The parent handler resolves the key once (when constructing the child handler) and passes the typed connection directly — the child stores `_pool: asyncpg.Pool`, not `_db_key: str`.
- **`NamedTuple`/small dataclass for multi-value returns.** Access fields by name (`result.can_reuse`) at call sites.
- **Exceptions for exceptional situations only.** Reserve exceptions (Python) and errors (Rust) for truly unexpected failures — not for end-of-iteration, "not found", or other expected state transitions. Use explicit return values (sentinel, enum variant, `None`) for those.
- **Single source of truth, delete dead code, honest names.** When the same value/logic appears in multiple places, consolidate. When changes make code unreachable, delete it (and its tests, and any dead config knobs). Function and field names should describe what the implementation does today.
- **One HTTP client: `httpx`.** Python code that speaks HTTP uses `httpx` (`httpx.AsyncClient` in async code): it is what the rest of the stack already pulls in, and `httpx.MockTransport` gives offline tests without monkeypatching. Use another client only when the protocol needs something `httpx` lacks, and say so in a comment at the import. The one case today is the Doris connector: Stream Load depends on `Expect: 100-continue` (the FE answers 307 before reading the body), which `aiohttp` implements and `httpx` does not.
- Rust code follows [dev/rust-guidelines.md](dev/rust-guidelines.md).
### Testing Guidelines
We prefer end-to-end tests on user-facing APIs, over unit tests on smaller internal functions. With this said, there're cases where unit tests are necessary, e.g. for internal logic with various situations and edge cases, in which case it's usually easier to cover various scenarios with unit tests.
#### Test Environment Setup
Use `common.create_test_env(__file__)` to create a CocoIndex `Environment` for tests. It derives a unique `db_path` from the test file path and picks up the current event loop automatically.
* **Sync tests** (module-level creation): Call at module level — the Environment creates a background loop for async callbacks.
```python
coco_env = common.create_test_env(__file__)
```
* **Async tests with async resources** (e.g., asyncpg pools): Call inside an async fixture so the Environment binds to the test's running event loop (same loop the pool is on). Use the `suffix` parameter when each test needs its own Environment:
```python
@pytest_asyncio.fixture
async def pg_env(request: pytest.FixtureRequest) -> Any:
pool = await asyncpg.create_pool(dsn)
coco_env = common.create_test_env(__file__, suffix=request.node.name)
coco_env.context_provider.provide(DB_KEY, pool)
yield pool, coco_env
await pool.close()
```
The `suffix` ensures each test gets a unique `db_path`, avoiding "environment already open" errors.
#### Testcontainers for Database Tests
Use `testcontainers[postgres]` (in the `build-test` dependency group) to spin up real database instances automatically — no manual setup or environment variables needed. Use a module-scoped sync fixture for the container and a function-scoped async fixture for per-test resources:
```python
_PG_IMAGE = "postgres:16-alpine"
pytestmark = [pytest.mark.requires_docker(_PG_IMAGE)]
@pytest.fixture(scope="module")
def pg_dsn() -> Any:
with PostgresContainer(_PG_IMAGE) as pg:
dsn = pg.get_connection_url().replace("postgresql+psycopg2://", "postgresql://")
yield dsn
@pytest_asyncio.fixture
async def pool(pg_dsn: str) -> Any:
p = await asyncpg.create_pool(pg_dsn)
yield p
await p.close()
```
List every image a test's containers use in its `requires_docker(...)` marker. `python/tests/conftest.py` pulls missing images after collection, before the first test runs; otherwise testcontainers pulls inside the fixture, and a cold-cache download counts against the first test's `pytest-timeout`.
### LMDB write paths
- All LMDB writes must go through `AppStore::run_txn`, which runs persisting bodies on the single-writer batcher (`Storage::run_txn`, private to `state_store`) and a preview's bodies against a read snapshot.
- Do not open a heed write txn directly or wrap the env in a separate mutex/semaphore — bypassing the batcher loses fsync coalescing and regresses concurrent-submit throughput by 10-100×.
- LMDB has no savepoints. If a sub-operation needs to "abort" without failing its caller, handle it at the body level (e.g. return a sentinel result without writing). A body that returns `Err` fails only its own caller: the batcher drops its writes by aborting the txn and re-running the bodies ahead of it, so every body must be safe to re-run.
- A preview writes nothing to the state store, and write paths do not check for a preview to achieve that: a preview run's `ComponentProcessorContext::app_store()` (inherited by every context of the operation, live components included) is an `AppStore::discarding_writes()` handle. Its bodies run one at a time, as a batch's do, on a blocking thread against a read snapshot plus an in-memory overlay of the run's own writes to its app's database instead of the write batcher, and a preview runs its commit into that handle too, so the run reads its writes back and plans exactly as an update would, and nothing reaches LMDB. The overlay keeps no function-memo value a preview writes (a later run in the same preview reads the memo the store holds, if any, and recomputes otherwise); it does keep tracking records, owner and existence rows, component memos and user states. A preview reserves ids in a range of its own, from `FIRST_ID_OF_A_RUN_THAT_PERSISTS_NOTHING`, so they never meet a persisting run's. Reach the store through that context handle; `AppContext::persisting_app_store()` (crate-private) is for deriving a root run's handle and for inspection. The one deliberate exception is `AppContext::next_id`, whose sequencer owns the app's persisting store and reserves user ids in a preview too (ids are opaque, and a gap costs nothing).
- An LMDB write txn must begin and end (commit or abort) on the same OS thread: the writer lock is thread-owned and LMDB ignores a failed release, so a txn that migrates between runtime workers blocks every later writer for good. heed marks `RwTxn` as `Send` regardless, so the compiler won't catch it. The batcher behind `AppStore::run_txn` runs each batch on one blocking thread for this reason; never hold a write txn across an `.await` on the multi-thread runtime.
### Sync vs Async
The Rust core (`rust/core`, `rust/utils`) uses **async-first** design with Tokio. The `rust/py` crate bridges Rust async to Python, offering both sync and async APIs:
* Rust core exposes async functions
* `rust/py` provides sync wrappers that use `block_on()` to call async Rust from sync Python
* Python's `cocoindex` API is **async-first**: mount APIs (`mount`, `use_mount`, `mount_each`, `map`) are all `async def`; `App.update()`/`App.drop()` are async; sync entry points (`App.update_blocking()`, `App.drop_blocking()`, `start_blocking()`, `stop_blocking()`) are available for CLI and blocking contexts
When adding new functionality that involves I/O or concurrency:
* Implement async in Rust
* Bridge to Python via `rust/py`, providing both sync and async variants if needed
## Versioning
The current codebase is for CocoIndex v1, which is a fundamental redesign from CocoIndex v0. CocoIndex v1 lives on the `main` branch; the legacy v0 code is preserved on the `v0` branch.
Releases are tagged `v<version>` and follow semver:
- A **minor** release (`1.2.0`) is cut from `main` and carries everything landed since the previous minor. Once it's published, its release branch, `release/1.2`, is created from its tag; a major release (`2.0.0`, `release/2.0`) does the same, and a prerelease (`1.2.0-rc.1`) creates none.
- A **patch** release (`1.1.1`) is cut from the minor's release branch, `release/1.1`. The branch starts at the minor's tag (`v1.1.0`) and takes only backports from `main`: fixes, dependency bumps that fix an advisory in what the wheel ships, and docs changes that describe released behavior. A fix makes released behavior match its documentation or intent without adding a way to call CocoIndex; a new API, parameter, connector or option is never backported, experimental or not. Only the newest minor's branch takes backports. `dev/agent-skills/backport/SKILL.md` lands a backport; `dev/agent-skills/create-release/SKILL.md` cuts the release.
- **Experimental** APIs, labelled so in their docs (such as `coco.DataDep`), are outside the compatibility promise: a later release, including a minor one, may change or remove them.
The docs site shows the docs of the latest release: a release deploys the docs from its tagged commit when GitHub lists it as the repository's "Latest" release. Docs for an unreleased feature stay on `main` until the minor that ships it; a docs change that describes released behavior is backported like a fix. The on-demand docs deploy (`docs_release.yml`) runs from the latest release's branch, not from `main`.
## Docs diagrams
All diagrams embedded in docs pages are built as inline Astro components under `docs/src/components/diagrams/`, not exported SVG files. See [docs/src/components/diagrams/README.md](docs/src/components/diagrams/README.md) for the palette, primitives, shared CSS classes, animation conventions, and step-by-step authoring guidelines. Follow that guide for any new or updated diagram.
## Agent Tooling
### Semantic code search over this repo (`cocoindex-code`)
To find where a concept lives, prefer semantic search over blind `grep`. This repo keeps a local
`cocoindex-code` index; query it from any agent's shell with the `ccc` CLI:
```bash
ccc search "where is X handled" # semantic search; add --refresh to reindex first
```
If `ccc search` reports no index, set it up once:
```bash
command -v ccc >/dev/null || uv tool install 'cocoindex-code[full]' # per machine; local embeddings, no API key
ccc init -f && ccc index # repo-scoped index (first build ~1-2 min)
```
Everything is local — SQLite index, local embeddings, no keys — and a background daemon keeps it
fresh. On a brand-new machine `ccc init` asks once for an embedding provider; choose the local
default. The `-f` matters: if a parent directory is also a `ccc` project, plain `ccc init` refuses
and search silently uses the parent's index, so run `ccc status` to confirm the root is this repo.
We register no MCP server on purpose — an `ccc mcp` mode exists, but the CLI costs no context until
invoked and behaves the same across agents. Source:
[github.com/cocoindex-io/cocoindex-code](https://github.com/cocoindex-io/cocoindex-code).
## Agent Playbooks
Reusable task-specific agent guidance lives under `dev/agent-skills/`. These playbooks use a `SKILL.md` format so Claude Code can load them directly from `.claude/skills/`, but the content is intended to be readable by any coding agent.
- For docs diagrams, read `dev/agent-skills/cocoindex-diagrams/SKILL.md`.
- For new target connectors, read `dev/agent-skills/target-connector/SKILL.md`.
- For upgrading example package versions, read `dev/agent-skills/upgrade-examples/SKILL.md`.
- For cutting a release (next version, release notes, GitHub release, PyPI publish), read `dev/agent-skills/create-release/SKILL.md`.
- For backporting a merged pull request to a release branch, for the next patch release, read `dev/agent-skills/backport/SKILL.md`.
Agent Skills-compatible clients, including Codex, discover the checked-in symlinks under `.agents/skills/`. The public CocoIndex skill is exposed there as `cocoindex`, and contributor playbooks are exposed there from `dev/agent-skills/`.
Claude Code compatibility remains under `.claude/`: hooks/settings live there, and `.claude/skills/` mirrors the contributor playbooks with symlinks. Do not duplicate skill content under `.agents/skills/` or `.claude/skills/`; edit the source directories (`skills/cocoindex/` for the public skill, `dev/agent-skills/` for contributor playbooks) and keep dot-directory entries as symlinks.
Portable agent checks live under `dev/agent-checks/`. Claude Code hooks in `.claude/hooks/` are thin adapters that call these scripts when relevant files changed.