ARCHITECTURE.md
Rama main · réplica de lectura del repositorio original.
# palaia Architecture
## Overview
palaia is a local, crash-safe memory and knowledge system for single- and multi-agent AI setups. It integrates natively with OpenClaw as a memory plugin (ContextEngine) and provides a standalone CLI for direct use.
**Key principles:**
- Flat files for entry content (human-readable, debuggable)
- Pluggable storage backends (SQLite default, PostgreSQL for scale)
- Provider-chain patterns for embeddings and storage
- Zero hard dependencies for core functionality
## Core Design Decisions
| ADR | Topic | Status |
|-----|-------|--------|
| [001](docs/adr/001-semantic-search-tiered.md) | Tiered Semantic Search | Accepted |
| [002](docs/adr/002-scope-tags-knowledge-transfer.md) | Scope Tags + Knowledge Transfer | Accepted |
| [003](docs/adr/003-wal-protocol.md) | WAL Protocol | Accepted |
| [004](docs/adr/004-hot-warm-cold-tiering.md) | HOT/WARM/COLD Tiering | Accepted |
| [005](docs/adr/005-git-as-knowledge-exchange.md) | Git as Knowledge Exchange | Accepted |
| [006](docs/adr/006-memory-entry-format.md) | Memory Entry Format | Accepted |
| [007](docs/adr/007-concurrent-write-locking.md) | Concurrent Write Locking | Accepted |
## Directory Structure
```
.palaia/
├── config.json # Configuration
├── palaia.db # SQLite backend (when database_backend=sqlite)
├── .lock # Advisory file lock
├── hot/ # Active memories (< 7 days or high score)
│ └── <uuid>.md
├── warm/ # Occasional access (7-30 days)
│ └── <uuid>.md
├── cold/ # Archive (> 30 days)
│ └── <uuid>.md
├── wal/ # Write-Ahead Log entries (legacy, migrated to DB)
├── index/ # JSON indexes (legacy, migrated to DB)
│ ├── metadata.json # → entries table in palaia.db
│ └── embeddings.json # → embeddings table in palaia.db
├── memos/ # Inter-agent messages
├── projects.json # Project definitions
├── locks/ # Project-level advisory locks
└── runs/ # Process execution state
```
## Module Map
```
palaia/
├── __init__.py # Version
├── cli.py # CLI entry point (argparse + thin command handlers)
├── config.py # Configuration loading/saving, root detection
├── enums.py # Type-safe string enums (Tier, EntryType, Scope, ...)
├── frontmatter.py # Shared YAML frontmatter parser/serializer
│
├── # ── Core Storage ──
├── store.py # Memory store with tier routing, WAL, locking
├── entry.py # Entry format (YAML frontmatter + markdown body)
├── wal.py # Write-Ahead Log (delegates to backend)
├── lock.py # File-based advisory locking (fcntl/msvcrt)
├── project_lock.py # Project-level advisory locks for multi-agent
│
├── # ── Storage Backends ──
├── backends/
│ ├── __init__.py # Factory + auto-detection (provider chain)
│ ├── protocol.py # StorageBackend Protocol (abstract contract)
│ ├── sqlite.py # SQLite + sqlite-vec backend (zero-config default)
│ ├── postgres.py # PostgreSQL + pgvector backend (distributed teams)
│ └── migrate.py # Flat-file → backend migration
│
├── # ── Search & Embeddings ──
├── search.py # SearchEngine: hybrid BM25 + semantic search
├── bm25.py # Shared BM25 implementation
├── embeddings.py # Multi-provider embedding chain (OpenAI, Gemini, FastEmbed, Ollama, ...)
├── index.py # EmbeddingCache (delegates to backend)
├── metadata_index.py # MetadataIndex (delegates to backend)
├── embed_server.py # Long-lived embedding subprocess (JSON-RPC)
│
├── # ── Services ──
├── services/
│ ├── __init__.py
│ ├── write.py # Write/edit orchestration
│ ├── query.py # Search/get/list orchestration
│ ├── status.py # System status collection
│ └── admin.py # Init, GC, config, recovery
│
├── # ── Features ──
├── decay.py # Decay scoring + tier classification
├── scope.py # Scope validation + access control
├── significance.py # Significance scoring for auto-capture
├── nudge.py # Adaptive agent coaching
├── memo.py # Inter-agent messaging (ADR-010)
├── project.py # Project management (ADR-008)
├── process_runner.py # Process step execution
├── ingest.py # Document ingestion / RAG (ADR-009)
├── packages.py # Knowledge package export/import
├── sync.py # Entry import/export
├── migrate.py # Format migration from external sources
├── priorities.py # Injection priority management (per-agent/project)
├── curate.py # Knowledge curation (clustering, dedup, KEEP/MERGE/DROP)
├── ui.py # Terminal formatting utilities
│
├── # ── Diagnostics ──
├── doctor/
│ ├── __init__.py # Public API: run_doctor(), apply_fixes()
│ ├── checks.py # Health check functions
│ ├── fixes.py # Auto-repair logic
│ └── detection.py # Legacy system detection
│
└── SKILL.md # Agent skill documentation
packages/openclaw-plugin/
├── index.ts # Plugin entry point (definePluginEntry)
├── src/
│ ├── context-engine.ts # ContextEngine adapter (7 lifecycle hooks)
│ ├── hooks/
│ │ ├── index.ts # Hook registration
│ │ ├── recall.ts # Memory injection (before_prompt_build)
│ │ ├── capture.ts # Auto-capture (agent_end)
│ │ ├── state.ts # Session state management
│ │ └── reactions.ts # Emoji reactions (Slack)
│ ├── runner.ts # CLI subprocess runner + embed server manager
│ └── types.ts # Local type definitions (OpenClawPluginApi)
├── skill/SKILL.md # Bundled agent skill documentation
└── openclaw.plugin.json # Plugin manifest
```
## Storage Backend Architecture
```
Detection order (provider chain):
1. Config: database_url → PostgreSQL + pgvector
2. Env: PALAIA_DATABASE_URL → PostgreSQL + pgvector
3. Default → SQLite (zero-config, single file, WAL mode)
4. Legacy fallback → JSON files (backward compatible, auto-migrated to SQLite)
StorageBackend Protocol
├── SQLiteBackend — Zero-config, embedded, single file
│ ├── sqlite3 (stdlib) — Metadata + WAL
│ ├── sqlite-vec — Vector KNN search (optional)
│ └── BM25 — Full-text keyword search
│
└── PostgresBackend — Production, distributed teams
├── psycopg — Connection (pip install 'palaia[postgres]')
├── pgvector HNSW — ANN vector search (millions of entries)
└── tsvector — Language-aware full-text search
```
## Data Flow
### Write Path
```
1. Validate body (non-empty)
2. Scope cascade: explicit > project default > config default > "team"
3. Dedup check via backend.find_by_hash() [O(1) with index]
4. Create entry (UUID, frontmatter, auto-title)
5. Acquire lock
6. WAL: log pending entry (with payload for recovery)
7. Write .md file to hot/ (atomic: write tmp → fsync → rename)
8. WAL: mark committed
9. Release lock
10. Update metadata index + compute embedding (fire-and-forget)
```
### Search Path
```
1. Build BM25 index from hot + warm (cached, invalidated on write)
2. Apply structured filters (project, type, status, ...)
3. BM25 keyword ranking
4. Semantic embedding ranking (cached vectors, cosine similarity)
5. Hybrid combination: 0.4 × BM25 + 0.6 × embedding
6. Return top-K with full metadata
```
### GC / Tier Rotation
```
1. Phase 1 (unlocked): Scan all entries, compute decay scores
2. Phase 2 (locked): Move entries between tiers via WAL
3. Budget enforcement: Prune lowest gc_score entries if over limits
4. Cleanup: WAL, embedding cache, metadata index
```
## OpenClaw Integration
```
OpenClaw Agent
↓ registerContextEngine("palaia")
palaiaContextEngine
├── bootstrap() → WAL recovery + embed-server start
├── ingest(messages) → Auto-capture (LLM + rule-based extraction)
├── assemble(budget) → Token-budget-aware contextual recall
├── compact() → palaia gc
├── afterTurn(turn) → State cleanup + emoji reactions
├── prepareSubagentSpawn → Scope/workspace propagation
└── onSubagentEnded → Sub-agent memory merge
Agent Tools:
├── memory_search(query) → palaia query --json
├── memory_get(path) → palaia get --json
└── memory_write(content) → palaia write --json (with dedup guard)
```
## Crash Safety
The WAL guarantees that no write is lost:
- Every write logs intent + payload to WAL before touching data
- On startup, `recover()` replays any pending entries
- Atomic file writes (tmp + fsync + rename) prevent partial writes
- File locking prevents concurrent corruption
- SQLite backend uses native WAL mode for additional safety