docs/ARCHITECTURE.md
Rama main · réplica de lectura del repositorio original.
# Architecture
## 🏗️ System Overview
Actual-sync is a scheduled automation service that orchestrates bank transaction synchronization across multiple Actual Budget server instances. It operates as a single-process Node.js application with scheduled job execution and robust error handling.
---
## 📐 System Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ Actual-sync Service │
│ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Scheduler │─────▶│ Sync Engine │ │
│ │ (node-schedule)│ │ │ │
│ └──────────────┘ └──────┬───────┘ │
│ │ │
│ ┌───────▼────────┐ │
│ │ Retry Handler │ │
│ │ (Exponential │ │
│ │ Backoff) │ │
│ └───────┬────────┘ │
│ │ │
│ ┌─────────────────┼─────────────────┐ │
│ │ │ │ │
│ ┌─────▼──────┐ ┌─────▼──────┐ ┌─────▼──────┐ │
│ │ Server 1 │ │ Server 2 │ │ Server N │ │
│ │ Client │ │ Client │ │ Client │ │
│ └─────┬──────┘ └─────┬──────┘ └─────┬──────┘ │
└──────────────┼────────────────┼────────────────┼──────────┘
│ │ │
│ │ │
┌───────▼────────┐ ┌────▼───────┐ ┌──────▼──────┐
│ Actual Server │ │ Actual │ │ Actual │
│ (Main) │ │ Server │ │ Server │
│ │ │ (Alejandro)│ │ (...) │
└───────┬────────┘ └────┬───────┘ └──────┬──────┘
│ │ │
└────────────────┼────────────────┘
│
┌───────▼────────┐
│ GoCardless/ │
│ Nordigen │
│ (Open Banking)│
└────────────────┘
```
---
## 🧩 Component Descriptions
### 1. Scheduler (`node-schedule`)
**Responsibility**: Trigger periodic bank synchronization at configured intervals
**Key Features**:
- Cron-based job scheduling
- Timezone-aware execution (config `timezone`, default the process timezone)
- Next invocation calculation and logging
**Location**: `sync_all_banks.js` (lines 143-156)
**Configuration**:
```javascript
const SCHEDULE_CRON = '03 03 */2 * *'; // Every other day at 03:03 AM
```
---
### 2. Sync Engine
**Responsibility**: Orchestrate synchronization workflow for all configured servers
**Key Functions**:
- Iterate through server configurations
- Initialize Actual API clients
- Download budget files
- Trigger account-level bank syncs
- Perform file synchronization
**Location**: `sync_all_banks.js`
**Main Functions**:
- `syncAllBanks()`: Iterate through all servers
- `syncBank(server)`: Sync a single server instance
**Workflow**:
1. Create/verify data directory
2. Initialize Actual API connection
3. Download budget file using sync ID
4. Fetch all accounts; partition into syncable / manual / closed (`lib/accountFilter.js`)
5. Persist the account snapshot to SQLite for the dashboard (best-effort; a write failure is logged but never aborts the sync — see Account Metadata below)
6. Trigger file sync
7. For each syncable account, trigger bank sync
8. Perform final file sync
9. Shutdown API connection
**Account Metadata (dashboard syncability)**: After partitioning, each account is classified `syncable` (bank-linked + open), `manual` (not bank-linked), or `closed` and written to the `account_metadata` table via `SyncHistoryService.replaceAccountMetadata()` (the server's previous snapshot is replaced in one transaction). The dashboard reads it at `GET /api/dashboard/accounts`, so it can show per-account badges **without a live Actual connection**. The data is as fresh as the last sync.
**Payment alert snapshot (dashboard payments)**: Right after `maybeRunScheduleAlerts` (still inside the sync, Actual session open), `persistPaymentSnapshot()` in `lib/scheduleAlertsStep.js` saves the #258 evaluation into `payment_alert_status` and `payment_alert_run` through `SyncHistoryService.replacePaymentSnapshot()` (one transaction: upsert the evaluations, delete rows for rules that no longer evaluate, write the run row). A failed evaluation only updates the run row (`recordPaymentRunFailure`), a server without a `scheduleAlerts` block has its rows deleted, and any persistence error is logged at WARN and never changes the sync result. The dashboard reads it at `GET /api/dashboard/payments` without touching Actual; `lastAlertAt` is joined from the `schedule_alerts` ledger at request time because the ledger is written after the snapshot. Same pattern as account metadata above.
**Schedule snapshot and dashboard rules (#262)**: in the same step, `persistPaymentSnapshot()` also saves the hydrated Actual schedules (name, cadence text, amount, account, payee, next date, completed) into `actual_schedule_snapshot` through `SyncHistoryService.replaceScheduleSnapshot()`, with its own error handling (a write failure logs a WARN and keeps the previous snapshot). A failed evaluation keeps the previous snapshot and a server without a block has it deleted. The dashboard's rule editor reads it at `GET /api/dashboard/payment-schedules/:server` with no connection to Actual. Rules created in the dashboard live in `payment_alert_rule`; `evaluateScheduleAlerts` merges them with the config rules through `mergeRules()` (config wins on an id clash), drops any that no longer pass the shared validator (`lib/scheduleAlertValidation.js`) and evaluates the rest with the unchanged engine. The routes are in `services/paymentRuleRoutes.js`; the preview and test send in `lib/scheduleAlertDelivery.js` (`previewRule`) never write the ledger.
**Payment alert overrides (#261)**: the dashboard's per-alert switch and mute live in `payment_alert_override` (one row per server and alert id, written by the `PUT`/`DELETE .../state` routes through `SyncHistoryService.setOverride()`). At the end of each sync `deliverScheduleAlerts` reads them once per run and `deliver()` applies the pure `ruleState()` from `lib/scheduleAlertRules.js` (config `enabled: false`, then the override, then the default): a muted event is recorded as a `muted` ledger row instead of sent, and once the mute day has passed one `muteEnded` summary per destination is sent. Evaluation and the snapshot are untouched, so the Payments tab always shows the real state.
---
### 3. Retry Handler
**Responsibility**: Implement resilient error handling with intelligent retry logic
**Key Features**:
- Exponential backoff for rate limits and network failures
- Configurable maximum retry attempts
- Specific error categorization (rate limits, network failures, non-retryable)
**Location**: `sync_all_banks.js` (lines 33-63)
**Configuration**:
```javascript
const MAX_RETRIES = 5;
const BASE_RETRY_DELAY_MS = 3000;
```
**Retry Logic**:
```javascript
retryDelay = BASE_RETRY_DELAY_MS * (2 ** attemptNumber)
```
**Retryable Errors**:
- Rate limit exceeded (Nordigen API)
- Network failures
- Connection resets (ECONNRESET)
- DNS resolution failures (ENOTFOUND)
---
### 4. Server Client Instances
**Responsibility**: Manage independent connections to Actual Budget servers
**Configuration Structure** (in `config.json`):
```json
{
"name": "ServerName",
"url": "http://server:5006",
"password": "password",
"syncId": "sync_id",
"dataDir": "/app/data_temp"
}
```
**Key Characteristics**:
- Isolated data directories per server
- Independent authentication
- Sequential processing (no parallel syncs)
- External configuration file (not hardcoded)
- Startup validation: business-logic checks and JSON-schema rules (type/range/required/format/pattern/enum) both hard-fail at startup (#121); unknown/typo'd keys only warn. `CONFIG_STRICT=false` downgrades the schema hard-fails to warnings for migration
**Location**: Loaded from `config/config.json` via `src/lib/configLoader.js`
---
### 5. Configuration Loader
**Responsibility**: Load and validate server configuration from external file
**Key Features**:
- Load configuration from `config.json`
- Validate against JSON schema (`config.schema.json`) — hard-fails on type/range/required/format/pattern/enum (#121, `CONFIG_STRICT=false` to downgrade), warns on unknown keys; business-logic validation (`validateLogic()`) is the other hard-failure layer
- Apply default values for optional settings
- Security warnings (weak passwords, HTTP usage)
- Duplicate server name detection
**Location**: `src/lib/configLoader.js`
**Validation Checks**:
- Required fields present
- Valid JSON syntax
- Schema compliance
- Logical constraints (e.g., unique names)
- Security best practices
---
### 6. Account Discovery Utility
**Responsibility**: List all bank accounts accessible on a specific server
**Purpose**:
- Verify server connectivity
- Discover account IDs for debugging
- Validate bank connection configurations
**Location**: `scripts/listAccounts.js`
**Usage**:
```bash
npm run list-accounts
```
---
## 🔄 Data Flow
### Scheduled Sync Flow
```
1. Scheduler triggers at cron time
↓
2. syncAllBanks() invoked
↓
3. For each server in configuration:
↓
4. syncBank(server) called
↓
5. Initialize Actual API with server credentials
↓
6. Download budget file (with retry)
↓
7. Fetch accounts from server
↓
8. Trigger file sync (with retry)
↓
9. For each account:
├─ Trigger bank sync (with retry)
└─ Log result (success/failure per account)
↓
10. Trigger final file sync (with retry)
↓
11. Shutdown Actual API connection
↓
12. Repeat for next server
```
### Force Sync Flow
```
1. User runs: node sync_all_banks.js --force-run
↓
2. run() function detects --force-run flag
↓
3. Immediately invokes syncAllBanks()
↓
4. [Same workflow as Scheduled Sync Flow steps 3-12]
↓
5. Process exits after completion
```
### Account Discovery Flow
```
1. User runs: node getAccounts.js
↓
2. Initialize Actual API with configured server
↓
3. Download budget file
↓
4. Fetch accounts using getAccounts()
↓
5. Log account names and IDs
↓
6. Shutdown API connection
```
---
## 🗂️ File Structure
```
Actual-sync/
├── index.js # Application entry point
├── package.json # Dependencies, scripts, and metadata
├── README.md # Project overview and quick start
├── .gitignore # Git ignore rules
├── .env # Environment variables (optional)
├── src/
│ ├── syncService.js # Main sync service with scheduler
│ ├── lib/
│ │ └── configLoader.js # Configuration loader with validation
│ └── utils/ # Utility modules (future)
├── config/
│ ├── config.json # Server configuration (not in repo)
│ ├── config.example.json # Example configuration template
│ └── config.schema.json # JSON schema for validation
├── scripts/
│ └── listAccounts.js # Account discovery utility
├── docs/ # Project documentation
│ ├── README.md # Documentation index
│ ├── ARCHITECTURE.md # This document
│ ├── CONFIG.md # Configuration reference
│ ├── MIGRATION.md # Migration guide
│ └── ... # Additional documentation
└── node_modules/ # Installed npm packages
```
---
## 🔌 External Dependencies
### Actual Budget API (`@actual-app/api`)
**Purpose**: Official client library for interacting with Actual Budget servers
**Key Methods Used**:
- `init(options)`: Initialize API connection
- `downloadBudget(syncId)`: Download budget file from server
- `getAccounts()`: Fetch all budget accounts
- `sync()`: Trigger file synchronization
- `runBankSync(options)`: Trigger bank transaction sync for specific account
- `shutdown()`: Clean up API connection
**Documentation**: https://actualbudget.org/docs/api/
---
### Node-Schedule
**Purpose**: Cron-style job scheduling within Node.js process
**Key Features Used**:
- `scheduleJob(cronExpression, callback)`: Create scheduled job
- `job.nextInvocation()`: Get next scheduled execution time
**Cron Expression Format**: `minute hour day month dayOfWeek`
---
### Moment-Timezone
**Purpose**: Timezone-aware date/time handling
**Usage**: Log current time in the configured timezone for debugging
---
## 🛡️ Error Handling Strategy
### Error Categories
1. **Retryable Errors**:
- Rate limit exceeded (exponential backoff)
- Network failures (exponential backoff)
- Connection resets
- DNS failures
2. **Non-Retryable Errors**:
- Authentication failures
- Invalid sync IDs
- API errors (non-rate-limit)
### Error Propagation
- **Account-level errors**: Logged but don't stop other accounts from syncing
- **Server-level errors**: Logged but don't stop other servers from syncing
- **Fatal errors**: Stop the current server sync and move to next server
### Logging Strategy
- Console-based logging (stdout/stderr)
- Detailed error messages with stack traces
- Progress indicators for each operation
- Next schedule time displayed on startup
---
## 🔑 Design Decisions
### Sequential Server Processing
**Decision**: Process servers sequentially, not in parallel
**Rationale**:
- Simplifies error handling and logging
- Avoids overwhelming GoCardless API with concurrent requests
- Reduces risk of rate limiting
- Easier to debug issues with specific servers
- Required for correctness: `@actual-app/api` is one process-wide session, so two syncs at once
would share (and corrupt) the same open budget
**Enforcement** (#265): every sync goes through `syncBank`, which queues it on the single
`SyncQueue` in `src/lib/syncQueue.js`. Scheduled jobs, auto-retries, the dashboard (including
"Sync all") and Telegram `/sync` all wait their turn, in request order. A server that is already
waiting or running is not queued again; the dashboard and Telegram answer "already queued". A
failing sync releases the queue for the next one. Telegram does not wait for the sync to finish,
so a long queue never blocks the bot's polling loop.
The queue is per process. `--force-run` (`npm run sync`) and `scripts/listAccounts.js` start
their own process with their own API session, so do not run them while the service is running.
Hung calls (#272): each Actual API call a sync makes (`init`, `downloadBudget`, `loadBudget`,
`aqlQuery`, `sync`, `shutdown`) is bounded by `sync.phaseTimeoutSeconds` (default 300), and
`runBankSync` by 60 seconds per account. A call that runs out of time fails the sync with the
phase named. A timeout only stops waiting: the call itself keeps running inside the one shared
Actual session, so the sync keeps its queue slot for up to one more phase timeout while that call
finishes, and only then runs `shutdown`, so a late download cannot leave its budget open for the
next server. If a call is still running when the next sync starts, that sync waits up to its own
phase timeout and then fails with `ACTUAL_SESSION_BUSY` rather than open a second budget in the
same session. When the calls did finish in time, a budget a late download left open is closed by
Actual itself (`close-budget` inside `download-budget` and `load-budget`); Actual's `shutdown`
is a no-op before `init`, so Actual-sync runs no extra one. A sync that refuses also stops tracking the stuck call, so a call that does not finish in
time costs one refused sync, not every sync until restart. Limit: the Actual API cannot cancel a
call, so one that outlives both waits and lands later can still disturb the following sync, which
fails and lets the next one recover. A `runBankSync` that times out gets one more phase timeout to
finish before the next account starts; if it finishes in that time and succeeded, the account
counts as succeeded; if it does not finish, the remaining accounts are reported as failed and the
final file sync and schedule alerts are skipped (`shutdown` syncs before closing). A late call that settles is logged with its phase and outcome. With the default of 300 seconds, a server that
never answers holds the queue for up to about 20 minutes (the stuck call, the wait, a stuck
`shutdown`, and the next sync's wait). The dashboard status
(`/api/dashboard/status`, `runningSync`) names the server holding the queue and when it started;
public `/health` does not, so server names stay private.
### Exponential Backoff
**Decision**: Use exponential backoff for retries (3s, 6s, 12s, 24s, 48s)
**Rationale**:
- Prevents overwhelming external APIs during rate limit periods
- Gives transient network issues time to resolve
- Standard industry practice for API retry logic
### Isolated Data Directories
**Decision**: Each server uses a separate data directory
**Rationale**:
- Prevents cross-contamination of budget data
- Allows concurrent budget file management
- Simplifies cleanup and debugging
### External Configuration Files
**Decision**: Use `config.json` for server configuration with JSON schema validation
**Rationale**:
- Keeps sensitive data out of source control
- Easy to modify without code changes
- Supports multiple servers cleanly
- Schema validation prevents configuration errors
- No code changes needed to add/remove servers
- Better structure than environment variables for complex config
---
## 🔄 State Management
### Stateless Operation
The service is **stateless** between sync runs:
- No persistent storage of sync history
- No database or file-based state
- Each run is independent
### Temporary State
**Data Directories**: Actual API creates temporary local cache in configured data directories. These are reused across runs but can be deleted safely.
---
## 🚀 Deployment Considerations
### Single Instance
Currently designed to run as a single process. Not suitable for:
- Load balancing across multiple instances
- High-availability deployments
- Horizontal scaling
### Resource Requirements
**CPU**: Low (mostly I/O-bound waiting for API responses)
**Memory**: Moderate (depends on budget file size)
**Disk**: Minimal (temporary cache files)
**Network**: Outbound HTTPS to Actual servers and GoCardless API
### Recommended Deployment
- Docker container with scheduled restart policy
- Systemd service with automatic restart
- Kubernetes CronJob (requires refactoring to one-shot execution)
---
## 🔄 CI/CD Pipeline
### Overview
Actual-sync uses a comprehensive GitHub Actions-based CI/CD pipeline for automated testing, building, and deployment.
**Pipeline File**: `.github/workflows/ci-cd.yml`
### Pipeline Architecture
```
┌─────────────┐
│ Version │
│ Generation │ ← Dynamic Git-based versioning
└──────┬──────┘
│
├─────────────┬────────────┬─────────────────┐
▼ ▼ ▼ ▼
┌────────┐ ┌────────┐ ┌─────────┐ ┌──────────────┐
│ Lint │ │ Test │ │ Build │ │ Validate │
│ │ │ │ │ │ │ Docker Desc. │
└────┬───┘ └────┬───┘ └────┬────┘ └──────┬───────┘
│ │ │ │
└────────────┴────────────┴────────────────┘
│
▼
┌─────────────────┐
│ Docker Test │
│ Build │
└────────┬────────┘
│
┌────────────┴────────────┐
▼ ▼
┌──────────────┐ ┌──────────────┐
│ Docker │ │ Security │
│ Publish │ │ Scan │
└──────┬───────┘ └──────┬───────┘
│ │
└────────────┬───────────┘
▼
┌─────────────────┐
│ Deployment │
│ Test │
└────────┬────────┘
│
▼
┌──────────┐
│ Release │
│ (if tag) │
└──────────┘
```
### Pipeline Jobs
| Job | Purpose | Duration |
|-----|---------|----------|
| **Version** | Generate dynamic version from Git context | ~10s |
| **Lint** | Validate code syntax and formatting | ~30s |
| **Test** | Run Jest tests with coverage | ~2-3min |
| **Build** | Build application and verify artifacts | ~45s |
| **Validate Docker Desc** | Ensure Docker Hub description ≤100 chars | ~5s |
| **Docker Test** | Test Docker build without publishing | ~3-5min |
| **Docker Publish** | Build and push multi-platform images | ~10-15min |
| **Security Scan** | Trivy vulnerability scanning with SARIF | ~2-3min |
| **Deployment Test** | Verify published images work correctly | ~1-2min |
| **Release** | Create GitHub release (tagged builds only) | ~30s |
**Total Pipeline Duration**: ~15-20 minutes for full run
### Dynamic Versioning
The pipeline uses `scripts/get_version.sh` to generate context-aware version strings:
| Branch/Tag | Generated Version | Docker Tags |
|------------|------------------|-------------|
| `main` with tag `v1.0.0` | `1.0.0` | `latest`, `main`, `1.0.0` |
| `main` without tag | `1.0.0-main-abc1234` | `main`, `1.0.0-main-abc1234` |
| `develop` | `1.0.0-dev-abc1234` | `develop`, `1.0.0-dev-abc1234` |
| `feature/auth` | `1.0.0-feature-auth-abc1234` | `1.0.0-feature-auth-abc1234` |
**Format**: `<base_version>-<context>-<commit_hash>`
**Base Version**: Read from `package.json`
**Version Exposure**:
- Environment variable: `VERSION`
- Health endpoint: `/health` (includes version field)
- Metrics endpoint: `/metrics` (includes version field)
- Docker labels: OCI metadata
- Application startup: Logged to console
### Docker Publishing
**Registries**:
1. **Docker Hub**: `<username>/actual-sync:<tag>`
2. **GitHub Container Registry**: `ghcr.io/<owner>/actual-sync:<tag>`
**Platforms**:
- `linux/amd64` (Intel/AMD 64-bit)
- `linux/arm64` (ARM 64-bit, Apple Silicon, Raspberry Pi)
**Build Strategy**:
- Multi-stage build (builder + production)
- Build cache via GitHub Actions cache
- VERSION passed as build argument
- OCI labels with version metadata
### Security Scanning
**Scanner**: [Trivy](https://github.com/aquasecurity/trivy) by Aqua Security
**Scans**:
- OS packages (Alpine Linux)
- Application dependencies (npm packages)
- Filesystem vulnerabilities
- Hardcoded secrets (basic detection)
**Severity Levels**: CRITICAL, HIGH, MEDIUM
**Report Formats**:
- SARIF → GitHub Security tab
- Table → Workflow summary
**Integration**: Results uploaded to GitHub Code Scanning for centralized tracking
### Triggers
| Trigger | Branches | Actions |
|---------|----------|---------|
| **Push** | `main`, `develop` | Full pipeline with Docker publish |
| **Pull Request** | `main`, `develop` | Testing only (no publish) |
| **Git Tags** | `v*` | Full pipeline + GitHub release |
| **Manual** | Any branch | Configurable (skip tests, custom tags) |
### Manual Workflow Dispatch
The workflow supports manual triggers with options:
**Parameters**:
1. `skip_tests` - Skip test execution (emergency use)
2. `skip_docker_publish` - Test pipeline without publishing
3. `docker_tag_suffix` - Add custom tag suffix
**Use Cases**:
- Hotfix deployments
- Testing pipeline changes
- Custom-tagged builds
- Emergency deployments
### Required Secrets
Configure in repository settings → Secrets → Actions:
| Secret | Purpose | Where to Get |
|--------|---------|--------------|
| `DOCKER_USERNAME` | Docker Hub username | Docker Hub account |
| `DOCKER_TOKEN` | Docker Hub access token | Docker Hub → Security → New Token |
| `GITHUB_TOKEN` | GHCR authentication | Auto-provided by GitHub |
### CI/CD Documentation
See **[docs/CI_CD.md](CI_CD.md)** for comprehensive documentation including:
- Complete setup instructions
- Manual trigger examples
- Troubleshooting guide
- Security best practices
- Workflow optimization tips
### Deployment Verification
The pipeline includes automated deployment testing:
1. Pull published images from both registries
2. Test container startup
3. Verify version environment variable
4. Confirm health check endpoint
**Ensures**: Published images are immediately usable by end users
### Release Management
**Automated Releases**: Created for Git tags on `main` branch
**Release Contents**:
- Version number
- Docker pull commands (Docker Hub + GHCR)
- Generated changelog from commits
- Security scan status link
- Auto-generated release notes
**Example**:
```bash
# Create release
git tag -a v1.0.0 -m "Release v1.0.0"
git push origin v1.0.0
# Pipeline automatically:
# 1. Runs full test suite
# 2. Builds and publishes Docker images
# 3. Scans for vulnerabilities
# 4. Creates GitHub release with changelog
```
---
**Last Updated**: December 7, 2025