docs/NOTIFICATIONS.md
Rama main · réplica de lectura del repositorio original.
# Notification System
The Notification System provides automated alerts and interactive commands to monitor and control your Actual-sync service. Get instant notifications for sync events via email and webhooks, or use the Telegram bot for real-time status updates and configuration changes.
## Table of Contents
- [Features](#features)
- [Configuration](#configuration)
- [Telegram Bot](#telegram-bot)
- [Email Notifications](#email-notifications)
- [Webhook Notifications](#webhook-notifications)
- [Message templates](#message-templates)
- [Notification Thresholds](#notification-thresholds)
- [Rate Limiting](#rate-limiting)
- [Use Cases](#use-cases)
- [Troubleshooting](#troubleshooting)
- [Best Practices](#best-practices)
## Features
- **Interactive Telegram Bot**: Real-time commands for status, history, and configuration
- **Multiple Notification Channels**: Email, Slack, Discord, Telegram, ntfy, and generic webhooks (ntfy/Gotify/Home Assistant/n8n/custom)
- **Smart Thresholds**: Trigger notifications based on consecutive failures or failure rates
- **Rate Limiting**: Prevent notification spam with configurable intervals and limits
- **Rich Context**: Includes error details, correlation IDs, and sync statistics
- **Automatic Tracking**: Monitors sync results and tracks failure patterns
- **Configurable Notifications**: Choose to receive notifications for all syncs, only failures, or never — globally, per channel, or per individual webhook (see [Notification mode](#notification-mode-notifyonsuccess))
- **Graceful Degradation**: Continues operation even if notifications fail
## Sync notification content
Every sync notification (across Telegram/text, Slack, Discord and email) explains
the per-account outcome, on **both** success and failure:
- An **`Accounts: N synced, M failed, K skipped`** summary line.
- A **`Transactions: N new, M updated`** line: how many transactions the bank
sync imported, and how many existing ones it matched and updated, summed over
the run. It appears only when at least one count is nonzero, so a quiet sync
looks exactly as it did before the line existed. The counts come from Actual
itself (`runBankSync`, `@actual-app/api` 26.11.0 or later). An account whose
bank sync finished after its 60 s timeout adds 0, because its result arrived
too late to be read. The numbers are Actual's own, so they can differ from
what the bank shows: a split transaction counts its parent plus each part,
and a newly linked account's first sync includes its starting-balance
transaction.
- **✅ Synced**: each bank-linked account that synced, with its own count:
`Account 1: 5 new transactions`, `Account 2: no new transactions`,
`Account 3: 1 new transaction, 2 updated` (`, M updated` only when M is above
0). An account whose bank sync finished after its 60 s timeout reads
`synced, count unknown`, because its result arrived too late to be read; it is
never shown as `no new transactions`. On Slack and Discord a long list is cut
with `…and N more`.
- **❌ Failed** — accounts whose bank sync errored (with the error). This appears on
the failure path too — e.g. a single rate-limited account shows up here rather
than only as a terse top-level error.
- **⏭️ Skipped (not bank-linked / closed)** — accounts that were *not* attempted,
each with its reason: `closed` or `not-linked` (no bank link). Only bank-linked,
open accounts are bank-synced (see the account-syncability behavior), so this
explains why e.g. a budget with 7 accounts may report `1 synced … 6 skipped`.
A **server-level failure** that never reached the accounts (e.g. connection or
auth error) shows only the `Error:` line — the account summary and synced/failed/
skipped sections are **suppressed** (no misleading `0 synced` clutter).
## Configuration
Add the `notifications` section to your `config/config.json`:
```json
{
"notifications": {
"email": {
"enabled": true,
"host": "smtp.gmail.com",
"port": 587,
"secure": false,
"auth": {
"user": "your-email@gmail.com",
"pass": "your-app-specific-password"
},
"from": "actual-sync@example.com",
"to": ["admin@example.com", "team@example.com"]
},
"webhooks": {
"slack": [
{
"name": "Main Channel",
"url": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
}
],
"discord": [
{
"name": "Alerts Channel",
"url": "https://discord.com/api/webhooks/YOUR/WEBHOOK"
}
],
"telegram": [
{
"name": "Production Alerts",
"botToken": "123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11",
"chatId": "123456789"
}
]
},
"thresholds": {
"consecutiveFailures": 3,
"failureRate": 0.5,
"ratePeriodMinutes": 60
},
"rateLimit": {
"minIntervalMinutes": 15,
"maxPerHour": 4
}
}
}
```
### Configuration Schema
#### Notification mode (`notifyOnSuccess`)
Decides **which sync results reach a channel**. Every channel supports it and the behaviour is identical everywhere, with one exception: the deprecated [`webhooks.telegram`](#webhook-settings) entries, which have no such key — set it on `notifications.telegram` instead, which governs them too.
| Mode | `success` | `partial` | `failure` | Startup | Test notification |
|------|-----------|-----------|-----------|---------|-------------------|
| `always` (default) | send | send | send¹ | send | send |
| `errors_only` | — | send | send¹ | send | send |
| `activity_only` | send² | send | send¹ | send | send |
| `never` | — | — | — | — | send |
¹ Failures additionally pass through [thresholds](#notification-thresholds) and [rate limiting](#rate-limiting), which are unchanged and apply to failures only.
² Only when the sync imported or updated at least one transaction (the counts on the Transactions line, see below); a quiet success sends nothing. On an `@actual-app/api` older than 26.11.0 the counts are always 0, so this mode then behaves like `errors_only`.
**Key points:**
- **`partial` counts as an error.** A sync where some accounts failed is a failure signal, so `errors_only` reports it. (This matches ntfy, which already maps partial to `priorityOnFailure`.)
- **`never` turns the channel off entirely**, failures included. It is an explicit opt-out, not a noise filter. The guarantee that holds is narrower: **`always`, `errors_only` and `activity_only` never suppress a failure.**
> ⚠️ **Upgrading from 1.10.x or earlier?** `notifyOnSuccess` previously existed in the schema but gated nothing — a channel set to `never` still received everything, failures included. From 1.11.0 it works, so that same config now receives **nothing at all**. If you meant "don't tell me about successful syncs", you want **`errors_only`**, not `never`. The service logs a `WARN` at startup naming every channel it has muted.
- **Test notifications always send**, on every mode including `never` — otherwise you could not verify a channel you had muted. `enabled: false` still wins over everything.
- **Startup notifications** have no sync status, so `errors_only` and `activity_only` do not apply to them; only `never` suppresses them.
- **`activity_only` follows the same precedence** as the other modes (webhook entry, then channel, then global), and a channel set to it is not listed as muted at startup.
**Resolution order** — first value found wins:
1. The individual webhook entry (`webhooks.slack[]`, `webhooks.discord[]`, `webhooks.generic[]`)
2. The channel (`notifications.email`, `notifications.telegram`, `notifications.ntfy`)
3. The global default (`notifications.notifyOnSuccess`)
4. Built-in default: `always`
```json
"notifications": {
"notifyOnSuccess": "errors_only",
"email": { "notifyOnSuccess": "always" },
"webhooks": {
"discord": [
{ "name": "alerts", "url": "https://discord.com/api/webhooks/xxx" },
{ "name": "audit-log", "url": "https://discord.com/api/webhooks/yyy", "notifyOnSuccess": "always" }
]
}
}
```
Above: everything defaults to `errors_only`; email overrides back to `always`; the `alerts` Discord webhook inherits `errors_only` while `audit-log` records every sync.
> **Telegram note:** the `/notify` bot command uses `errors` where the config key uses `errors_only`, and `activity` for `activity_only`: same modes, two spellings. `/notify` changes take effect immediately. They persist across restarts **only if** `notifications.telegram.notifyOnSuccess` is unset in config — an explicit config value wins on restart, and the bot says so when you set the mode.
#### Email Settings
| Property | Type | Required | Default | Description |
|----------|------|----------|---------|-------------|
| `enabled` | boolean | No | `false` | Enable email notifications |
| `host` | string | No | `smtp.gmail.com` | SMTP server hostname |
| `port` | integer | No | `587` | SMTP server port |
| `secure` | boolean | No | `false` | Use TLS for SMTP |
| `auth.user` | string | Yes* | - | SMTP username |
| `auth.pass` | string | Yes* | - | SMTP password or app password |
| `from` | string | Yes* | - | From email address |
| `to` | array | Yes* | - | Recipient email addresses |
| `notifyOnSuccess` | string | No | inherits | `always`, `errors_only`, `activity_only`, or `never` — see [Notification mode](#notification-mode-notifyonsuccess) |
*Required when email is enabled
#### Webhook Settings
Each webhook type (Slack, Discord, Telegram) accepts an array of webhook configurations:
**Slack, Discord:**
| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `name` | string | No | Descriptive name for logs |
| `url` | string | Yes | Webhook URL |
| `enabled` | boolean | No | Set `false` to disable without removing (default `true`) |
| `notifyOnSuccess` | string | No | `always`, `errors_only`, `activity_only`, or `never` for this entry — see [Notification mode](#notification-mode-notifyonsuccess) |
**Telegram (Legacy Webhooks - deprecated):**
| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `name` | string | No | Descriptive name for logs |
| `botToken` | string | Yes | Bot token from @BotFather |
| `chatId` | string | Yes | Chat ID (user, group, or channel) |
**Note**: The legacy Telegram webhook configuration is deprecated. Use the new `telegram` object for interactive bot features.
An entry here has no `enabled` flag — **its presence is its enablement**. If a `webhooks.telegram` entry exists it is used whenever the top-level `notifications.telegram` is absent or `enabled: false`, so setting `telegram.enabled: false` does *not* switch Telegram off while a legacy entry remains.
These entries are also the **one exception** to [`notifyOnSuccess`](#notification-mode-notifyonsuccess) being available on every channel: the key does not exist on them, and setting it there is silently ignored (an unknown key is only an advisory warning, so the service still starts). To silence Telegram, remove the entry, or set `notifyOnSuccess: "never"` on the top-level `notifications.telegram` block — which applies to the legacy entry too.
**Generic webhooks** (`webhooks.generic`): POST a documented JSON payload to any URL, so notifications work out of the box with ntfy (JSON publishing), Gotify, Home Assistant, n8n, and custom endpoints. **Microsoft Teams**: point a `generic` webhook at your Teams incoming-webhook URL — there is no dedicated `teams` channel.
| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `name` | string | No | Descriptive name for logs |
| `url` | string | Yes | Destination the JSON payload is POSTed to |
| `enabled` | boolean | No | Set `false` to disable without removing (default `true`) |
| `notifyOnSuccess` | string | No | `always`, `errors_only`, `activity_only`, or `never` for this entry — see [Notification mode](#notification-mode-notifyonsuccess) |
| `headers` | object | No | Extra HTTP headers (e.g. `Authorization`, `Priority`). Secrets are redacted in logs. |
| `contentType` | string | No | Override request Content-Type (default `application/json`) |
The JSON payload:
```json
{
"event": "sync",
"status": "success",
"server": "My Budget",
"durationMs": 1200,
"message": "Sync Successful",
"accounts": { "synced": 9, "failed": 0, "skipped": 3 },
"transactionsAdded": 2,
"transactionsUpdated": 1,
"succeededAccounts": ["..."],
"accountResults": [
{ "id": "...", "name": "...", "status": "synced", "added": 5, "updated": 0, "countsKnown": true }
],
"failedAccounts": [{ "name": "...", "error": "..." }],
"skippedAccounts": [{ "name": "...", "reason": "not-linked" }],
"timestamp": "<ISO8601>"
}
```
`transactionsAdded` and `transactionsUpdated` are integers and always present:
`0` when nothing was imported or updated, or when the bank sync did not report
counts. They are counts only; no transaction ids, payees or amounts are sent.
`accountResults` has one entry per account (synced, failed and skipped), with
`status` one of `synced`, `failed` or `skipped`. `added` and `updated` are
integers; `countsKnown` is `false` when the count could not be read (a late
success) and for failed and skipped accounts, whose `added` and `updated` are 0.
`succeededAccounts` stays the list of synced account names.
#### ntfy Settings
`notifications.ntfy` sends push notifications to an [ntfy](https://ntfy.sh) topic (header-based API: `Title` / `Priority` / `Tags`).
| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `enabled` | boolean | No | Enable ntfy (default `false`) |
| `url` | string | Yes | Full topic URL, e.g. `https://ntfy.sh/my-topic` |
| `token` | string | No | Access token sent as `Authorization: Bearer <token>` (redacted in logs) |
| `priorityOnFailure` | string | No | Priority for failures (default `high`). **Partial syncs use this too** — a partial counts as a failure. |
| `priorityOnSuccess` | string | No | Priority for success/info (default `default`) |
| `notifyOnSuccess` | string | No | `always`, `errors_only`, `activity_only`, or `never` — see [Notification mode](#notification-mode-notifyonsuccess) |
| `tags` | array | No | Base tags/emoji added to every notification |
| `icon` | string | No | Icon URL shown in the notification (sent as the `Icon` header). Defaults to the Actual-sync project icon; set to an empty string `""` to send no icon (e.g. on an air-gapped network, or set `notifications.branding: false`). |
Tip: ntfy users can also just point a `webhooks.generic` entry at their topic for JSON publishing; the dedicated `ntfy` block adds zero-config priority/tags mapping.
#### Branding
| Property | Type | Default | Description |
|----------|------|---------|-------------|
| `branding` | boolean | `true` | Stamp the Actual-sync project name and icon onto Slack/Discord/ntfy notifications. Set `false` to keep a webhook's own configured app identity, or to stop ntfy from fetching the project icon over the network (air-gapped setups). |
`notifications.branding` is a top-level flag (sibling of the channel blocks), e.g. `"notifications": { "branding": false, ... }`.
#### Telegram Bot Settings
| Property | Type | Required | Default | Description |
|----------|------|----------|---------|-------------|
| `enabled` | boolean | No | `false` | Enable Telegram bot |
| `botToken` | string | Yes* | - | Bot token from @BotFather |
| `chatId` | string | Yes** | - | Chat ID (user, group, or channel) |
| `chatIds` | array | No** | - | One or more chat IDs (alternative to `chatId`). The first entry is used for outbound notifications; the bot also accepts numeric IDs. |
| `notifyOnSuccess` | string | No | inherits | Notification mode: `always`, `errors_only`, `activity_only`, or `never` — see [Notification mode](#notification-mode-notifyonsuccess). Also settable at runtime with [`/notify`](#notify-mode), which uses `errors` for `errors_only` and `activity` for `activity_only`. |
| `pollInterval` | integer | No | `2000` | Polling interval in milliseconds for the interactive bot (minimum 100) |
*Required when Telegram bot is enabled. **Provide either `chatId` or `chatIds` (at least one).
`botToken` and `chatId` can also be supplied via the `TELEGRAM_BOT_TOKEN` and `TELEGRAM_CHAT_ID` environment variables as a fallback when not set in config.
#### Threshold Settings
**Applies to `failure` results only.** Success and partial results never pass through thresholds — use [`notifyOnSuccess`](#notification-mode-notifyonsuccess) to control those.
| Property | Type | Default | Range | Description |
|----------|------|---------|-------|-------------|
| `consecutiveFailures` | integer | `3` | 1-20 | Number of consecutive failures to trigger notification |
| `failureRate` | number | `0.5` | 0.0-1.0 | Failure rate threshold (50% = 0.5) |
| `ratePeriodMinutes` | integer | `60` | 5-1440 | Time window for failure rate calculation |
#### Rate Limit Settings
**Applies to `failure` results only**, like thresholds. It cannot quieten routine success notifications — that is [`notifyOnSuccess`](#notification-mode-notifyonsuccess).
Only a notification that **actually reached someone** consumes the budget:
- A channel muted by `notifyOnSuccess` does not consume it.
- A notification that failed to deliver on every channel does not consume it either. Otherwise a single transport blip would spend the hourly slot and silently suppress the *next* failure, even after the transport recovered.
The trade-off is that a persistently broken transport is retried on every scheduled sync rather than being rate-limited away. That is bounded by your sync schedule, and each attempt is capped by the 15-second webhook timeout. If you see repeated `Sync notification failed on every channel` warnings, fix or disable the channel — the retries are the symptom, not the problem.
| Property | Type | Default | Range | Description |
|----------|------|---------|-------|-------------|
| `minIntervalMinutes` | integer | `15` | 1-1440 | Minimum minutes between notifications |
| `maxPerHour` | integer | `4` | 1-20 | Maximum notifications per hour |
## Telegram Bot
The Telegram Bot provides interactive commands for real-time monitoring and configuration of your Actual-sync service. Unlike simple webhook notifications, the bot responds to your commands and allows you to query status, view history, and change settings directly from Telegram.
### Setup
#### 1. Create a Bot
1. Open Telegram and search for **@BotFather**
2. Send `/newbot` command
3. Follow the prompts to choose a name and username
4. **BotFather** will give you a bot token (e.g., `123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11`)
5. Save this token - you'll need it for configuration
#### 2. Get Your Chat ID
**Method 1: Using @userinfobot**
1. Search for **@userinfobot** in Telegram
2. Start a chat and send any message
3. The bot will reply with your user ID
**Method 2: Using API**
1. Start a chat with your new bot
2. Send any message to the bot
3. Visit: `https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getUpdates`
4. Look for `"chat":{"id":123456789}` in the JSON response
#### 3. Configure in config.json
```json
{
"notifications": {
"telegram": {
"enabled": true,
"botToken": "123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11",
"chatId": "123456789",
"notifyOnSuccess": "errors_only"
}
}
}
```
### Available Commands
#### `/help`
Shows all available commands and current notification mode.
**Example:**
```
/help
```
**Response:**
```
🤖 Actual-sync Bot Commands
/help - Show available commands
/ping - Check if bot is responsive
/status - Get current sync status
/history - Show recent sync history
/stats - Show sync statistics
/servers - List configured servers
/notify - Change notification settings (always/errors/never)
/errors - Show recent errors
Current notification mode: errors_only
```
#### `/ping`
Tests bot responsiveness. Useful for verifying the bot is running and can communicate.
**Example:**
```
/ping
```
**Response:**
```
🏓 Pong! Bot is responsive.
```
#### `/status`
Shows current service health, sync statistics, and last sync time.
**Example:**
```
/status
```
**Response:**
```
✅ Service Status: HEALTHY
Uptime: 127 minutes
Total Syncs: 15
Successful: 14
Failed: 1
Success Rate: 93.3%
Last Sync: 12/6/2024, 4:30 PM
```
#### `/history [limit]`
Shows recent sync history. Optionally specify the number of records to show (default: 5).
**Examples:**
```
/history
/history 10
```
**Response:**
```
📜 Recent Syncs (last 5):
✅ My-Finances
12/6/24, 4:30 PM
Duration: 2156ms
✅ My-Finances
12/6/24, 4:00 PM
Duration: 1987ms
❌ My-Finances
12/6/24, 3:30 PM
Duration: 543ms
Error: Connection timeout
```
#### `/stats`
Shows comprehensive sync statistics including averages and totals.
**Example:**
```
/stats
```
**Response:**
```
📊 Sync Statistics
Total Syncs: 25
Successful: 23
Failed: 2
Success Rate: 92.0%
Avg Duration: 2145ms
Total Accounts: 50
Failed Accounts: 2
Last Sync: 12/6/2024, 4:30 PM
```
#### `/servers`
Lists all configured servers and their sync schedules.
**Example:**
```
/servers
```
**Response:**
```
📋 Configured Servers (2):
🖥️ My-Finances
URL: http://finance-actual-budget-main:5006
Schedule: */30 * * * *
🖥️ Backup-Server
URL: http://backup-actual:5006
Schedule: 0 3 * * *
```
#### `/notify [mode]`
Changes notification preferences. Without arguments, shows current mode.
**Modes:**
- `always` - Get notified on every sync (success or failure)
- `errors` - Get notified only on failures and partial syncs
- `activity` - Failures and partial syncs, plus successful syncs that imported or updated transactions
- `never` - No notifications at all, including failures (commands still work)
**Examples:**
```
/notify
/notify always
/notify errors
/notify activity
/notify never
```
**Response (change mode):**
```
✅ Notification mode changed to: always
You will receive sync notifications.
```
**Response (view current):**
```
⚙️ Current notification mode: errors_only
Usage: /notify [always|errors|activity|never]
• always - Notify on all syncs (default)
• errors - Notify only on failures and partial syncs
• activity - Failures and partial syncs, plus successful syncs that imported or updated transactions
• never - No notifications at all, INCLUDING failures (commands still work)
```
#### `/errors [limit]`
Shows recent errors with details. Optionally specify the number of errors to show (default: 5).
**Examples:**
```
/errors
/errors 10
```
**Response:**
```
❌ Recent Errors (last 3):
🖥️ My-Finances
12/6/24, 3:30 PM
Connection timeout
Code: ETIMEDOUT
🖥️ Backup-Server
12/5/24, 11:45 PM
Invalid credentials
Code: AUTH_ERROR
```
**Response (no errors):**
```
✅ No recent errors!
```
### Non-Command Messages
If you send a message that's not a recognized command, the bot will acknowledge it and suggest using `/help`:
**Example:**
```
hello
```
**Response:**
```
I received your message: "hello"
I don't understand that command. Type /help to see available commands.
```
### Notification Examples
#### Startup Notification
When the service starts, you'll receive:
```
🚀 Actual-sync Service Started
✅ Service is now running
Servers: My-Finances
Schedules:
- My-Finances: */30 * * * *
Next sync: 12/6/24, 5:00 PM
Type /help to see available commands
```
#### Success Notification (when mode is `always`)
```
✅ Sync Successful
Server: My-Finances
Duration: 2156ms
Accounts Processed: 5
```
#### Failure Notification (when mode is `always` or `errors_only`)
```
❌ Sync Failed
Server: My-Finances
Duration: 543ms
Error: Connection timeout
Code: ETIMEDOUT
```
### Message Logging
All incoming Telegram messages are logged for auditing and troubleshooting:
```json
{
"timestamp": "2024-12-06T16:30:00.000Z",
"level": "INFO",
"component": "TelegramBot",
"message": "Telegram message received",
"from": "John Doe",
"username": "johndoe",
"chatId": "123456789",
"text": "/status"
}
```
### Security Considerations
1. **Chat ID Verification**: The bot only responds to messages from the configured `chatId`, ignoring messages from other users.
2. **Token Security**: Keep your bot token secret. Never commit it to version control. Consider using environment variables or Docker secrets.
3. **Private Bots**: By default, only you (the chat ID owner) can interact with the bot. For team use, consider:
- Creating a group chat and adding the bot
- Getting the group's chat ID
- All group members can then use commands
4. **Command Authorization**: All commands are available to the configured chat. Implement additional authorization if needed.
### Troubleshooting
#### Bot not responding to commands
**Check bot is running:**
```bash
docker logs actual-sync | grep "Telegram bot started"
```
**Verify configuration:**
```json
{
"telegram": {
"enabled": true, // Must be true
"botToken": "...", // Valid bot token
"chatId": "..." // Your chat ID as string
}
}
```
**Test with /ping:**
- If no response, check logs for errors
- Verify chat ID matches your Telegram user ID
- Restart the service: `docker-compose restart actual-sync`
#### Notifications not sent
**Check notification mode:**
```
/notify
```
If set to `never`, notifications are disabled. Change with:
```
/notify errors
```
**Check logs:**
```bash
docker logs actual-sync | grep "notifySync"
```
#### Wrong chat ID
**Symptoms:**
- Bot doesn't respond to your messages
- Logs show "Message from unauthorized chat"
**Fix:**
1. Get your correct chat ID using @userinfobot
2. Update `config.json` with the correct chat ID
3. Restart the service
### Best Practices
1. **Start with `errors_only`**: Reduces notification noise while staying informed of issues. Set it on `notifications` to apply it to every channel at once, or on a single channel to quieten just that one. Partial syncs still come through.
2. **Use `/status` regularly**: Quick health check without waiting for scheduled syncs.
3. **Monitor `/history`**: Track sync patterns and identify intermittent issues.
4. **Review `/errors`**: Investigate root causes of failures.
5. **Adjust notification mode**: Switch to `always` during troubleshooting, back to `errors` for normal operation.
6. **Keep bot token secure**: Use environment variables in production:
```bash
TELEGRAM_BOT_TOKEN=your_token_here
TELEGRAM_CHAT_ID=your_chat_id_here
```
7. **Test commands after updates**: Run `/ping` and `/status` after service restarts to verify bot functionality.
## Email Notifications
### Gmail Configuration
To use Gmail for email notifications:
1. **Enable 2-Factor Authentication** on your Google account
2. **Generate an App Password**:
- Go to Google Account → Security → 2-Step Verification → App passwords
- Generate a new app password for "Mail"
- Use this password in the `auth.pass` field
3. **Configure in config.json**:
```json
{
"notifications": {
"email": {
"enabled": true,
"host": "smtp.gmail.com",
"port": 587,
"secure": false,
"auth": {
"user": "your-email@gmail.com",
"pass": "your-16-char-app-password"
},
"from": "your-email@gmail.com",
"to": ["admin@example.com"]
}
}
}
```
### Other Email Providers
**Office 365 / Outlook.com:**
```json
{
"host": "smtp.office365.com",
"port": 587,
"secure": false
}
```
**SendGrid:**
```json
{
"host": "smtp.sendgrid.net",
"port": 587,
"secure": false,
"auth": {
"user": "apikey",
"pass": "your-sendgrid-api-key"
}
}
```
**Amazon SES:**
```json
{
"host": "email-smtp.us-east-1.amazonaws.com",
"port": 587,
"secure": false,
"auth": {
"user": "your-smtp-username",
"pass": "your-smtp-password"
}
}
```
### Email Content
Notification emails include:
- **Subject**: `[Actual Budget Sync] Error: [ServerName]`
- **Server name** and **timestamp**
- **Error message** and **error code**
- **Correlation ID** for tracking
- **Alert details**: Consecutive failures count and failure rate
- **Additional context**: Accounts processed, succeeded, failed, duration
- Both plain text and HTML formatted versions
## Webhook Notifications
### Slack Webhooks
1. **Create Incoming Webhook**:
- Go to your Slack workspace settings
- Navigate to Apps → Incoming Webhooks
- Add Configuration → Choose channel → Copy webhook URL
2. **Configure**:
```json
{
"webhooks": {
"slack": [
{
"name": "Production Alerts",
"url": "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXX"
}
]
}
}
```
3. **Notification Format**:
- Rich message with formatted blocks
- Server name, timestamp, error details
- Consecutive failures and failure rate
- Color-coded severity indicators
### Discord Webhooks
1. **Create Webhook**:
- Go to Server Settings → Integrations → Webhooks
- Create Webhook → Choose channel → Copy webhook URL
2. **Configure**:
```json
{
"webhooks": {
"discord": [
{
"name": "Sync Alerts",
"url": "https://discord.com/api/webhooks/123456789/abcdefghijklmnop"
}
]
}
}
```
3. **Notification Format**:
- Embedded message with red color coding
- Structured fields for easy reading
- Timestamp and correlation ID
### Microsoft Teams
There is no dedicated `teams` channel. Reach Microsoft Teams through the
**generic webhook**: create an Incoming Webhook in your Teams channel
(channel → Connectors → "Incoming Webhook" → copy URL) and point a
`webhooks.generic` entry at that URL.
```json
{
"webhooks": {
"generic": [
{
"name": "DevOps Alerts",
"url": "https://outlook.webhook.office.com/webhookb2/xxx@yyy/IncomingWebhook/zzz"
}
]
}
}
```
See [Generic webhooks](#webhook-settings) for the payload shape and options.
### Telegram Bot Notifications
1. **Create Telegram Bot**:
- Open Telegram and search for @BotFather
- Send `/newbot` command
- Follow instructions to create bot
- Copy the bot token (format: `123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11`)
2. **Get Chat ID**:
**For Personal Messages:**
- Start a chat with your bot
- Send any message to the bot
- Visit `https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getUpdates`
- Find your chat ID in the response (under `message.chat.id`)
**For Group Chats:**
- Add the bot to your group
- Send a message in the group
- Visit `https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getUpdates`
- Find the group chat ID (negative number like `-123456789`)
**For Channels:**
- Add the bot as admin to your channel
- Use the channel username with @ (e.g., `@mychannel`)
- Or use the numeric channel ID (starts with `-100`)
3. **Configure**:
```json
{
"webhooks": {
"telegram": [
{
"name": "Production Alerts",
"botToken": "123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11",
"chatId": "123456789"
},
{
"name": "Team Group",
"botToken": "123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11",
"chatId": "-987654321"
}
]
}
}
```
4. **Notification Format**:
- MarkdownV2 formatted message
- Clean, readable text layout
- Emoji indicators (🚨 for errors)
- Server name, timestamp, error details
- Consecutive failures and failure rate
- Special characters automatically escaped
5. **Tips**:
- One bot can send to multiple chats (personal, groups, channels)
- Create separate bots for different environments
- Test with `/start` command to ensure bot is working
- Bot needs admin rights to post in channels
## Message templates
> This section documents the template engine itself (`src/lib/templateRenderer.js` and `src/lib/channelEscape.js`). A server's `scheduleAlerts.alerts[].templates` (or block-level `scheduleAlerts.templates`) is the first configuration surface that uses it, to customize missing-payment alert wording; see [SCHEDULE_ALERTS.md](./SCHEDULE_ALERTS.md). Every notification described earlier in this document keeps its own current hard-coded text.
### Syntax
Templates use [Handlebars](https://handlebarsjs.com/) `{{ variable }}` syntax, the same engine and placeholder style Actual Budget uses for rule action templates, so Actual users already know it:
```handlebars
{{name}}'s payment of {{amount}} is due {{deadline}}.
```
A template can also use a small set of helpers and block helpers:
```handlebars
{{upper name}} owes {{default amount "an unknown amount"}}.
{{#if (eq status "overdue")}}This payment is overdue.{{else}}This payment is on time.{{/if}}
{{#each items as |item|}}- {{item.name}}: {{item.amount}}
{{/each}}
```
### Variable and helper whitelist
A template can only reference:
- a variable the consumer explicitly documented (each notification type defines its own list, for example `name`, `amount`, `deadline`), used bare (`{{name}}`), never through a dotted path (`{{name.length}}` fails validation even if `name` is documented, since a variable's shape is not part of its contract);
- `this`, or one of exactly four `@data` built-ins: `@index`, `@key`, `@first`, `@last`. `@root` (the whole render context) and any other `@name` are rejected, since `@root` would bypass the variable whitelist entirely;
- a block parameter introduced by `#each`/`#with` (for example the `item` in `{{#each items as |item|}}`), which may use a dotted path (`{{item.name}}`) since its shape comes from the caller's own data, but never through `__proto__`, `constructor`, or `prototype`;
- one of four helpers: `eq`, `default`, `upper`, `lower`;
- one of four block helpers: `if`, `unless`, `each`, `with`.
Everything else, including `lookup`, `log`, any other custom helper name, a literal used where a name is expected (`{{"&"}}`, `{{1}}`), partials (`{{> something}}`), and decorators (`{{*foo}}`, `{{#*inline}}`), fails validation at startup. This is checked by walking the template's parsed syntax tree rather than by rendering a sample, so an unknown name is caught even when it only appears inside a helper argument, inside a false `#if` branch, or inside an `#each` body that happens to be empty for a given sample. Validation failures throw with the template key, the offending token, and the line number, so a typo such as `{{nmae}}` is reported precisely rather than silently rendering blank.
Because the render context only ever contains the variables a consumer explicitly whitelisted, a template can never reach configuration secrets such as a Telegram bot token or an SMTP password, no matter how it is written.
### Per-channel output
Escaping happens once, at output time, per channel, never before a helper runs. `{{upper payee}}` on `Bob & Co` always produces `BOB & CO` from the helper itself; whether that ends up as `BOB & CO` or `BOB & CO` depends only on the channel it is rendered for:
| Channel | Compiled with | Post-processing | Notes |
|---|---|---|---|
| Telegram | HTML escaping on | `truncateTelegramHtml` (4096 char limit) | sent with `parse_mode: HTML` |
| Email (HTML part) | HTML escaping on | none | |
| Email (text part) | raw, no escaping | none | plain text |
| ntfy | raw, no escaping | none | plain text |
| Slack | raw, no escaping | `escapeSlack` on the whole rendered string, then capped at 40000 characters (Slack's hard limit; Slack truncates anything longer) with a trailing `…` | plain-text semantics; Slack's own markup is not available to a template author |
| Discord | raw, no escaping | `escapeDiscordMarkdown` on the whole rendered string, then capped at 2000 characters (Discord rejects more with a 400) with a trailing `…` | plain-text semantics; also sent with `allowed_mentions: { parse: [] }` so a rendered value can never ping `@everyone`, `@here`, or a user/role id |
| Generic webhook | raw, no escaping, for the `text` field | none | raw variable values are also sent as separate structured JSON fields; a field named `text` never overrides the rendered `text` |
Truncation is a safety net for a single oversized message. A schedule-alert digest never relies on it: it is split to fit the channel's cap (see [Digest mode](SCHEDULE_ALERTS.md#digest-mode)). Truncation counts Unicode code points, never splits a surrogate pair, and the ellipsis counts toward the cap. Slack escaping covers only `&`, `<` and `>`, so mrkdwn characters (`*`, `_`, `~`, `` ` ``) inside a rendered value such as a payee name still format on Slack. This matches the spec; a payee called `*Rent*` shows as bold.
The non-templated Discord notifications (sync results, startup, dashboard test) are also sent with `allowed_mentions: { parse: [] }`, so account, payee and error text in an embed can never ping `@everyone`, `@here`, or a user/role id.
`upper` and `lower` render a missing (`undefined`) or `null` value as an empty string, not the words `UNDEFINED` or `NULL`.
### Telegram markup rules
Telegram rejects a message outright (HTTP 400) if its literal text contains a bare `&` or `<`, or a tag other than `b`, `i`, `u`, `s`, `code`, `pre`, or `a`. A template's literal content, the parts the template author typed directly rather than a rendered variable, is checked against this rule at startup, so a template like:
```handlebars
Rent & fees due {{deadline}}
```
fails validation immediately, naming the `&` and its line, instead of failing the first time it is sent. To write a literal ampersand, `<` or `>` use one of the four entities Telegram's HTML mode defines: `&`, `<`, `>` and `"`. Any other entity (` `, `&`) is rejected.
Literal-display caveat: those entities are only decoded by Telegram (and the HTML part of an email). Every other channel renders the template's literal text as typed, so `Rent & fees` shows as `Rent & fees` on ntfy, in the plain-text email part, in the plain-text Telegram fallback, and on the generic webhook, and Slack and Discord escape it a second time. If a template set is rendered for non-Telegram channels, avoid a literal `&` in the wording rather than writing the entity.
Allowed tags must also be balanced. `<b>{{name}}` (never closed), `x</b>` (never opened) and `<b><i>x</b></i>` (crossed) fail validation with `telegram_markup`, naming the tag. A tag pair may span mustaches (`<b>{{name}}</b>`), but each block body (`{{#if}}...{{/if}}`, including `{{else}}`) is its own scope: a tag opened outside a block cannot be closed inside it. A template using only balanced allowed tags passes:
```handlebars
<b>{{name}}</b> owes {{amount}}
```
If Telegram still rejects a rendered message at send time (for example because a variable's value produced markup the startup check could not foresee), the notification service retries exactly once, sending the plain-text rendering with no `parse_mode`, and logs a warning. That fallback text is also truncated to Telegram's 4096 code-point limit, counting whole Unicode code points rather than UTF-16 units so the cut never splits a surrogate pair. Any other delivery failure is not retried.
### Links in templates
`<a href="{{url}}">` is not supported for now, and fails `telegram_markup` because the tag spans a mustache. This is a deliberate decision (#294, L4): no template consumer needs a link today, and supporting one safely means deciding how to escape an attribute value per channel, how to validate the scheme (a `javascript:` or other unexpected scheme must not be reachable), and what Slack, Discord, ntfy and plain-text email should show instead of an anchor. Link text and URLs can still appear as plain variable values. Revisit when a consumer needs it.
### Validation errors
`validateTemplate` throws a `TemplateValidationError` carrying:
- `key`: the template's identifier, so an error in a multi-template set names the right one;
- `token`: the offending variable name, helper name, or character;
- `line`: the 1-based line number inside the template source (for a mismatched block such as `{{#if x}}...{{/unless}}` this is the line of the closing tag);
- `reason`: one of `unknown_variable`, `unknown_helper`, `parse_error`, or `telegram_markup`.
### Locale loading
A consumer formats money and dates before handing values to a template, rather than doing it inside the template itself: for example `expected_amount: "80,00 €"` alongside a raw twin `expected_amount_raw: 80`, using `Intl.NumberFormat` for currency and `moment-timezone` for dates. A non-English locale's data is loaded on demand; an unrecognized locale code is a startup error rather than a silent fallback to English formatting.
## Notification Thresholds
The system uses two threshold mechanisms to determine when to send notifications:
### 1. Consecutive Failures Threshold
Triggers notification after N consecutive sync failures for a server.
**Example**: With `consecutiveFailures: 3`:
```
Server1: ✓ ✗ ✗ ✗ → Notification sent (3 consecutive failures)
Server1: ✗ ✓ ✗ ✗ → No notification (success breaks the chain)
```
**Use case**: Detect persistent issues that affect every sync attempt.
### 2. Failure Rate Threshold
Triggers notification when failure rate exceeds threshold over a time period.
**Example**: With `failureRate: 0.5` and `ratePeriodMinutes: 60`:
```
Last hour: 6 syncs total, 4 failed → 66% failure rate → Notification sent
Last hour: 6 syncs total, 2 failed → 33% failure rate → No notification
```
**Use case**: Detect intermittent issues that don't fail every time but indicate a problem.
### Threshold Logic
A notification is sent if **either** threshold is exceeded:
- Consecutive failures ≥ threshold **OR**
- Failure rate ≥ threshold
This ensures both persistent and intermittent issues are caught.
## Rate Limiting
Rate limiting prevents notification spam while ensuring important alerts are delivered.
### Minimum Interval
Ensures at least N minutes between notifications, regardless of other conditions.
**Example**: With `minIntervalMinutes: 15`:
```
12:00 PM: Notification sent
12:10 PM: Threshold exceeded → No notification (only 10 min since last)
12:16 PM: Threshold exceeded → Notification sent (16 min since last)
```
### Maximum Per Hour
Caps total notifications sent within any 60-minute rolling window.
**Example**: With `maxPerHour: 4`:
```
12:00 PM: Notification 1 sent
12:20 PM: Notification 2 sent
12:40 PM: Notification 3 sent
12:50 PM: Notification 4 sent
12:55 PM: Threshold exceeded → No notification (4 already sent in last hour)
```
### Rate Limit Strategy
Both limits work together:
1. Check minimum interval first (fast check)
2. Check max per hour if interval passed
3. Send notification only if both allow
This prevents:
- **Notification storms** from rapid successive failures
- **Alert fatigue** from too many notifications
- **Missing critical issues** by allowing reasonable alert frequency
## Use Cases
### Production Monitoring
Monitor critical sync operations with immediate alerts:
```json
{
"thresholds": {
"consecutiveFailures": 2,
"failureRate": 0.4,
"ratePeriodMinutes": 30
},
"rateLimit": {
"minIntervalMinutes": 10,
"maxPerHour": 6
}
}
```
- Lower thresholds for faster detection
- Higher rate limits for timely alerts
- Multiple channels (email + Slack)
### Development Environment
Less aggressive monitoring for dev/test systems:
```json
{
"thresholds": {
"consecutiveFailures": 5,
"failureRate": 0.7,
"ratePeriodMinutes": 120
},
"rateLimit": {
"minIntervalMinutes": 30,
"maxPerHour": 2
}
}
```
- Higher thresholds to reduce noise
- Longer rate period for trend analysis
- Stricter rate limits to prevent spam
### Team Collaboration
Route notifications to appropriate teams:
```json
{
"email": {
"enabled": true,
"from": "actual-sync@example.com",
"to": ["on-call@example.com"]
},
"webhooks": {
"slack": [
{
"name": "Ops Team",
"url": "https://hooks.slack.com/services/ops-channel"
}
],
"telegram": [
{
"name": "Mobile Alerts",
"botToken": "123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11",
"chatId": "-987654321"
}
]
}
}
```
### Silent Operation
Mute notifications for maintenance or testing **without dismantling your configuration**. Set `notifyOnSuccess: "never"` globally — every channel goes quiet, credentials and webhook URLs stay in place, and dashboard test notifications still work so you can verify the channels before switching back:
```json
{
"notifications": {
"notifyOnSuccess": "never"
}
}
```
To mute one channel and leave the rest alone, set it on that channel instead:
```json
{
"notifications": {
"email": {
"enabled": true,
"from": "actual-sync@example.com",
"to": ["admin@example.com"],
"notifyOnSuccess": "never"
}
}
}
```
(An **enabled** email channel must still carry `from` and a non-empty `to` — muting it does not exempt it from validation.)
Prefer this to setting `enabled: false` and emptying the webhook arrays. That older approach also silences genuine failures, loses the configuration, and has to be reassembled from scratch afterwards. Reach for `enabled: false` only when you want the channel gone entirely.
## Troubleshooting
### Email Not Sending
**Problem**: Notifications enabled but emails not received
**Solutions**:
1. **Check SMTP credentials**:
- Verify username and password
- For Gmail, use app-specific password
- Check 2FA is enabled for app passwords
2. **Verify SMTP settings**:
```bash
# Test SMTP connection
telnet smtp.gmail.com 587
```
3. **Check logs**:
```bash
# Look for email errors
grep -i "email" logs/*.log
```
4. **Check spam folder**: Emails might be filtered
5. **Verify email addresses**: Check `from` and `to` addresses are valid
### Webhook Not Sending
**Problem**: Webhook configured but notifications not appearing
**Solutions**:
1. **Test webhook URL manually**:
```bash
curl -X POST YOUR_WEBHOOK_URL \
-H "Content-Type: application/json" \
-d '{"text": "Test notification"}'
```
2. **Check webhook URL format**:
- Slack: `https://hooks.slack.com/services/...`
- Discord: `https://discord.com/api/webhooks/...`
- Teams (via `generic`): `https://outlook.webhook.office.com/...`
3. **Verify webhook is active**: Recreate if deleted/expired
4. **Check channel permissions**: Webhook must have post permission
### No Notifications Despite Failures
**Problem**: Syncs failing but no notifications sent
**Solutions**:
1. **Check the channel is not muted**: `notifyOnSuccess: "never"` turns a channel off entirely, failures included — globally, on the channel, or on an individual webhook entry. For Telegram, check `/notify` too: the mode is settable at runtime, and persists across restarts unless config sets it explicitly.
```bash
grep -i "channels muted\|notifyOnSuccess" logs/*.log
```
2. **Check thresholds**: Failures might not exceed configured thresholds
```bash
# View sync history to analyze failure patterns
npm run history --server YourServer --days 1
```
3. **Check rate limiting**: Previous notifications might be blocking new ones
3. **Verify configuration loaded**:
```bash
npm run validate-config
```
4. **Check service logs**:
```bash
# Look for notification decisions
grep "threshold\|rate limit" logs/*.log
```
### Too Many Notifications
**Problem**: Receiving excessive notifications
**First, work out what is noisy.** Thresholds and rate limiting apply to **failures only** — if the noise is routine *successful* syncs, changing them will do nothing.
**Solutions**:
1. **Noisy on successful syncs** — this is the common case on a frequent schedule. Switch the channel to `errors_only`:
```json
{
"notifications": {
"notifyOnSuccess": "errors_only"
}
}
```
Set it per channel to keep a low-traffic channel verbose while quietening a busy one. See [Notification mode](#notification-mode-notifyonsuccess).
2. **Noisy on partial syncs** — a partial counts as an error, so `errors_only` still reports it, every time and without rate limiting. That is usually a genuine standing problem (e.g. one account whose bank link is broken). Fix the account, or set that channel to `never` if you have accepted it.
3. **Noisy on failures** — raise the thresholds:
```json
{
"thresholds": {
"consecutiveFailures": 5,
"failureRate": 0.7
}
}
```
4. **Still noisy on failures** — increase the rate limit intervals:
```json
{
"rateLimit": {
"minIntervalMinutes": 30,
"maxPerHour": 2
}
}
```
5. **Fix underlying sync issues** rather than suppressing notifications
## Best Practices
### 1. Start Conservative
Begin with higher thresholds and stricter rate limits, then adjust based on experience:
```json
{
"thresholds": {
"consecutiveFailures": 5,
"failureRate": 0.6
},
"rateLimit": {
"minIntervalMinutes": 30,
"maxPerHour": 2
}
}
```
### 2. Use Multiple Channels
- **Email**: On-call teams, important alerts
- **Slack** (or Teams via a generic webhook): Real-time team collaboration
- **Discord**: Community or informal monitoring
### 3. Correlate with Sync History
Use correlation IDs to link notifications to detailed sync history:
```bash
# After receiving notification with correlation ID
npm run history | grep "correlation-id-from-email"
```
### 4. Test Notification Configuration
Before deploying, test your notification setup:
1. Set very low thresholds temporarily
2. Force a sync failure
3. Verify notifications arrive
4. Reset thresholds to production values
### 5. Monitor Notification Statistics
Periodically review notification frequency:
- Too many notifications → Increase thresholds or rate limits
- Too few notifications → Lower thresholds
- No notifications during known issues → Check configuration
### 6. Secure Credentials
- **Never commit** SMTP passwords or webhook URLs to version control
- Use **environment variables** for sensitive values
- **Rotate credentials** periodically
- Use **app-specific passwords** for email services
### 7. Document Your Configuration
Keep documentation of:
- Why specific thresholds were chosen
- Who receives notifications and how to update
- Webhook channel owners and renewal dates
- On-call rotation for email recipients
### 8. Regular Maintenance
- **Test email/webhooks** quarterly
- **Review thresholds** based on actual failure patterns
- **Update recipient lists** when team changes
- **Archive old notification logs** to save space
## Integration with Other Features
### Sync History Integration
Notification service tracks sync results independently but integrates with sync history:
- **Correlation IDs**: Link notifications to detailed history records
- **Query history**: Use sync history to analyze notification patterns
- **Failure trends**: Combine notification stats with historical data
### Health Check Integration
Notification service complements the health check endpoints:
- **Health checks**: Passive monitoring (query when needed)
- **Notifications**: Active monitoring (alerts pushed to you)
- **Combined**: Health checks for status, notifications for problems
### Structured Logging Integration
All notification activities are logged with:
- **Correlation IDs**: Track notification decisions
- **Debug logs**: Threshold checks, rate limit decisions
- **Error logs**: Notification failures, webhook errors
## Future Enhancements
Potential future improvements:
1. **Notification History**: Track sent notifications in database
2. **Escalation Rules**: Different channels for different severity
3. **Quiet Hours**: Suppress non-critical notifications during off-hours
4. **Custom Templates**: User-defined email/webhook message formats
5. **Aggregated Notifications**: Batch multiple failures into digest
6. **SMS Integration**: Twilio/SNS for critical alerts
7. **PagerDuty Integration**: Incident management system integration
8. **Notification Dashboard**: Web UI to view notification status
9. **Smart Thresholds**: ML-based adaptive threshold adjustment
10. **Notification Replay**: Manually trigger notification for testing
## Summary
The Error Notification System provides:
✅ **Multiple delivery channels** (Email, Slack, Discord, Telegram)
✅ **Smart thresholds** (consecutive failures and failure rates)
✅ **Rate limiting** (prevent notification spam)
✅ **Rich context** (errors, stats, correlation IDs)
✅ **Flexible configuration** (adapt to your needs)
✅ **Production-ready** (34 comprehensive tests, graceful error handling)
Configure it once, and stay informed about sync issues that matter without being overwhelmed by noise.