docs/SETUP_PRODUCTION.md

Rama main · réplica de lectura del repositorio original.

# AgentRQ Production Deployment

> **For deploying AgentRQ in production with high availability and security.**

## Pre-Deployment Checklist

- [ ] Public domain registered and DNS configured
- [ ] Google OAuth2 credentials created
- [ ] PostgreSQL database (external or Docker Compose service)
- [ ] Backup strategy planned
- [ ] SSL/TLS ready (Let's Encrypt automatic or manual)

## Production Setup

### 1. Run Setup Script for Production

```bash
bash scripts/setup.sh production
```

Will prompt you for:
- Domain name
- Email for Let's Encrypt
- Google OAuth credentials
- PostgreSQL connection details (optional)
- Slack integration (optional)

### 2. Enable PostgreSQL (Recommended)

SQLite, the default, runs only on a single host: its database is a file on
that host's disk, in WAL mode, which can't be shared between machines or put on
a network volume. Use it for one instance only. To run more than one instance,
use PostgreSQL.

Edit `.env`:
```env
AGENTRQ_POSTGRES_ENABLED=true
AGENTRQ_SQLITE_ENABLED=false
AGENTRQ_POSTGRES_HOST=postgres  # or external host
AGENTRQ_POSTGRES_PORT=5432
AGENTRQ_POSTGRES_USER=postgres
AGENTRQ_POSTGRES_PASSWORD=<strong-password>
AGENTRQ_POSTGRES_DBNAME=agentrq
```

To use the included PostgreSQL service:
```bash
docker-compose --profile postgres up -d
```

### 3. Configure TLS/SSL

Edit `.env`:
```env
AGENTRQ_SSL_ENABLED=true
AGENTRQ_SSL_LETSENCRYPT_EMAIL=admin@your-domain.com
AGENTRQ_DOMAIN=your-domain.com
AGENTRQ_BASE_URL=https://your-domain.com
```

**Important**: Ports 80 and 443 must be publicly accessible for Let's Encrypt to work.

**Optional: share certificates through S3.** Set a private bucket of its own, separate from the `AGENTRQ_S3_*` one files are stored in, and a redeploy or another instance reuses the certificate instead of obtaining a new one. Every day, plus a random 1 to 5 minutes so instances do not check at once, the server serves the certificate on disk while it is valid for 30 more days and covers every domain; otherwise it takes the bucket's, if that one is; otherwise it obtains a new one and saves it to disk and to the bucket, at `certs/<domain>/combined.crt` and `combined.key`. The key is private: never allow public reads of this bucket.

| Variable | Default | Description |
|---|---|---|
| `AGENTRQ_SSL_S3_ENABLED` | `false` | Share certificates through the bucket below. |
| `AGENTRQ_SSL_S3_ENDPOINT` | | The S3 endpoint URL. Addressed path-style. |
| `AGENTRQ_SSL_S3_ACCESS_KEY` | | Access key id. |
| `AGENTRQ_SSL_S3_SECRET_ACCESS_KEY` | | Secret access key. |
| `AGENTRQ_SSL_S3_REGION` | `us-east-1` | Bucket region. |
| `AGENTRQ_SSL_S3_BUCKET` | | Bucket name, required when enabled. It must already exist. |

### 4. Start Services

```bash
# Create necessary directories
mkdir -p storage certs
chmod 0777 storage certs

# Start with PostgreSQL
docker-compose --profile postgres up -d

# Or without PostgreSQL (if using external)
docker-compose up -d
```

### 5. Verify TLS Certificate

```bash
# Check certificate was created
ls -la certs/

# View logs
docker-compose logs -f agentrq | grep -i ssl
```

## Security Hardening

### 1. Disable Root Login

Make sure in `.env`:
```env
AGENTRQ_AUTH_ROOT_LOGIN_ENABLED=false
```

### 2. Where a Sign-in May Return To

`/login?redirect_url=<address>` sends the person back to that address once they
have signed in. Addresses on this server always work. Other hosts must be
listed, comma separated, and must match exactly; any other address goes to the
app instead. A malformed entry (with `https://` or a path) stops startup.
```env
AGENTRQ_AUTH_ALLOWED_REDIRECT_DOMAINS=app.example.com,docs.example.com
```

### 3. Strong Secrets

Verify secrets are strong (32+ characters):
```bash
openssl rand -base64 32  # JWT secret
openssl rand -hex 16     # Token key (32 bytes)
```

### 4. Backup Strategy

```bash
# Backup PostgreSQL
docker-compose exec postgres pg_dump -U postgres agentrq > backup.sql

# Backup storage (attachments)
tar -czf storage-backup.tar.gz storage/

# Backup .env (securely)
cp .env .env.backup
```

### 5. Network Security

- Place behind a reverse proxy (Nginx, Traefik, Cloudflare)
- Enable rate limiting (default: 60 req/IP, 100 req/user per minute)
- Enable DDoS protection (default: 20 req/sec)
- Behind a proxy, set `AGENTRQ_TRUSTED_PROXIES` (see [Client addresses](#client-addresses))
- Use firewall rules to restrict non-HTTP(S) traffic

### 6. Regular Updates

```bash
# Pull latest image
docker-compose pull

# Restart services
docker-compose up -d

# View changelog: https://github.com/agentrq/agentrq/releases
```

## Monitoring

### View Logs

```bash
# Real-time
docker-compose logs -f agentrq

# Last 100 lines
docker-compose logs --tail 100 agentrq

# Filter by level
docker-compose logs agentrq | grep ERROR
```

### Health Probes

Every instance answers two unauthenticated probes on its own port, whatever
the `Host` header:

- `GET /healthz` — liveness: `200 ok` while the process serves HTTP. It checks
  nothing else, so a slow database never gets an instance restarted.
- `GET /readyz` — readiness: `200 ok` when the database answers a ping within
  2 seconds, `503` otherwise, and `503 draining` from the moment shutdown starts.

When several instances sit behind a load balancer, set
`AGENTRQ_APP_DRAIN_DELAY` (e.g. `10s`, longer than the probe interval): on
SIGTERM the instance fails `/readyz` but keeps serving for that long, so the
balancer moves traffic away before the listener closes. Allow for it in the
orchestrator's termination grace period.

After the delay, live-update and agent event streams are ended, so their
clients reconnect to another instance at once. Other requests get up to 15
seconds to finish, and emails already queued are sent, before the process exits.

### Database Health

```bash
# PostgreSQL
docker-compose exec postgres pg_isready -U postgres

# SQLite (if used)
sqlite3 storage/agentrq.db "SELECT COUNT(*) FROM tasks;"
```

### Storage Usage

```bash
# Check attachment storage
du -sh storage/

# Check database size (PostgreSQL)
docker-compose exec postgres psql -U postgres -c "SELECT pg_size_pretty(pg_database_size('agentrq'));"
```

### Attachment Auto-Cleanup

AgentRQ automatically deletes attachment files older than a configurable retention period. The cleanup runs daily at midnight UTC and skips database files.

| Variable | Default | Description |
|---|---|---|
| `AGENTRQ_APP_ATTACHMENT_RETENTION` | `7d` | Retention period for attachment files. Accepts day values (`7d`, `30d`) or Go duration strings (`168h`). |

Set in `.env`:
```env
AGENTRQ_APP_ATTACHMENT_RETENTION=30d
```

Skill files are not attachments: cleanup never deletes them.

### Skill and Attachment Storage

Skill files, attachments and the [Pano](PANO.md)'s files are kept under `AGENTRQ_STORAGE_DIR` by default (`/storage` in docker-compose, `./_storage` otherwise), in `skills/`, `artifacts/` (where attachments live) and `pano/`. Set `AGENTRQ_SKILLS_STORAGE=s3`, `AGENTRQ_ARTIFACTS_STORAGE=s3` or `AGENTRQ_PANO_STORAGE=s3` to keep any of them in an S3-compatible bucket instead (AWS S3, MinIO, Cloudflare R2…), under the same prefixes; the server refuses to start on any value other than `local` or `s3`.

Every file is kept at a key that names its workspace and task, or its account and skill, and ends in a fresh id:

- attachments: `artifacts/w-<workspace id>/<task id>/<id>`
- skill files: `skills/u-<account id>/skill-<skill id>/<id>`
- Pano docs' files: `pano/u-<account id>/pano-<doc id>/<id>` (the markdown itself is in the database)

**Each file has a public link, and the link is the only credential needed to read it.** Without knowing both the workspace id and the task id (or the account id and the skill id), a link is nearly impossible to guess. Ids are time-ordered, though, so someone who holds one link has both, and can work out the links of the other files on that task or skill: share a link only with someone who may see everything there. The web app's Download button, the agents' `task.getAttachment` (by default) and `agentrq-ws attachment get` all use it. Local files are served by the server itself at `<AGENTRQ_BASE_URL>/storage/artifacts/...` and `/storage/skills/...` and `/storage/pano/...`, with no sign-in; files in S3 are read from the bucket, at `<AGENTRQ_S3_PUBLIC_URL>/artifacts/...`, `/skills/...` and `/pano/...`.

| Variable | Default | Description |
|---|---|---|
| `AGENTRQ_STORAGE_DIR` | `./_storage` | Writable directory for local files. Its root also holds the SQLite database and older attachments. |
| `AGENTRQ_SKILLS_STORAGE` | `local` | `local` or `s3`. |
| `AGENTRQ_ARTIFACTS_STORAGE` | `local` | `local` or `s3`, for attachments. |
| `AGENTRQ_PANO_STORAGE` | `local` | `local` or `s3`, for the files of the Pano's docs. |
| `AGENTRQ_PANO_MAX_TTL_DAYS` | `365` | Most days a Pano doc may live after its last save; a doc asks for 30 by default. A whole number; 0 means the default. |
| `AGENTRQ_S3_ENDPOINT` | | The S3 endpoint URL, e.g. `https://s3.us-east-1.amazonaws.com`. Addressed path-style. |
| `AGENTRQ_S3_ACCESS_KEY` | | Access key id. |
| `AGENTRQ_S3_SECRET_ACCESS_KEY` | | Secret access key. |
| `AGENTRQ_S3_REGION` | `us-east-1` | Bucket region. |
| `AGENTRQ_S3_BUCKET` | | Bucket name. It must already exist. |
| `AGENTRQ_S3_PUBLIC_URL` | `<endpoint>/<bucket>` | Base of public links to the bucket, e.g. a CDN in front of it. |

**Serving local files.** The server answers `/storage/...` only for a path of exactly the shape above — 11-character base62 ids, nothing else — and reads only from `skills/`, `artifacts/` and `pano/`, never following a link out of them. Do not serve the storage directory any other way (a static file handler, a proxy `alias`): its root holds the database. A file that could run in a browser, such as HTML or SVG, is sent as a download.

**Serving from S3.** No object ACL is set, since new AWS buckets and several S3-compatible stores refuse them, so allow anonymous reads of those prefixes, and nothing else, with a bucket policy (AWS example):

```json
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": "*",
    "Action": "s3:GetObject",
    "Resource": ["arn:aws:s3:::<bucket>/artifacts/*", "arn:aws:s3:::<bucket>/skills/*", "arn:aws:s3:::<bucket>/pano/*"]
  }]
}
```

An attachment's upload type is kept only when a browser cannot run it; anything else is stored as `application/octet-stream`.

**Retention.** The daily cleanup deletes local attachments, in `artifacts/` and flat in the storage directory where they were kept before, once they pass `AGENTRQ_APP_ATTACHMENT_RETENTION`. With S3, add a lifecycle rule that expires `artifacts/` after the same period. Pano docs need no rule: the server deletes each one, with its files, from either store within the hour after it expires.

**Switching** does not move files already saved. Attachments kept flat in the storage directory before this layout are still found there, whichever store is in use, but have no public link. Copy skills' `w-*` directories from `<storage dir>/skills/` to `skills/` in the bucket, or re-import them.

### Memory and Skill Limits

| Variable | Default | Description |
|---|---|---|
| `AGENTRQ_MEMORY_MAX_FILE_SIZE` | `16384` | Most bytes one workspace memory may hold; at most `131072`. |
| `AGENTRQ_SKILLS_MAX_FILE_SIZE` | `98304` | Most bytes a skill's `SKILL.md` may hold. Its other files are capped at 64 KiB. |
| `AGENTRQ_SKILLS_MAX_FILES` | `256` | Most files in one skill, `SKILL.md` included. |

Values are whole numbers. The server refuses to start on anything else, or on a memory size above `131072`.

### MCP

| Variable | Default | Description |
|---|---|---|
| `AGENTRQ_MCP_OPENAI_APPS_CHALLENGE` | (empty) | The domain verification token OpenAI issues when you submit the MCP server as a plugin, served as plain text at `/.well-known/openai-apps-challenge` on every host. Empty answers 404. |

### Native ACP

| Variable | Default | Description |
|---|---|---|
| `AGENTRQ_ACP_V1_ENABLED` | `true` | Serve `/api/acp/v1`, the only way acp-gateway connects. Off, the endpoint answers 404 and the gateway cannot run. |

The value is `true` or `false`. The server refuses to start on anything else.

`/api/acp/v1` is a WebSocket, so a reverse proxy in front of the server must:

- pass `Upgrade` and `Connection` for `/api/acp/`. The `location /` block below already does.
- keep its read timeout above 25 seconds, since the server pings an idle socket every 25 seconds.
- with more than one instance, send a gateway back to the same instance. A dropped connection resumes without loss only on the instance that holds it. On another instance the gateway starts over with a fresh connection, and its agent restarts.

## Reverse Proxy Setup (Nginx Example)

```nginx
server {
    listen 80;
    server_name your-domain.com;

    location / {
        proxy_pass http://localhost:2026;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Increase timeouts for long-running connections
        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
    }
}
```

### Client addresses

The DDoS guard, the rate limiter and the admin sign-in throttle count each client by address. The server believes a proxy's client-address header only from the proxies in `AGENTRQ_TRUSTED_PROXIES` (comma-separated CIDRs or addresses). It is empty by default: no header is believed, and each client is the address that connected. That is right when clients connect to the server directly, and a client can never choose its own address by sending a header.

Behind a proxy, list the address the proxy connects from, or every client behind it shares one limit. The server logs `no proxy is trusted` once when it sees that. A blocked client then blocks everybody.

| Proxy | `AGENTRQ_TRUSTED_PROXIES` | `AGENTRQ_CLIENT_IP_HEADER` |
|---|---|---|
| nginx on the same host, server not in Docker | `127.0.0.1,::1` | `X-Real-IP` |
| nginx on the Docker host, server in docker-compose | `172.16.0.0/12` (Docker's bridge networks) | `X-Real-IP` |
| Cloudflare, proxied | [Cloudflare's ranges](https://www.cloudflare.com/ips/) | `CF-Connecting-IP` |
| AWS ALB, Caddy, Traefik | the proxy's subnet | `X-Forwarded-For` |

`X-Forwarded-For` is read from the right: the client is the first entry that is not a trusted proxy, since the client writes the leftmost entries itself.

With no proxy, make sure the server sees the real client. Rootless Docker, `userland-proxy`, or `"iptables": false` show every client as Docker's gateway, so all of them share one limit. Use `network_mode: host`, or put a proxy in front.

## Troubleshooting

See [docs/SETUP_TROUBLESHOOTING.md](./SETUP_TROUBLESHOOTING.md)