docs/s3.md

Rama master · réplica de lectura del repositorio original.

[Table of contents](README.md#table-of-contents)

# S3 Storage Backend

cozy-stack supports S3-compatible object storage as a file system backend,
alongside the existing local filesystem (afero) and OpenStack Swift backends.
It supports S3-compatible providers such as OVH, MinIO and Scaleway.

## Configuration

Set `fs.url` to the default S3 connection and configure destinations in
`fs.s3.buckets`. Each entry contains a bucket `name` and an optional `url`.
Bucket names are never generated from prefixes or organization IDs.

```yaml
fs:
  url: s3://s3.example.net?access_key=ACCESS&secret_key=SECRET&region=gra
  s3:
    buckets:
      default:
        name: company-storage
```

This stores everything in one bucket, with separate object paths for each
storage type. All organizations share the files bucket; each instance has
its own path beneath `files/`.

| URL parameter | Description | Default |
|---------------|-------------|---------|
| `access_key` | S3 access key ID | — |
| `secret_key` | S3 secret access key | — |
| `region` | Region for requests and bucket creation | SDK discovery/default |
| `use_ssl` | Use HTTPS | `true` |

URL-encode credentials containing reserved query characters. Specify `region`
in every connection URL to avoid location discovery requests.

### Separate or mixed buckets

Configure a bucket name and connection together for each storage type:

```yaml
fs:
  url: s3://s3.example.net?access_key=DEFAULT_KEY&secret_key=DEFAULT_SECRET&region=gra
  s3:
    auto_create_buckets: false
    buckets:
      default:
        name: company-storage
      files:
        name: company-files
        url: s3://files.example.net?access_key=FILES_KEY&secret_key=FILES_SECRET&region=rbx
      exports:
        name: company-backups
```

The supported types are `files`, `apps_web`, `apps_konnectors`, `assets`,
`previews`, and `exports`. Types without an entry use the complete `default`
entry. To use only separate destinations, omit `default` and specify all six
types. Several types can use the same bucket name and URL.

Every entry requires `name`. Its optional `url` supplies the endpoint,
credentials, region and TLS setting; omitting it uses `fs.url`. An explicit
entry does not inherit fields from `default`. Entries with the same URL reuse
one connection, and each bucket on that connection is initialized once.
The same bucket name can also be used on different connections.
Copies between different connections stream through the stack.

Invalid names, unknown storage types, missing destinations and invalid
connection URLs cause startup errors. Configuration errors do not print
connection credentials.

### Bucket provisioning

`auto_create_buckets` defaults to `true`: startup creates each distinct bucket
once, using its connection's region. Existing buckets are accepted. Credentials
must allow bucket creation; bucket enumeration is not used.

With `auto_create_buckets: false`, create the configured buckets before starting
the stack. Startup checks each distinct bucket with `HeadBucket`, without listing
or creating buckets. Missing or inaccessible buckets fail startup with a
bucket-specific error. All startup bucket operations have a 30-second timeout.
Loading or initializing an instance does not probe S3.

Credentials must allow `HeadBucket` on each configured bucket and the object
operations used by the stack, including listing, reads, writes, deletes and
multipart uploads. Consult your storage provider's documentation for the
corresponding permissions. Permission to enumerate or create buckets is not
required when creation is disabled. Dynamic asset health checks also use
`HeadBucket` and honor shorter caller deadlines.

Environment variables override YAML:

- `COZY_FS_S3_BUCKETS_DEFAULT_NAME` sets the default bucket name.
- `COZY_FS_S3_BUCKETS_FILES_NAME` and `COZY_FS_S3_BUCKETS_FILES_URL` configure
  the files destination; the other types follow the same pattern, such as
  `COZY_FS_S3_BUCKETS_APPS_WEB_NAME`.
- `COZY_FS_S3_AUTO_CREATE_BUCKETS=false` disables creation; `true` enables it.

### Local development with MinIO

This tutorial explains how to set up a local S3 backend using MinIO for
development and testing.

**1. Start MinIO with Docker:**

```bash
docker run -d --name minio \
  -p 9000:9000 \
  -p 9001:9001 \
  -e MINIO_ROOT_USER=minioadmin \
  -e MINIO_ROOT_PASSWORD=minioadmin \
  quay.io/minio/minio:RELEASE.2025-02-28T09-55-16Z server /data --console-address ":9001"
```

MinIO is now running:
- S3 API: `http://localhost:9000`
- Web console: `http://localhost:9001` (login: `minioadmin` / `minioadmin`)

**2. Configure cozy-stack:**

Buckets are created automatically at startup. No manual bucket creation
is needed.

Edit your `~/.cozy/cozy.yaml`:

```yaml
fs:
  url: s3://localhost:9000?access_key=minioadmin&secret_key=minioadmin&use_ssl=false
  s3:
    buckets:
      default:
        name: company-storage
```

**3. Build and start:**

```bash
go build -o ~/go/bin/cozy-stack .
~/go/bin/cozy-stack serve
```

You should see in the logs:

```
Successfully connected to S3 endpoint localhost:9000
```

**4. (Re)install your apps:**

When switching from a different storage backend (e.g. `file://`), you need
to reinstall the apps so their assets are stored in S3:

```bash
cozy-stack apps uninstall drive --domain your.domain.localhost:8080
cozy-stack apps install drive --domain your.domain.localhost:8080
cozy-stack apps uninstall home --domain your.domain.localhost:8080
cozy-stack apps install home --domain your.domain.localhost:8080
```

**5. Verify:**

Check that objects appear in MinIO:

```bash
docker exec minio mc ls --recursive local/company-storage/apps-web/
```

Upload a file via the Drive UI or the API:

```bash
TOKEN=$(cozy-stack instances token-cli your.domain.localhost:8080 io.cozy.files)
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: text/plain" \
  "http://your.domain.localhost:8080/files/io.cozy.files.root-dir?Type=file&Name=test.txt" \
  -d "Hello S3!"
```

Verify the file is in MinIO:

```bash
docker exec minio mc ls --recursive local/company-storage/files/
```

**6. Switching back to local filesystem:**

Comment out the S3 URL in your config and restart cozy-stack:

```yaml
fs:
  # url: s3://localhost:9000?access_key=minioadmin&secret_key=minioadmin&use_ssl=false
```

Note: files uploaded to S3 won't be accessible when using the local
filesystem backend, and vice versa. Each backend has its own storage.

## Migrating an instance from Swift to S3

Instances can be moved from Swift to S3 one at a time, without changing the
storage backend for the rest of the fleet. This is useful to validate S3 on
a few instances before committing the whole platform to it.

### 1. Configure the migration target

Set `fs.migration_target` to the S3 URL, keeping `fs.url` on `swift://`.
Configure the target buckets in `fs.s3.buckets`. Both connections (Swift and S3)
are then initialized at startup. Bucket entries without their own `url` use
`fs.migration_target`:

```yaml
fs:
  url: swift://openstack/?UserName=...
  migration_target: s3://s3.example.net?access_key=ACCESS&secret_key=SECRET&region=rbx
  s3:
    auto_create_buckets: false
    buckets:
      default:
        name: company-storage
```

With `auto_create_buckets: false`, create the configured buckets before
starting the stack. See [`fs.migration_target`](config.md#storage-backend-migration)
in the configuration documentation for details on this key.

### 2. Run the migration

```bash
cozy-stack instances migrate-storage <domain> --to s3
```

Use `--dry-run` to preview file and version counts, their combined byte size,
and avatar presence. It does not copy data, create target containers, block the
instance, or switch backends. This also applies with `--flag-only --force`.
The instance remains active, so these counts can change before migration.
Target verification happens only during an actual migration.

Without `--dry-run`, the command:

- blocks HTTP access to the instance for the duration of the migration;
- compares the file and version databases' `update_seq` values across a
  five-second quiet period, aborting before copying if either changes;
- holds the instance's VFS write lock through copying, verification, backend
  switching and optional source cleanup;
- copies the files, file versions, and the user's avatar to the S3 target
  (thumbnails and installed apps are not copied; they regenerate on the
  target backend);
- verifies the copied objects against the source;
- flips the instance's storage backend to S3 and restores its previous block state.

By default the Swift source is kept as-is after a successful migration, so
it stays available for a rollback.

The CLI waits for completion without the usual 30-minute admin-client timeout.
Follow the stack logs for the instance domain and the `storagemigration` namespace
to see each phase and the final success or error. During copying, completed file,
version and byte counts are logged when an object finishes and at least 30 seconds
have passed since the last progress update. A single large object can therefore
take longer without a progress update.

Inspect the instance in another terminal:

```bash
cozy-stack instances show <domain>
```

The output includes `fs_scheme`, `blocked` and `blocking_reason` when set. A missing
`fs_scheme` means the global `fs.url` scheme is used. A switch to the target means
copy verification and cutover succeeded; optional source cleanup or block-state
restoration may still be pending or have failed. `blocking_reason: "MOVING"` marks
the migration block, but is not proof that the operation is still running.

If the CLI disconnects, the server can continue migrating. Check the logs and
instance state before retrying, unblocking the instance, or purging source data.
Migration progress is not persisted as a job and cannot resume after a server
restart.

**Known limitations:** the quiet-period check cannot detect uploads that are still
streaming or avatar changes. The VFS lock only protects operations that use it;
an upload already in progress can finalize against the source after it is released.
Run migrations during low-activity periods until this is addressed.

### 3. Roll back if needed

If something looks wrong shortly after the switch, before any real write
has landed on the S3 target, flip the instance back to the retained Swift
source instantly, without copying anything back:

```bash
cozy-stack instances migrate-storage <domain> --to swift --flag-only --force
```

`--force` is required with `--flag-only` because any writes made against S3
since the cutover are lost.

If real data now lives on S3 and needs to be preserved, run a full
migration back to Swift instead, which copies the data:

```bash
cozy-stack instances migrate-storage <domain> --to swift
```

### 4. Reclaim the source

Once confident the instance is stable on its new backend, delete the
retained source objects:

```bash
cozy-stack instances migrate-storage <domain> --to s3 --purge-source
```

Since the instance already uses `s3`, this runs in purge-only mode: nothing
is copied, verified, or flipped, and the previously retained Swift data is
simply deleted. The instance is not blocked for this step. Running the same
command again is safe and is the way to retry a purge that failed right
after an earlier switch.

(`--purge-source` can also be supplied on the initial migration if no
rollback window is needed.)

### 5. Switch the global default

After the whole fleet has been migrated to S3, change the global `fs.url`
to the S3 URL and remove `fs.migration_target`. From then on, the
per-instance backend flag set by earlier migrations simply matches the
global default.

## Storage namespaces

The same object paths are used whether types share a bucket or use separate
buckets. Every list and deletion is restricted to its type or instance path.

| Storage type | Object prefix | Content |
|--------------|---------------|---------|
| `files` | `files/<DBPrefix>/` | Files, versions, avatars and thumbnails |
| `apps_web` | `apps-web/` | Installed web applications, markers and tarballs |
| `apps_konnectors` | `apps-konnectors/` | Installed konnectors, markers and tarballs |
| `assets` | `assets/` | Dynamic assets, grouped by context |
| `previews` | `previews/` | PDF previews and icons |
| `exports` | `exports/` | Instance export archives, grouped by domain |

## Object key structure

Within a bucket, each instance's data is isolated by a key prefix derived
from `DBPrefix()` (typically the instance domain or a CouchDB prefix).

### VFS files

```
files/<DBPrefix>/<docID_part1>/<docID_part2>/<docID_part3>/<internalID>
```

The document ID (a 32-character UUID v7 hex string) is split into virtual
subfolders to avoid flat hierarchies:

```
files/cozy218def.../019d35b1-9dc3-78ec-994d-f5/44336/7f1b6/e0AbCdEfGh123456
                    ^^^^^^^^^^^^^^^^^^^^^^       ^^^^^  ^^^^^  ^^^^^^^^^^^^^^^^
                    first 22 chars               5 ch   5 ch   16-char internalID
```

This structure mirrors the Swift V3 layout (`MakeObjectNameV3`).

### Thumbnails

```
files/<DBPrefix>/thumbs/<docID_split>-<format>
```

Formats: `small`, `medium`, `large`.

### Avatar

```
files/<DBPrefix>/avatar
```

## Memory consumption

The S3 backend is designed to have comparable memory usage to Swift:

| Scenario                    | Memory per upload |
|-----------------------------|-------------------|
| Known size, file < 5 GiB   | ~32 KB (single PUT, stream) |
| Unknown size (rare)         | ~5 MiB (multipart, PartSize=5MiB, NumThreads=1) |

When `ByteSize` is known on the file document (the common case for drive
uploads), the backend passes the exact size to `PutObject`, which uses a
single PUT request that streams directly to S3 with minimal buffering — the
same behavior as Swift's `ObjectCreate`.

Multipart upload is only used for files with unknown size or exceeding 5 GiB,
with `PartSize=5MiB` and `NumThreads=1` to limit memory.

## Encryption at rest

The S3 backend does not implement client-side encryption. Encryption should
be configured at the infrastructure level (S3 bucket default encryption /
SSE-S3), the same approach used for the Swift backend.

## Differences from Swift

| Aspect             | Swift                         | S3                                     |
|--------------------|-------------------------------|----------------------------------------|
| Container/Bucket   | One per instance              | One per organization (shared)          |
| Instance isolation | Container name                | Key prefix within bucket               |
| Delete instance    | Delete entire container       | Delete all objects with key prefix     |
| File streaming     | Native `io.WriteCloser`       | `io.Pipe` + `PutObject` goroutine      |
| Bulk delete        | `BulkDelete` API              | `RemoveObjects` channel API            |
| Server-side copy   | `ObjectCopy`                  | `CopyObject` (same endpoint only)      |

## Testing

The VFS integration tests run against all three backends (afero, swift, s3)
using a table-driven approach. The S3 tests use
[testcontainers-go](https://testcontainers.com/guides/getting-started-with-testcontainers-for-go/)
with a MinIO container that is started automatically.

```bash
# Run VFS tests (requires CouchDB + Docker)
COZY_COUCHDB_URL=http://admin:admin@localhost:5984/ \
  go test ./model/vfs/ -run TestVfs -v -count=1 -timeout 300s

# Run naming unit tests (no external deps)
go test ./model/vfs/vfss3/ -run "TestMakeObjectKey|TestMakeDocID" -v

# Exercise all storage types in one bucket (requires Docker, no CouchDB)
go test ./model/move/ -run TestS3SharedStorage -v -count=1 -timeout 2m
```