docs/troubleshooting.md
Rama main · réplica de lectura del repositorio original.
```
╔════════════════════════════════════════════════════════════════════════════╗
║ ║
║ TROUBLESHOOTING ║
║ The handful of things that actually break, and what to do about them. ║
║ ║
╚════════════════════════════════════════════════════════════════════════════╝
```
# Troubleshooting
The things that go wrong in practice, in roughly the order they tend to
hit you. Each entry follows the same shape: a framed symptom box, then
the cause and the fix.
---
```
┃ ⚠ Symptom
┃ Startup log shows "✔ Email off (...)" or "✖ Email disabled: <reason>."
┃ Invitations only show a copyable link instead of sending mail,
┃ "Forgot password" reports itself unavailable, and email-based 2FA
┃ can't be enabled.
```
**Cause.** "Email off" means `SMTP_HOST`, `SMTP_USER`, or `SMTP_PASS` is
empty in `.env`, which is a supported way to run. "Email disabled: SMTP
connection failed" means all three are set and the server refused the
connection test, which is a fault to fix. Mail is optional:
the server boots either way and degrades those three features instead of
exiting (the admin-credentials gate below is the only thing that's still
boot-fatal). Squad invitations still work: the in-app notification is the
reliable channel and the email was only ever a convenience.
**An email change on an account with no password is refused too,** since
its confirmation code goes out by email; see the account-page entry below.
**Two-factor authentication is affected, in both directions.** Logging in
with authenticator-app (TOTP) 2FA works normally, but *both* of these
refuse with `503` while mail is off, because both deliver a code by email:
- **Login for an account using email 2FA.** The verification code cannot
be sent, so the login is refused outright rather than issuing a
challenge nobody can answer. The account is unreachable until mail is
restored, and "Forgot password" is unavailable too, so there is no
self-service way out.
- **Turning 2FA off,** for email *and* TOTP accounts alike. The disable
confirmation code is emailed in both cases.
**Recovery: an admin can clear 2FA without touching the database.**
`POST /api/admin/users/:id/2fa/reset` (also exposed as a "Reset 2FA" button
in the admin console's Users panel, next to a user whose 2FA is enabled)
clears `two_factor_method`, `totp_secret`, any outstanding
`two_factor_codes`, and any unused `password_reset_tokens` rows left behind
by an in-progress setup or disable confirmation. The affected user gets an
in-app notification either way; restoring mail is only needed if they want
to set 2FA up again with email delivery. See `docs/api/admin.md`.
**Fix.**
1. Confirm all three SMTP variables are set in `.env`.
2. Try the credentials manually with `swaks` or any SMTP CLI.
3. If you're using Gmail / Workspace, you need an **app password**, not
your account password. TLS port 465 also works
(`SMTP_PORT=465` triggers TLS automatically).
4. Restart with `docker compose -f docker-compose-prod.yml up -d` and
watch `docker logs` for `✔ SMTP connection verified`.
**The mail capability is decided once, at boot, and never re-checked.**
`initMail()` runs before the port opens and the answer holds for the life
of the process, so fixing `.env` or bringing the SMTP host back up does
nothing until you **restart the app**. The same applies in reverse: a
server that dies after boot is not noticed, sends just start failing.
Connection and greeting attempts time out after 10 seconds each, so an
unreachable SMTP host delays startup by seconds, not minutes.
---
```
┃ ⚠ Symptom
┃ Server exits at startup with "Missing required admin configuration".
```
**Cause.** `ADMIN_USERNAME`, `ADMIN_EMAIL`, and `ADMIN_PASSWORD` are not
all set. The admin super-user is synced from `.env` on every boot — there
is no way to bootstrap the system without one.
**Fix.** Set all three in `.env`. Change them later in `.env`, not in the
app: every boot resets the admin's email and password from `ADMIN_EMAIL` and
`ADMIN_PASSWORD`.
---
```
┃ ⚠ Symptom
┃ "admin sync: ... matches an existing non-admin account (user N),
┃ refusing to promote it" in the server log at startup.
```
**Cause.** An account that is not an admin holds the name in `ADMIN_USERNAME`
or the address in `ADMIN_EMAIL`, usually because the admin renamed or changed
email and a member took the old one. The boot sync never promotes an account,
so it changed nothing and carried on starting.
**Fix.** If account `N` should be the admin, promote it in the admin console
(Users, click its **User** badge); from the next boot on its email and password
are reset from `.env` like any admin's. Otherwise set `ADMIN_USERNAME` and
`ADMIN_EMAIL` to the admin's current name and address, or to unused ones to
have boot create a fresh admin, and restart. See
[deployment.md, The boot admin](./deployment.md#the-boot-admin).
---
```
┃ ⚠ Symptom
┃ "Another Cloud Codex process (MySQL connection N) already serves this
┃ database." and the app exits at startup.
```
**Cause.** Something else is already running Cloud Codex against the same
database: a second container, an `npm run dev` left open in another terminal,
or an old container `docker compose up` did not replace. Two processes would
each keep their own copy of every open document and overwrite each other's
edits, so the second one refuses.
**Fix.** Find connection `N` on the MySQL server
(`SELECT * FROM performance_schema.processlist WHERE ID = N;` shows its host)
and stop that process; the lock goes with it, and the next start succeeds.
Only if you know exactly why two processes must share one schema, set
`C2_INSTANCE_LOCK=0`. See
[deployment.md, One process per database](./deployment.md#one-process-per-database).
If the same message appears in a process that was already running, followed by
`another process took the instance lock while this one had lost it`, MySQL
restarted while a second process was waiting to start, and that one reconnected
first. The first process stopped on purpose, so that only one keeps serving.
The fix is the same: stop the one you did not mean to run.
---
```
┃ ⚠ Symptom
┃ "ECONNREFUSED 127.0.0.1:3306" or "Access denied for user … "
┃ in the Node container's logs.
```
**Cause.** The MySQL container is not up yet, or the credentials in
`.env` don't match what MySQL was initialized with. The very first
`docker compose up` initializes MySQL using `MYSQL_ROOT_PASSWORD` plus
the user/db env vars — once that volume exists, changing those env
vars in `.env` will not retroactively change MySQL's credentials.
**Fix.**
1. `docker compose ps` — confirm the MySQL container is `healthy`.
2. `docker logs <mysql-container>` — check for init errors.
3. If you've been changing `DB_USER`/`DB_PASS` after first boot, the
credentials inside the volume don't match. Either reset the user
inside MySQL (`make db-shell`, then `ALTER USER`) or — if you don't
need the data — remove the volume and start fresh: `docker compose
down -v` (this **destroys** all data; back up first).
---
```
┃ ⚠ Symptom
┃ On a fresh install, every request fails and the app's log shows
┃ "Table 'c2.users' doesn't exist". The MySQL container's log shows
┃ "/docker-entrypoint-initdb.d/init.sql: Permission denied".
```
**Cause.** The host enforces SELinux (Fedora, RHEL and their relatives), and
the compose file mounted `init.sql` without a relabel, so the MySQL container
could not read it. The first boot's initialisation then failed, MySQL
restarted on a data directory that was no longer empty, skipped
initialisation, and came up with no tables. `docker-compose-release.yml` and
`docker-compose-prod.yml` up to and including 0.11.0 mount it this way; later
versions mount it `:ro,z`.
**Fix.** On a version that still has the old line, change the `init.sql`
mount in the compose file you run to end in `:ro,z`, then start again from an
empty data directory: `docker compose -f <file> down -v` (this **destroys**
the database, which on a fresh install holds nothing yet) and `docker compose
-f <file> up -d`. The app's log then shows `admin sync: created ...`.
---
```
┃ ⚠ Symptom
┃ Google sign-in lands back on the sign-in form with "Your Google
┃ account domain is not allowed" (`/?oauth_error=domain_not_allowed`).
```
**Cause.** `GOOGLE_OAUTH_DOMAIN` is set and the Google account's hosted
domain is a different one (or it is a consumer account with none).
**Fix.** Either set `GOOGLE_OAUTH_DOMAIN` to the right domain, or remove
it to accept any Google account. Removing it also stops automatic account
creation for everyone, so a newcomer then needs an invitation first. While
it is set, the domain check runs before any account lookup, so an
existing user whose Google account is on another domain is refused too.
---
```
┃ ⚠ Symptom
┃ Google sign-in lands back on the sign-in form with "This email is
┃ already linked to a different Google account"
┃ (`/?oauth_error=identity_conflict`).
```
**Cause.** A Cloud Codex user has this email, and already has a Google
account linked under a different Google identity. That happens when a
Workspace address is reassigned to a new person, or a Google account is
deleted and recreated. Sign-in refuses rather than hand the newcomer the
previous owner's account.
**Fix.** Decide whose account it is. If it really is the same person on a
new Google account, they sign in with their password and choose **Unlink**
next to Google in the account menu (it needs a password set), then sign in
with Google again, which links the new account. Without a password, an
operator deletes the old link:
`DELETE FROM oauth_accounts WHERE user_id = <id> AND provider = 'google'`
(`make db-shell`). If it is a different person, the address has changed
hands and the old account still holds it; deal with that account first,
because two users cannot share an email.
---
```
┃ ⚠ Symptom
┃ Google sign-in lands back on the sign-in form with "This account has
┃ two-factor authentication on, so it cannot be linked to Google by
┃ email" (`/?oauth_error=two_factor_enabled`).
```
**Cause.** A Cloud Codex user has this email, has two-factor
authentication on (an authenticator app or an email code, either one), and
has no Google account linked yet. Google sign-in links an existing account
by its verified email only when two-factor is off, because a linked Google
account signs in from then on without the local code. Nothing was written.
Rarely, the same answer comes back when the account's email address changed
while the sign-in was under way; signing in again then gets the right answer.
**Fix.** Sign in with the username, password and code; the account works
exactly as before. If its owner also wants Google sign-in on it, they turn
two-factor off from the account menu, sign in with Google once (which links
the account), and can then turn two-factor back on. Know the trade-off before
doing that: a linked Google account signs in through Google's own sign-in,
its MFA included, and is never asked for the local code, whether two-factor
is on or off; password sign-in still asks for it. Turning two-factor off
emails a confirmation code, so on an instance without mail an admin's
**Reset 2FA** does it instead (see the email entry above).
**Reviewing Google links made before this release.** The database cannot
tell a link the owner made from one made without their second factor: a link
made by email records the account's own address as `provider_email`, and
nothing records when two-factor was turned on or off. Two-factor may also have
been turned off since a link was made, so an account showing `none` today is
not cleared by that alone. List every Google link made by email, that is,
made more than a minute after its account (a Google sign-in that creates an
account links it in the same moment), accounts with two-factor on now first
(`make db-shell`):
```sql
SELECT u.id, u.name, u.email, u.two_factor_method,
u.created_at AS account_created, o.created_at AS google_linked
FROM users u
JOIN oauth_accounts o ON o.user_id = u.id AND o.provider = 'google'
WHERE o.created_at > u.created_at + INTERVAL 1 MINUTE
ORDER BY u.two_factor_method IN ('email', 'totp') DESC, o.created_at;
```
For each row, ask the owner whether they linked Google themselves. Where that
cannot be confirmed, delete the link (the owner can link again deliberately,
as the Fix above describes) and the account's session:
```sql
DELETE FROM oauth_accounts WHERE user_id = <id> AND provider = 'google';
DELETE FROM sessions WHERE user_id = <id>;
```
The second statement is not optional. Deleting the link stops new sign-ins
through it, but not the sessions those sign-ins already hold, and it has to
delete every session of the account because no narrower query can find the
right ones. Each sign-in has a session row of its own, one made through the
link included, and nothing ties a row to the link that minted it.
`sessions.auth_provider` does not help: it names the flow only for sessions
minted after the upgrade that added it, and every row from before is tagged
`local`, so a Google sign-in from then looks like a password one. A password
reset through **Forgot password** deletes every session of the account, and so
does an email or password change (the next entry), so either can stand in for
the second statement. Once the sessions are gone, have the owner sign in again
and check that the account's email address, password and two-factor setting
are theirs: before the release that added the current-password check, a
session alone was enough to change the email and the password. If the email
address is not theirs, an operator restores it before the owner resets the
password.
---
```
┃ ⚠ Symptom
┃ Changing your email on the account page asks for your current
┃ password, answers "Your current password is incorrect.", or
┃ refuses with "This account has no password, so an email change is
┃ confirmed with a code sent to your current address, and this
┃ instance cannot send email."
```
**Cause.** A signed-in session is no longer enough to change an account's
email or password. The account page asks for the current password as soon
as the email field differs from the saved address; the API
(`POST /api/update-account`) refuses an email or password change without a
correct `currentPassword`, with a 400 when it is missing and a 401 when it
is wrong, and changes nothing either way. A name change needs no password.
An account with no password (one an external sign-in created) confirms an email
change with a 6-digit code sent to its **current** address instead. That
needs mail, so with mail disabled the change is refused with the sentence
above.
After an email change goes through, every other device signed in to the
account is signed out and has to sign in again; the device that made the
change keeps working on a new session. The old address gets a notice.
**Fix.**
- Wrong password: enter the account's current password. Forgotten it? Sign
out and use "Forgot Password?" on the sign-in screen, then try again.
- No password and mail disabled: an administrator can change the address in
the database (`UPDATE users SET email = ? WHERE id = ?`) or restore mail
(the email-disabled entry above) so the code can be sent.
- A script or integration calling the API: add `currentPassword` to the
body for email and password changes, and store the `token` the response
returns, because the one it sent has been deleted. See `docs/api/auth.md`.
---
```
┃ ⚠ Symptom
┃ GitHub sign-in succeeds but the OAuth callback errors with
┃ "redirect_uri_mismatch".
```
**Cause.** The callback URL registered on the GitHub OAuth app does not
exactly match `${APP_URL}/api/oauth/github/callback`.
**Fix.** Open the GitHub OAuth app settings. The callback URL must
match the public `APP_URL` exactly — including the scheme (https), host,
no trailing slash. If you're testing on a local domain, register a
second callback URL for it.
---
```
┃ ⚠ Symptom
┃ Linked GitHub accounts suddenly fail with "decryption failed" after
┃ redeploying.
```
**Cause.** `GITHUB_CLIENT_SECRET` changed. That value is also the seed
for the AES-256-GCM key that encrypts stored OAuth tokens. Changing
it invalidates every stored token in `oauth_accounts`.
**Fix.** If the change was accidental, restore the previous value. If
intentional (e.g. rotated GitHub OAuth app), every linked user must
**re-link** their GitHub account from `/account`. Tokens cannot be
recovered.
---
```
┃ ⚠ Symptom
┃ Editor opens but other users' cursors / live edits don't appear,
┃ or the doc reverts after refresh. Notifications never push live.
```
**Cause.** A reverse proxy in front of the app is stripping or not
forwarding the WebSocket upgrade headers. Both `/collab/:logId` and
`/notifications-ws` rely on `Upgrade: websocket` + `Connection: Upgrade`
passing through unchanged.
**Fix.**
- **Caddy:** `reverse_proxy` handles WS automatically.
- **nginx:** add the standard upgrade block, beside the `X-Forwarded-For`
line every Cloud Codex proxy needs (`$remote_addr` when nginx is the only
proxy; behind a load balancer, see
[Rate limiters](deployment.md#rate-limiters)):
```
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header X-Forwarded-For $remote_addr;
```
- **Cloudflare:** WebSockets are on by default for paid plans; verify
the WebSockets toggle is on in the dashboard.
Confirm by opening the browser dev console — the WS connection to
`/collab/:logId` should be `101 Switching Protocols`, not `502` or
`400`.
---
```
┃ ⚠ Symptom
┃ "/notifications-ws" closes immediately with code 4002 or 403.
```
**Cause.** Either the proxy is rewriting the `Origin` header (so the
same-origin check in `services/user-channel.js` rejects the upgrade),
or the client did not send `{type:'auth', token:'…'}` within 5 seconds.
**Fix.**
- Make sure your proxy preserves both `Origin` and `Host` so the public
hostname appears on both — the WS server compares them as exact
matches.
- The auth message must arrive within 5 s of upgrade. If a custom
client is being used, send it immediately on `open`.
---
```
┃ ⚠ Symptom
┃ Image upload silently fails or the server logs include
┃ "Could not load the sharp module" / "Cannot find module
┃ '@img/sharp-…'".
```
**Cause.** `sharp` is a native module and ships per-platform binaries.
Building inside one image and running on another (e.g. building on Mac
and running on Linux/arm64) leaves it without a matching native binary.
**Fix.** Always build the production image on the same platform you'll
run it on, or use Docker buildx with a matching `--platform`. Locally,
`npm rebuild sharp` after switching platforms.
---
```
┃ ⚠ Symptom
┃ CI fails with "ERROR: Coverage … below threshold for routes/foo.js
┃ (lines 78%, expected 85%)".
```
**Cause.** A change reduced coverage on a glob with a per-glob threshold
in `vitest.config.js`. The global floor is intentionally low; the
per-glob floors lock in achieved coverage on security-critical modules.
**Fix.** Either add tests to bring coverage back, or — if the drop is
genuinely justified (e.g. removing dead code increases the percent
denominator) — adjust the threshold for that glob. Don't blanket-lower
without good reason; the threshold's job is exactly to make this
visible.
---
```
┃ ⚠ Symptom
┃ `npm test` floods stdout with React 19 warnings during frontend
┃ tests.
```
**Cause.** React 19 emits dev-only warnings for patterns it's
deprecating (act-less updates, certain ref usages). The frontend
project uses the dev React build for jsdom.
**Fix.** Expected. Don't suppress globally — fix the underlying pattern
in the component or test (usually wrapping the action in `act(…)` or
adopting `userEvent`). If a third-party library is the source, file an
upstream issue and silence narrowly with a `console.error` mock around
the offending block.
---
```
┃ ⚠ Symptom
┃ Vite dev server fails to bind because port 3000 is in use.
```
**Cause.** Another instance of `npm run dev` is already running, or
something else (a previous container, an unrelated app) is holding the
port.
**Fix.** `lsof -iTCP:3000 -sTCP:LISTEN` to find the offender and kill
it, or start this instance on another port with `PORT=<n>` (`server.js`
honours it, defaulting to 3000); set `APP_URL` to match.
---
```
┃ ⚠ Symptom
┃ `npm install` fails with native build errors on `bcrypt` or
┃ `sharp`.
```
**Cause.** Both are native modules. On Linux you typically need build
tools (`build-essential`, `python3`), and on macOS Xcode CLI tools.
**Fix.** Install the platform's build tools, then retry. The container
image already includes them — if you're building images, this only hits
local installs.