Troubleshooting

Green containers prove little

Both worst failures below leave docker compose ps green and log nothing. Run the checks.

• 2 failures that leave containers green
• 3 scripts that prove the stack works
• ✓ self-hosted stack verified end to end

The two things that break quietly

1. The JWT hook is not wired. Migrations create public.custom_access_token_hook, which puts platform_role and roles into every token. Self-hosted GoTrue calls it only when the two GOTRUE_HOOK_CUSTOM_ACCESS_TOKEN_* variables in docker-compose.yml are set. Miss them and users sign in normally, but System admins are silently not System admins.
2. NODE_IP is not reachable by clients. Left unset, the media server advertises its internal Docker IP, which nothing outside the bridge can route to. Calls negotiate and carry no media. 127.0.0.1 is right only when every client runs on the server machine. Otherwise use the host’s LAN or public address. LIVEKIT_PUBLIC_URL must be dialable by clients for the same reason; it defaults to ws://${NODE_IP}:${LK_PORT_HTTP}.

The three checks

Run them from the deploy/teamhud directory of the bundle.

verify-stack.sh

./verify-stack.sh

Needs the stack up and a readable .env (it refuses a group- or world-readable .env; fix with chmod 600 .env). It signs up a throwaway verify-<timestamp>@example.test user through the server’s own /auth/signup, so it creates that account on your box. It stops at the first failure and prints a line starting with ✗. The numbered steps it prints:

Step	Proves	A failure usually means
1-2	Schemas and the access-token hook function exist	Migrations did not run
2a-2b	The gateway and LiveKit run this directory’s configuration (verify-gateway.sh, verify-livekit.sh)	A stale container from an older bundle; recreate with docker compose up -d
3a-3	GoTrue’s own sign-up is denied at the gateway; sign-up through the server works	Gateway route or sign-up mode wrong
4	The token carries the custom claims	The JWT hook is not wired
5-6	The API rejects anonymous callers and accepts the issued token	Auth or SUPABASE_URL mismatch
7	The server reaches its own LiveKit	Wrong LIVEKIT_* credentials or URL
8	A real join is routed to this stack’s LiveKit, at an address clients can dial	NODE_IP / LIVEKIT_PUBLIC_URL is an internal name
9	The first System admin exists	BOOTSTRAP_ADMIN_EMAIL did not reach the server (only checked when you set it)
10	Only Caddy’s resolved address gets its X-Forwarded-For honored (TRUST_PROXY_HOST)	Client IPs and ADMIN_IP_ALLOWLIST are wrong
11	A per-org LiveKit row outranks LIVEKIT_* env, and deleting it restores env	Media provider precedence
12	The public edge’s GoTrue and LiveKit routes cannot render as a page	Gateway headers
13	Every provider in SOCIAL_PROVIDERS is offered as a sign-in button	Provider not configured
14	The Teams webhook echoes Microsoft’s validation token (only when all three MSTEAMS_* variables are set)	MSTEAMS_* incomplete or the route unreachable
games	Server to /game/* proxy to the game server (only with games on)	Game server not running
14-15	TURN/TLS through the 443 mux (14), then its preflight (15)	Skipped, never failed, when TURN_TLS_ENABLED is off

It ends with ✓ self-hosted stack verified end to end.

verify-turn.mjs

node verify-turn.mjs teamhud.example.com 3478                       # TURN over UDP
node verify-turn.mjs --tls --servername TURN HOST 443               # TURN over TLS

Needs only Node and that one file, no npm install. It sends a real STUN Binding Request and waits for the success response. OK: means the port is published and the server speaks STUN. FAIL: prints the usual causes. For UDP: turn.enabled is false in livekit.yaml, 3478/udp is not published, or a firewall between you and the server eats UDP/3478. For TLS: TURN_TLS_ENABLED is not true or the stack is down, no DNS for TURN_DOMAIN or --servername does not match it (the mux routes by SNI), or 443 is unreachable from here (or --ca does not match the certificate). Run it from a machine on the network you are worried about, not from the server.

verify-media-planes.mjs

node verify-media-planes.mjs [--expect livekit]

Run from the bundle with the stack up. It asks the running server container’s own resolver which media provider each organization and team would get, then prints ok <org>: mediaProviders=... lines. It exits non-zero on MISMATCH (the advertised list disagrees with what a join would pick), on EXPECTED <x> (the first provider is not what --expect named), or on SILENT PIN (a team is pinned to a plane that is not in force and no reason is given). It prints no organizations on this stack and passes when there are none.

Symptom, cause, fix

No audio or video in calls

The call joins and everyone looks connected, but nothing flows. Work down this list.

1. NODE_IP / LIVEKIT_PUBLIC_URL unreachable. verify-stack.sh step 8 catches it. Set NODE_IP to a public or LAN address clients can reach, then docker compose up -d.
2. UDP not open. Open 7882/udp, 7881/tcp, 3478/udp and 30000-30063/udp on the server (Going live).
3. A client network that allows only outbound 443. Enable TURN over TLS and have their IT follow Corporate network requirements. Confirm with Preferences, Calls, Diagnostics, Test network on that client.
4. Their proxy inspects or SNI-filters the TURN name. “could not establish pc connection” with a working page is the sign. Same page.

A user cannot sign in

Symptom	Cause and fix
The desktop refuses the server	The desktop refuses a non-https:// server unless it is loopback. Put a real hostname and certificate on the server (Going live).
New accounts are refused	Sign-up is not allowed. SIGNUP_MODE=closed or sso_only refuses new accounts (Users and roles).
Sign-in works but the user is not a System admin	The JWT hook (above).
Sign-in works but calls fail before media starts	Behind a proxy: it strips WebSocket Upgrade.
A social sign-in button is missing	verify-stack.sh step 13.
/admin returns 404	ADMIN_IP_ALLOWLIST is unset, which fails closed, or the address is not in it, or TRUST_PROXY_HOST is wrong (step 10).

A blank panel in the app

A Preferences tab or panel that renders empty with nothing red anywhere is a client build defect, not a server setting. The known cause: a renderer that imports @teamhud/utils without resolving its browser export condition fails at import. Nothing in your deployment fixes it; report the app version and which tab. If your dashboard page is blank instead, check that the server container is healthy and that you are on https://.

Recording, transcription or captions do nothing

Symptom	Cause and fix
Nothing records or captions	Org policy is off. Everything is off until an Org owner or Org admin allows it; see Org policies and Recording and transcription.
Recordings capture nothing	RECORDER_TOKEN is empty. The UI says so.
Transcripts say they have no provider	No transcription provider. Recording works. Set TRANSCRIPTION_PROVIDER or have the org add its own.
“ENCRYPTION_KEY is not configured”	ENCRYPTION_KEY missing. An org’s own provider key fails while the rest looks healthy.
No saved audio/video files	Egress is opt-in and needs all three settings together.
Captions arrive per sentence, not per word	DRAFTS_URL is empty or the draft service is full.

Clients are stuck on an old version

Your server does not serve the client update feed, so you cannot push or pin client builds; see Updates and minimum version. Check on the client: Preferences, Updates, Check now. Linux only notifies; upgrade with the package manager or a new AppImage. To nudge old clients set MIN_CLIENT_VERSION; it is a soft floor that shows a persistent Update TeamHUD warning and does not sign anyone out.