# Troubleshooting
Common issues and how to diagnose them.
## Connection failures
### Dashboard shows "Connection error" or blank charts
**Check the ClickHouse URL:**
```bash
## Test the URL directly (replace with your values)
curl -v "https://your-ch-host:8443/?query=SELECT+1" \
-u monitoring:password
```
Common mistakes:
- Missing port (`8443` for HTTPS, `8123` for HTTP)
- Wrong scheme — ClickHouse uses `https://` not `clickhouse://`
- Trailing slash in `CLICKHOUSE_HOST` that the client double-appends
**Check env var format for multi-host:**
```bash
## Correct: comma-separated, same position across all four vars
CLICKHOUSE_HOST=https://host-a:8443,https://host-b:8443
CLICKHOUSE_USER=monitoring,monitoring
CLICKHOUSE_PASSWORD=secret-a,secret-b
```
If host count differs from user/password count, the second host silently falls back to the first credential.
**TLS / certificate errors:**
If ClickHouse uses a self-signed cert, set `CLICKHOUSE_VERIFY_CERT=false` or pass `?verify=false` in the URL. In production, install the CA cert instead.
### Health endpoint returns 503
`/healthz` always returns 200 (static liveness probe). `/api/healthz` returns 503 when the ClickHouse connection fails. Check:
1. `CLICKHOUSE_HOST`, `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD` are set correctly.
2. The ClickHouse host is reachable from the dashboard container / Worker.
3. Network policy / firewall allows port 8443 (or 8123).
```bash
curl -s https://your-dashboard/api/healthz | jq .
## {"status":"ok","clickhouse":"ok"} on success
## {"status":"error","clickhouse":"error","message":"..."} on failure
```
### "Query timeout" on large tables
Increase `CLICKHOUSE_MAX_EXECUTION_TIME` (default 60 seconds):
```bash
CLICKHOUSE_MAX_EXECUTION_TIME=120
```
For heavy queries (full query-log scans), the dashboard also respects ClickHouse-level query complexity limits. Consider raising `max_execution_time` on the monitoring user profile.
---
## Auth failures (401 / white screen)
### Every page returns 401
`CHM_AUTH_PROVIDER` is set to something other than `none` but is not configured properly:
- `clerk` — check `CLERK_SECRET_KEY` and `VITE_AUTH_PROVIDER=clerk` (build-time).
- `proxy` — check `CHM_CF_ACCESS_TEAM_DOMAIN` (Cloudflare Access) or `CHM_PROXY_AUTH_SECRET` (shared secret).
- `trusted` — check `CHM_TRUSTED_AUTH_SECRET` or `CHM_TRUSTED_ALLOW_INSECURE`.
Quick check:
```bash
curl -s https://your-dashboard/api/v1/auth/me | jq .
## {"subject":"anonymous","provider":"none"} when no auth is configured
## {"error":"Authentication required"} when auth is on but request is not authenticated
```
### White screen after login (Clerk)
Clerk is a build-time client variable. If `VITE_AUTH_PROVIDER` was not set at build time, the client JS doesn't know about Clerk and may loop or blank out.
Confirm the build included Clerk:
```bash
## On the deployed bundle, grep for the publishable key prefix
curl https://your-dashboard/ | grep pk_live
```
If it is missing, rebuild with:
```bash
VITE_AUTH_PROVIDER=clerk NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=pk_live_... bun run build
```
### API key returns 401
- The token must be prefixed `chm_` — tokens issued by `/api/v1/auth/api-key`.
- The token was issued against a specific `CHM_API_KEY_SECRET`. If the secret was rotated, old tokens are invalid.
- Pass the token as `Authorization: Bearer chm_...` (not as `x-api-key`).
```bash
## Issue a new token
curl -X POST https://your-dashboard/api/v1/auth/api-key \
-H "Authorization: Bearer $CHM_API_KEY_SECRET"
```
### MCP client returns 401
The MCP endpoint is closed by default. One of `CHM_API_KEY_SECRET`, `CLERK_SECRET_KEY`, or `CHM_MCP_PUBLIC=true` must be set. See [MCP Server — Permissions](/features/mcp#permissions--access).
---
## Performance / timeouts
### Dashboard is slow to load
Most dashboard pages are a static shell with client-side data fetching. A slow initial page usually means:
- The static bundle is not cached at the edge (first request after a deploy).
- The ClickHouse query itself is slow.
To identify slow queries, open the AI agent and ask:
> "What were the slowest queries in the last 5 minutes?"
Or from the Queries → Slow Queries page.
### Charts refresh too frequently
Default refresh is 60 seconds for most charts. Reduce load by setting a longer interval:
```bash
## No per-chart env override is available; tune at the ClickHouse level:
## Give the monitoring user a query-cache profile to avoid hitting ClickHouse on every refresh
```
Alternatively, enable query caching on the monitoring user profile (see [ClickHouse User & Grants](/getting-started/clickhouse-requirements#recommended-clickhouse-profile)).
### `max_query_size exceeded` errors
Some dashboard queries on very large query logs can exceed ClickHouse's default `max_query_size`. Add to the monitoring user profile:
```xml
1073741824
```
---
## AI agent / LLM failures
### Agent returns "LLM API key not set"
Set `LLM_API_KEY` to your OpenRouter, Anthropic, or OpenAI key:
```bash
LLM_API_KEY=sk-...
```
The agent uses OpenRouter by default. To switch provider:
```bash
LLM_BASE_URL=https://api.anthropic.com/v1
LLM_MODEL=claude-opus-4-5
```
### Agent does not respond / hangs
Check:
1. `LLM_API_KEY` is a server-side variable (not `VITE_LLM_API_KEY`).
2. The provider endpoint is reachable from the Worker / container.
3. The model name is correct for the provider — OpenRouter uses `provider/model` format (e.g. `anthropic/claude-3-5-sonnet`).
### Agent answers are vague or wrong
The agent is bounded by the ClickHouse user's grants. If it cannot read `system.query_log` (the most data-rich source), answers will be shallow.
Confirm grants:
```sql
SELECT count() FROM system.query_log;
```
If this returns a permission error, grant the monitoring user `SELECT ON system.*`.
### Agent session metrics show high token usage
Long conversations accumulate context. Start a new conversation to reset the context window. If you see large token counts on simple questions, the system prompt or a long skill is being included — this is expected and keeps answers accurate.
---
## Kubernetes / Helm health probes
### Pod stuck in `CrashLoopBackOff`
Common cause: the image pulled is stale (`latest` tag was not updated). Check the image digest:
```bash
kubectl describe pod | grep Image
```
If the digest matches an old build, force a re-pull:
```bash
kubectl rollout restart deployment chmonitor
```
### Liveness probe failing
`/healthz` (liveness) should always return 200 — it is a static response. If it is failing, the container is not starting at all. Check container logs:
```bash
kubectl logs --previous
```
Common causes:
- Missing required env vars (`CLICKHOUSE_HOST`)
- Port mismatch — the Worker listens on `8080` by default; confirm the probe and container port match
### Readiness probe failing
`/api/healthz` (readiness) checks the ClickHouse connection. A 503 here means the pod is up but cannot reach ClickHouse. The pod will be removed from the Service endpoints until it recovers.
Check:
1. ClickHouse is running and the connection string is correct.
2. NetworkPolicy allows port 8443 from the dashboard namespace.
3. The monitoring user credentials work: `curl http://pod:8080/api/healthz`.
Probe settings that work well:
```yaml
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 15
periodSeconds: 20
readinessProbe:
httpGet:
path: /api/healthz
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
failureThreshold: 3
startupProbe:
httpGet:
path: /healthz
port: 8080
failureThreshold: 12
periodSeconds: 5
```
### Helm chart: dashboard can reach ClickHouse but shows no data
The chart creates a `ConfigMap` for env vars. Confirm the values are correct:
```bash
kubectl get configmap chmonitor -o yaml | grep CLICKHOUSE
```
If the host uses `https://`, confirm TLS is allowed or `CLICKHOUSE_VERIFY_CERT=false` is set.
---
## Related
- [ClickHouse User & Grants](/getting-started/clickhouse-requirements)
- [MCP Server — Permissions](/features/mcp#permissions--access)
- [Authentication](/authentication)
- [Upgrading ClickHouse](/guides/upgrade-clickhouse)