Troubleshooting

Common issues and how to diagnose them.

Connection failures

Dashboard shows “Connection error” or blank charts

Check the ClickHouse URL:

## Test the URL directly (replace with your values)
curl -v "https://your-ch-host:8443/?query=SELECT+1" \
  -u monitoring:password

Common mistakes:

• Missing port (8443 for HTTPS, 8123 for HTTP)
• Wrong scheme — ClickHouse uses https:// not clickhouse://
• Trailing slash in CLICKHOUSE_HOST that the client double-appends

Check env var format for multi-host:

## Correct: comma-separated, same position across all four vars
CLICKHOUSE_HOST=https://host-a:8443,https://host-b:8443
CLICKHOUSE_USER=monitoring,monitoring
CLICKHOUSE_PASSWORD=secret-a,secret-b

If host count differs from user/password count, the second host silently falls back to the first credential.

TLS / certificate errors:

If ClickHouse uses a self-signed cert, set CLICKHOUSE_VERIFY_CERT=false or pass ?verify=false in the URL. In production, install the CA cert instead.

Health endpoint returns 503

/healthz always returns 200 (static liveness probe). /api/healthz returns 503 when the ClickHouse connection fails. Check:

1. CLICKHOUSE_HOST, CLICKHOUSE_USER, CLICKHOUSE_PASSWORD are set correctly.
2. The ClickHouse host is reachable from the dashboard container / Worker.
3. Network policy / firewall allows port 8443 (or 8123).

curl -s https://your-dashboard/api/healthz | jq .
## {"status":"ok","clickhouse":"ok"} on success
## {"status":"error","clickhouse":"error","message":"..."} on failure

”Query timeout” on large tables

Increase CLICKHOUSE_MAX_EXECUTION_TIME (default 60 seconds):

CLICKHOUSE_MAX_EXECUTION_TIME=120

For heavy queries (full query-log scans), the dashboard also respects ClickHouse-level query complexity limits. Consider raising max_execution_time on the monitoring user profile.

---

Auth failures (401 / white screen)

Every page returns 401

CHM_AUTH_PROVIDER is set to something other than none but is not configured properly:

• clerk — check CLERK_SECRET_KEY and VITE_AUTH_PROVIDER=clerk (build-time).
• proxy — check CHM_CF_ACCESS_TEAM_DOMAIN (Cloudflare Access) or CHM_PROXY_AUTH_SECRET (shared secret).
• trusted — check CHM_TRUSTED_AUTH_SECRET or CHM_TRUSTED_ALLOW_INSECURE.

Quick check:

curl -s https://your-dashboard/api/v1/auth/me | jq .
## {"subject":"anonymous","provider":"none"} when no auth is configured
## {"error":"Authentication required"} when auth is on but request is not authenticated

White screen after login (Clerk)

Clerk is a build-time client variable. If VITE_AUTH_PROVIDER was not set at build time, the client JS doesn’t know about Clerk and may loop or blank out.

Confirm the build included Clerk:

## On the deployed bundle, grep for the publishable key prefix
curl https://your-dashboard/ | grep pk_live

If it is missing, rebuild with:

VITE_AUTH_PROVIDER=clerk NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=pk_live_... bun run build

API key returns 401

• The token must be prefixed chm_ — tokens issued by /api/v1/auth/api-key.
• The token was issued against a specific CHM_API_KEY_SECRET. If the secret was rotated, old tokens are invalid.
• Pass the token as Authorization: Bearer chm_... (not as x-api-key).

## Issue a new token
curl -X POST https://your-dashboard/api/v1/auth/api-key \
  -H "Authorization: Bearer $CHM_API_KEY_SECRET"

MCP client returns 401

The MCP endpoint is closed by default. One of CHM_API_KEY_SECRET, CLERK_SECRET_KEY, or CHM_MCP_PUBLIC=true must be set. See MCP Server — Permissions.

---

Performance / timeouts

Dashboard is slow to load

Most dashboard pages are a static shell with client-side data fetching. A slow initial page usually means:

• The static bundle is not cached at the edge (first request after a deploy).
• The ClickHouse query itself is slow.

To identify slow queries, open the AI agent and ask:

“What were the slowest queries in the last 5 minutes?”

Or from the Queries → Slow Queries page.

Charts refresh too frequently

Default refresh is 60 seconds for most charts. Reduce load by setting a longer interval:

## No per-chart env override is available; tune at the ClickHouse level:
## Give the monitoring user a query-cache profile to avoid hitting ClickHouse on every refresh

Alternatively, enable query caching on the monitoring user profile (see ClickHouse User & Grants).

max_query_size exceeded errors

Some dashboard queries on very large query logs can exceed ClickHouse’s default max_query_size. Add to the monitoring user profile:

<max_query_size>1073741824</max_query_size>

---

AI agent / LLM failures

Agent returns “LLM API key not set”

Set LLM_API_KEY to your OpenRouter, Anthropic, or OpenAI key:

LLM_API_KEY=sk-...

The agent uses OpenRouter by default. To switch provider:

LLM_BASE_URL=https://api.anthropic.com/v1
LLM_MODEL=claude-opus-4-5

Agent does not respond / hangs

Check:

1. LLM_API_KEY is a server-side variable (not VITE_LLM_API_KEY).
2. The provider endpoint is reachable from the Worker / container.
3. The model name is correct for the provider — OpenRouter uses provider/model format (e.g. anthropic/claude-3-5-sonnet).

Agent answers are vague or wrong

The agent is bounded by the ClickHouse user’s grants. If it cannot read system.query_log (the most data-rich source), answers will be shallow.

Confirm grants:

SELECT count() FROM system.query_log;

If this returns a permission error, grant the monitoring user SELECT ON system.*.

Agent session metrics show high token usage

Long conversations accumulate context. Start a new conversation to reset the context window. If you see large token counts on simple questions, the system prompt or a long skill is being included — this is expected and keeps answers accurate.

---

Kubernetes / Helm health probes

Pod stuck in CrashLoopBackOff

Common cause: the image pulled is stale (latest tag was not updated). Check the image digest:

kubectl describe pod <chmonitor-pod> | grep Image

If the digest matches an old build, force a re-pull:

kubectl rollout restart deployment chmonitor

Liveness probe failing

/healthz (liveness) should always return 200 — it is a static response. If it is failing, the container is not starting at all. Check container logs:

kubectl logs <pod> --previous

Common causes:

• Missing required env vars (CLICKHOUSE_HOST)
• Port mismatch — the Worker listens on 8080 by default; confirm the probe and container port match

Readiness probe failing

/api/healthz (readiness) checks the ClickHouse connection. A 503 here means the pod is up but cannot reach ClickHouse. The pod will be removed from the Service endpoints until it recovers.

Check:

1. ClickHouse is running and the connection string is correct.
2. NetworkPolicy allows port 8443 from the dashboard namespace.
3. The monitoring user credentials work: curl http://pod:8080/api/healthz.

Probe settings that work well:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 15
  periodSeconds: 20
readinessProbe:
  httpGet:
    path: /api/healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 10
  failureThreshold: 3
startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 12
  periodSeconds: 5

Helm chart: dashboard can reach ClickHouse but shows no data

The chart creates a ConfigMap for env vars. Confirm the values are correct:

kubectl get configmap chmonitor -o yaml | grep CLICKHOUSE

If the host uses https://, confirm TLS is allowed or CLICKHOUSE_VERIFY_CERT=false is set.

---

Related

• ClickHouse User & Grants
• MCP Server — Permissions
• Authentication
• Upgrading ClickHouse