Monitoring, dashboards and troubleshooting
Monitoring stack
Prometheus runs locally with a 15-second scrape interval, seven-day retention and 3 GB retention-size cap. Scrape jobs are node (Node Exporter) and vllm (vLLM /metrics). The GPU collector runs nvidia-smi every 15 seconds and writes node-textfile/gpu.prom for Node Exporter. This is not DCGM Exporter; it covers GPU utilization, used/total VRAM, temperature and power, but not DCGM's full diagnostics.
Grafana: PRIVATE_HOST_IP:13000 (private interface; ACL/VPN still require verification)
Prometheus: 127.0.0.1:19090
Node Exporter:127.0.0.1:9100
vLLM metrics: 127.0.0.1:14041/metrics
Dashboard JSON is file-provisioned in /opt/services/monitoring/dashboards. The v4 dashboard has 88 panels, covering GPU, CPU, memory, disks, network, inference performance and scheduler/cache metrics. All panels rendered correctly is not yet proven; several earlier panels showed No data despite valid Prometheus queries, so acceptance includes real Grafana UI checks.
Monitoring verification
cd /opt/services/monitoring
docker compose ps
curl -fsS http://127.0.0.1:19090/-/ready
curl -fsS http://127.0.0.1:9100/metrics >/dev/null
systemctl is-active homa-gpu-metrics.timer
curl -fsS http://127.0.0.1:9100/metrics | grep '^homa_gpu_'
Check up{job="node"} and up{job="vllm"} equal 1 in Prometheus. These target-health metrics are necessary but not sufficient: verify actual Grafana data and historical queries too.
Useful real vLLM metrics observed:
| Metric | Meaning |
|---|---|
vllm:num_requests_running | Requests active in model execution |
vllm:num_requests_waiting | Requests awaiting execution |
vllm:num_requests_waiting_by_reason | Scheduler wait reasons |
vllm:kv_cache_usage_perc | KV cache usage fraction; multiply by 100 for percent |
vllm:time_to_first_token_seconds | TTFT histogram |
vllm:request_time_per_output_token_seconds | Request-level time per output token histogram |
vllm:e2e_request_latency_seconds | End-to-end request latency histogram |
vllm:request_queue_time_seconds | Queue-time histogram |
vllm:prompt_tokens_total, vllm:generation_tokens_total | Processed input/output tokens |
vllm:prefix_cache_hits_total, vllm:prefix_cache_queries_total | Prefix-cache counters |
For GPU sample age use time() - node_textfile_mtime_seconds{file="/textfiles/gpu.prom"}. At idle, waiting requests and KV cache usage can correctly equal zero.
Troubleshooting encountered on deployment night
| Symptom | Cause | Applied resolution |
|---|---|---|
| Failed Python virtual environment | python3.11-venv missing | Install matching Debian venv package |
Triton compilation fails with Python.h missing | Python development headers absent | Install python3.11-dev |
| FlashInfer sampling kernel compilation fails | Prebuilt kernels not initially available; CUDA compiler unavailable | Install/download compatible FlashInfer prebuilt kernels via vLLM |
| Docker Hub image returns HTTP 403 | Registry access restricted | Pull mirrored images from docker.arvancloud.ir |
| Quay manifest succeeds but image layers time out | CDN inaccessible | Use Arvan image mirror instead |
| Grafana/Prometheus restarting | Config files owned root 0600, container users cannot read | Restrict fix to non-secret provisioning YAML and Prometheus YAML, read mode 0644 |
| Dashboard loading fails | Dashboard JSON file mode 0600 | Change only public dashboard JSON permissions to 0644 |
| Grafana inaccessible from internal network | Grafana bound to loopback only | Bind Grafana to assigned private interface and rely on VPN/network ACL |
No data in dashboards | Not conclusively diagnosed; exporter and query APIs worked | Inspect Grafana panel JSON and Query Inspector rather than resetting collectors |
Do not set secrets to 0644; credentials belong in root-restricted files. The early Grafana logs also contained failed optional plugin downloads due to geography restrictions; this was not required for core dashboards.
Dashboard version history and acceptance
| Dashboard | Evidence | Acceptance state |
|---|---|---|
| v2 | Dashboard upgrade script and iteration recorded | Full browser validation not supplied |
| v3 | Subsequent query/layout repair script recorded | Full browser validation not supplied |
| v4 | JSON generated: 88 panels, 78 query targets. One GPU query returned 1 frame / 61 rows via Grafana query API. | Not certified for every panel's browser rendering |
| v5 | Proposed visual and LLMOps improvements | Not deployed |
Do not equate a successful API query with the correct panel datasource, field mapping, time range, graph type, or visual rendering. For each panel: check Prometheus series, inspect Grafana Query Inspector, confirm datasource UID (homa-prom in the reference), the panel's format=time_series and range settings, and view a period containing real samples. An idle metric legitimately can be zero; No data is a separate condition.
Representative PromQL checks
up{job="node"}
up{job="vllm"}
homa_gpu_utilization_percent
homa_gpu_memory_used_mib
vllm:num_requests_running
vllm:num_requests_waiting
100 * vllm:kv_cache_usage_perc
sum(rate(vllm:generation_tokens_total[5m]))
histogram_quantile(0.95, sum by (le) (rate(vllm:e2e_request_latency_seconds_bucket[5m])))
These expressions are diagnostic examples based on metric names recorded in the source material; validate actual series labels and buckets before incorporating them in a production dashboard. The label value for node_textfile_mtime_seconds depends on the collector's configured textfile path and must be confirmed from actual exposition rather than guessed.
Docker Compose architecture (partial excerpt, not a deployment manifest)
services:
prometheus:
image: docker.arvancloud.ir/prom/prometheus:v3.7.3
network_mode: host
# Bind only to loopback via the actual Prometheus command flags.
node-exporter:
image: docker.arvancloud.ir/prom/node-exporter:v1.9.1
network_mode: host
grafana:
image: docker.arvancloud.ir/grafana/grafana:12.2.0
network_mode: host
# Store admin credentials in a protected local secret env file.
This shows image selection and connectivity, not a complete, safe or directly executable Compose configuration. The registry mirror resolved observed access failures on one host; availability elsewhere must be checked independently.
Verification and failure handling
- Confirm host-exporter metric presence, target health and scrape timestamps.
- Confirm a real Grafana query result using the same datasource, labels and time range as the panel.
- Validate browser rendering for every v4 panel, including empty/idle metrics, and record exceptions.
- Never set
grafana.env, model authentication files, or any file containing credentials to world-readable permissions. Only non-secret provisioning YAML and dashboard JSON were considered for read-mode repairs in this deployment. - Avoid restarting live monitoring or changing network bindings during diagnostics without an approved maintenance window.