Skip to main content

Monitoring, dashboards and troubleshooting

Monitoring stack​

Prometheus runs locally with a 15-second scrape interval, seven-day retention and 3 GB retention-size cap. Scrape jobs are node (Node Exporter) and vllm (vLLM /metrics). The GPU collector runs nvidia-smi every 15 seconds and writes node-textfile/gpu.prom for Node Exporter. This is not DCGM Exporter; it covers GPU utilization, used/total VRAM, temperature and power, but not DCGM's full diagnostics.

Grafana: PRIVATE_HOST_IP:13000 (private interface; ACL/VPN still require verification)
Prometheus: 127.0.0.1:19090
Node Exporter:127.0.0.1:9100
vLLM metrics: 127.0.0.1:14041/metrics

Dashboard JSON is file-provisioned in /opt/services/monitoring/dashboards. The v4 dashboard has 88 panels, covering GPU, CPU, memory, disks, network, inference performance and scheduler/cache metrics. All panels rendered correctly is not yet proven; several earlier panels showed No data despite valid Prometheus queries, so acceptance includes real Grafana UI checks.

Monitoring verification​

cd /opt/services/monitoring
docker compose ps
curl -fsS http://127.0.0.1:19090/-/ready
curl -fsS http://127.0.0.1:9100/metrics >/dev/null
systemctl is-active homa-gpu-metrics.timer
curl -fsS http://127.0.0.1:9100/metrics | grep '^homa_gpu_'

Check up{job="node"} and up{job="vllm"} equal 1 in Prometheus. These target-health metrics are necessary but not sufficient: verify actual Grafana data and historical queries too.

Useful real vLLM metrics observed:

MetricMeaning
vllm:num_requests_runningRequests active in model execution
vllm:num_requests_waitingRequests awaiting execution
vllm:num_requests_waiting_by_reasonScheduler wait reasons
vllm:kv_cache_usage_percKV cache usage fraction; multiply by 100 for percent
vllm:time_to_first_token_secondsTTFT histogram
vllm:request_time_per_output_token_secondsRequest-level time per output token histogram
vllm:e2e_request_latency_secondsEnd-to-end request latency histogram
vllm:request_queue_time_secondsQueue-time histogram
vllm:prompt_tokens_total, vllm:generation_tokens_totalProcessed input/output tokens
vllm:prefix_cache_hits_total, vllm:prefix_cache_queries_totalPrefix-cache counters

For GPU sample age use time() - node_textfile_mtime_seconds{file="/textfiles/gpu.prom"}. At idle, waiting requests and KV cache usage can correctly equal zero.

Troubleshooting encountered on deployment night​

SymptomCauseApplied resolution
Failed Python virtual environmentpython3.11-venv missingInstall matching Debian venv package
Triton compilation fails with Python.h missingPython development headers absentInstall python3.11-dev
FlashInfer sampling kernel compilation failsPrebuilt kernels not initially available; CUDA compiler unavailableInstall/download compatible FlashInfer prebuilt kernels via vLLM
Docker Hub image returns HTTP 403Registry access restrictedPull mirrored images from docker.arvancloud.ir
Quay manifest succeeds but image layers time outCDN inaccessibleUse Arvan image mirror instead
Grafana/Prometheus restartingConfig files owned root 0600, container users cannot readRestrict fix to non-secret provisioning YAML and Prometheus YAML, read mode 0644
Dashboard loading failsDashboard JSON file mode 0600Change only public dashboard JSON permissions to 0644
Grafana inaccessible from internal networkGrafana bound to loopback onlyBind Grafana to assigned private interface and rely on VPN/network ACL
No data in dashboardsNot conclusively diagnosed; exporter and query APIs workedInspect Grafana panel JSON and Query Inspector rather than resetting collectors

Do not set secrets to 0644; credentials belong in root-restricted files. The early Grafana logs also contained failed optional plugin downloads due to geography restrictions; this was not required for core dashboards.

Dashboard version history and acceptance​

DashboardEvidenceAcceptance state
v2Dashboard upgrade script and iteration recordedFull browser validation not supplied
v3Subsequent query/layout repair script recordedFull browser validation not supplied
v4JSON generated: 88 panels, 78 query targets. One GPU query returned 1 frame / 61 rows via Grafana query API.Not certified for every panel's browser rendering
v5Proposed visual and LLMOps improvementsNot deployed

Do not equate a successful API query with the correct panel datasource, field mapping, time range, graph type, or visual rendering. For each panel: check Prometheus series, inspect Grafana Query Inspector, confirm datasource UID (homa-prom in the reference), the panel's format=time_series and range settings, and view a period containing real samples. An idle metric legitimately can be zero; No data is a separate condition.

Representative PromQL checks​

up{job="node"}
up{job="vllm"}
homa_gpu_utilization_percent
homa_gpu_memory_used_mib
vllm:num_requests_running
vllm:num_requests_waiting
100 * vllm:kv_cache_usage_perc
sum(rate(vllm:generation_tokens_total[5m]))
histogram_quantile(0.95, sum by (le) (rate(vllm:e2e_request_latency_seconds_bucket[5m])))

These expressions are diagnostic examples based on metric names recorded in the source material; validate actual series labels and buckets before incorporating them in a production dashboard. The label value for node_textfile_mtime_seconds depends on the collector's configured textfile path and must be confirmed from actual exposition rather than guessed.

Docker Compose architecture (partial excerpt, not a deployment manifest)​

services:
prometheus:
image: docker.arvancloud.ir/prom/prometheus:v3.7.3
network_mode: host
# Bind only to loopback via the actual Prometheus command flags.
node-exporter:
image: docker.arvancloud.ir/prom/node-exporter:v1.9.1
network_mode: host
grafana:
image: docker.arvancloud.ir/grafana/grafana:12.2.0
network_mode: host
# Store admin credentials in a protected local secret env file.

This shows image selection and connectivity, not a complete, safe or directly executable Compose configuration. The registry mirror resolved observed access failures on one host; availability elsewhere must be checked independently.

Verification and failure handling​

  1. Confirm host-exporter metric presence, target health and scrape timestamps.
  2. Confirm a real Grafana query result using the same datasource, labels and time range as the panel.
  3. Validate browser rendering for every v4 panel, including empty/idle metrics, and record exceptions.
  4. Never set grafana.env, model authentication files, or any file containing credentials to world-readable permissions. Only non-secret provisioning YAML and dashboard JSON were considered for read-mode repairs in this deployment.
  5. Avoid restarting live monitoring or changing network bindings during diagnostics without an approved maintenance window.