Deploying a self-hosted Gemma 4 inference node with vLLM and Grafana
We deployed a BF16 Gemma 4 12B model using vLLM on a Debian 12 NVIDIA GPU host, then added local monitoring with Grafana, Prometheus, Node Exporter and an NVIDIA SMI-based GPU collector. The work focused on reproducible deployment, operational visibility and measurements rather than just a working chat endpoint.
Architecture
The model exposes an authenticated OpenAI-compatible API on loopback. Prometheus scrapes vLLM and node-level metrics; Node Exporter's textfile collector ingests GPU memory, utilization, temperature and power. Grafana runs on a private network interface, with internal-only access.
Key implementation decisions
- Model:
google/gemma-4-12B-itwithbfloat16, 32,768 maximum model tokens. - Runtime: vLLM
0.32.0, systemd autostart and API-key authentication. - Metrics: vLLM
/metricsplus Prometheus, Node Exporter andnvidia-smicollector. - Deployability: an explicit-approval bootstrap script with preflight, dry-run, apply and verification modes.
Measured performance
| Test | Observation |
|---|---|
| Short burst of 64 concurrent requests | All 64 completed; p95 25.35 s |
| Sustained 32-worker closed-loop test (~10 min) | 1,820 completions, zero observed failures |
| Sustained output throughput | 291 tokens/s |
| Sustained end-to-end latency p95 | 10.92 s |
These are specific short-prompt benchmarks, not a general production traffic guarantee. Longer RAG inputs, streaming, model contention and arrival patterns need separate acceptance tests.
Lessons learned
The hardest issues were practical: Python headers for Triton compilation, prebuilt FlashInfer sampling kernels, restricted access to container registries, and Linux file permissions in non-root monitoring containers. Switching to an accessible container mirror and fixing only non-secret YAML/dashboard file permissions stabilized the monitoring stack.
For configuration details, see the HomaAi technical documentation covering architecture, monitoring queries, operational checks, security boundaries and benchmark methodology.
What is not complete: all-panel browser verification for Grafana v4, a clean-VM bootstrap acceptance run, long-context RAG tests, Phoenix-to-vLLM tracing, alerting, dashboard v5 and backend cutover.
Hardware caveat: the collected report names an RTX 4090 while also reporting approximately 48 GiB of GPU memory. We have not resolved that inventory mismatch, so the hardware SKU should not be treated as confirmed.