Performance measurements and operational checks
Recorded experiments — 2026-11-18
These results are observations, not a production capacity guarantee. They used short text prompts and limited response tokens.
Progressive concurrency test
| Concurrent HTTP requests | Completed | p50 (s) | p95 (s) | Aggregate output throughput (tokens/s) |
|---|---|---|---|---|
| 8 | 8/8 | 3.14 | 3.46 | 257.93 |
| 16 | 16/16 | 4.46 | 6.39 | 276.73 |
| 24 | 24/24 | 6.16 | 9.38 | 279.50 |
| 32 | 32/32 | 7.64 | 12.80 | 281.65 |
| 48 | 48/48 | 10.59 | 18.95 | 278.61 |
| 64 | 64/64 | 13.92 | 25.35 | 284.33 |
The benchmark used a 20-second p95 exploratory cutoff. Under this arbitrary exploratory criterion, 48 completed the short burst below threshold, whereas 64 exceeded it. These are not defined SLOs and do not automatically establish safe long-term traffic levels.
Sustained closed-loop test
| Metric | Result |
|---|---|
| Duration | 610.51 seconds (approximately 10 minutes) |
| Worker concurrency | 32 |
| Completed requests | 1,820 |
| Errors | 0 |
| Completion rate | 2.98 requests/s |
| Aggregate generation throughput | 291 tokens/s |
| Latency p50 | 10.73 s |
| Latency p95 | 10.92 s |
| Latency p99 | 11.03 s |
Workers sent another short request after each completed response; this is a closed-loop workload, not an open-loop arrival-rate test. Under realistic long-context RAG load, results may differ considerably. The script's max_tokens was 128, so tests did not validate 32K context throughput or per-token streaming latency.
Recommended production acceptance (not yet completed)
- Validate real RAG prompt lengths and response lengths, including 8K/16K/32K contexts where supported.
- Test open-loop arrival rates and streaming output; capture TTFT, TPOT and p95/p99 end-to-end.
- Test at least 30–60 minutes under realistic background load and confirm GPU temperature, errors, queue and KV utilization.
- Confirm API authentication and network ACLs end-to-end from the RAG application.
- Perform a controlled restart/boot recovery test in an approved maintenance window.
- Verify Grafana v4 panels using Query Inspector for any remaining
No dataviews. - Keep legacy inference backend until an explicit cutover and rollback plan is accepted.
Daily operations
systemctl status gemma12b.service --no-pager
journalctl -u gemma12b.service -n 60 --no-pager
nvidia-smi
cd /opt/services/monitoring && docker compose ps
systemctl status homa-gpu-metrics.timer --no-pager
Avoid printing credentials, changing Docker daemon settings, restarting production RAG or switching traffic during diagnostics. Restore/restart should use an approved change window.
How to interpret these measurements
- The 8–64 request sequence is a set of short bursts, not a sustained capacity sweep. Report the completed count and p95 together; lower burst latency does not imply unconstrained arrival-rate support.
- The 32-worker sustained test was closed-loop: workers waited for completion before submitting the next request. The observed 2.98 RPS is not an open-loop admitted traffic rate.
- A
max_tokens=128ceiling and short inputs do not establish feasibility at 32,768-token context windows. The--max-num-seqs 8model scheduler setting and simultaneous HTTP requests are different concepts. - No evidence here establishes time-to-first-token, streaming smoothness, memory pressure under realistic retrieval contexts, restart/reboot recovery or tenant-level SLAs.
Service acceptance gate
| Gate | Observed result | Status |
|---|---|---|
| vLLM service health + authenticated inference | Succeeded on recorded host | Observed |
| Node and vLLM scrape targets | Both up=1 | Observed |
| GPU collection through Node Exporter | VRAM/utilization/power/temperature samples | Observed |
| Grafana v4 all-panel rendering | One API query verified, not all panels | Pending |
| Model startup after reboot | No complete evidence | Pending |
| Realistic long-context RAG workload | Not run | Pending |
| External RAG-to-model network/auth | Not authorized/verified | Pending |
All pending gates need a change-reviewed test plan. No throughput or uptime guarantee should be derived from these data alone.