Skip to main content

Performance measurements and operational checks

Recorded experiments — 2026-11-18​

These results are observations, not a production capacity guarantee. They used short text prompts and limited response tokens.

Progressive concurrency test​

Concurrent HTTP requestsCompletedp50 (s)p95 (s)Aggregate output throughput (tokens/s)
88/83.143.46257.93
1616/164.466.39276.73
2424/246.169.38279.50
3232/327.6412.80281.65
4848/4810.5918.95278.61
6464/6413.9225.35284.33

The benchmark used a 20-second p95 exploratory cutoff. Under this arbitrary exploratory criterion, 48 completed the short burst below threshold, whereas 64 exceeded it. These are not defined SLOs and do not automatically establish safe long-term traffic levels.

Sustained closed-loop test​

MetricResult
Duration610.51 seconds (approximately 10 minutes)
Worker concurrency32
Completed requests1,820
Errors0
Completion rate2.98 requests/s
Aggregate generation throughput291 tokens/s
Latency p5010.73 s
Latency p9510.92 s
Latency p9911.03 s

Workers sent another short request after each completed response; this is a closed-loop workload, not an open-loop arrival-rate test. Under realistic long-context RAG load, results may differ considerably. The script's max_tokens was 128, so tests did not validate 32K context throughput or per-token streaming latency.

  1. Validate real RAG prompt lengths and response lengths, including 8K/16K/32K contexts where supported.
  2. Test open-loop arrival rates and streaming output; capture TTFT, TPOT and p95/p99 end-to-end.
  3. Test at least 30–60 minutes under realistic background load and confirm GPU temperature, errors, queue and KV utilization.
  4. Confirm API authentication and network ACLs end-to-end from the RAG application.
  5. Perform a controlled restart/boot recovery test in an approved maintenance window.
  6. Verify Grafana v4 panels using Query Inspector for any remaining No data views.
  7. Keep legacy inference backend until an explicit cutover and rollback plan is accepted.

Daily operations​

systemctl status gemma12b.service --no-pager
journalctl -u gemma12b.service -n 60 --no-pager
nvidia-smi
cd /opt/services/monitoring && docker compose ps
systemctl status homa-gpu-metrics.timer --no-pager

Avoid printing credentials, changing Docker daemon settings, restarting production RAG or switching traffic during diagnostics. Restore/restart should use an approved change window.

How to interpret these measurements​

  • The 8–64 request sequence is a set of short bursts, not a sustained capacity sweep. Report the completed count and p95 together; lower burst latency does not imply unconstrained arrival-rate support.
  • The 32-worker sustained test was closed-loop: workers waited for completion before submitting the next request. The observed 2.98 RPS is not an open-loop admitted traffic rate.
  • A max_tokens=128 ceiling and short inputs do not establish feasibility at 32,768-token context windows. The --max-num-seqs 8 model scheduler setting and simultaneous HTTP requests are different concepts.
  • No evidence here establishes time-to-first-token, streaming smoothness, memory pressure under realistic retrieval contexts, restart/reboot recovery or tenant-level SLAs.

Service acceptance gate​

GateObserved resultStatus
vLLM service health + authenticated inferenceSucceeded on recorded hostObserved
Node and vLLM scrape targetsBoth up=1Observed
GPU collection through Node ExporterVRAM/utilization/power/temperature samplesObserved
Grafana v4 all-panel renderingOne API query verified, not all panelsPending
Model startup after rebootNo complete evidencePending
Realistic long-context RAG workloadNot runPending
External RAG-to-model network/authNot authorized/verifiedPending

All pending gates need a change-reviewed test plan. No throughput or uptime guarantee should be derived from these data alone.