Architecture and reproducible deployment
Data flow
Boundary: The RAG application and Phoenix tracing are separate from this inference node. No public access to vLLM is enabled by the recorded configuration.
Important runtime settings
model: google/gemma-4-12B-it
precision: bfloat16
context length: 32768 tokens
GPU memory utilization target: 0.90
max_num_seqs: 8
OpenAI-compatible API: 127.0.0.1:14041
systemd unit: gemma12b.service
max_num_seqs=8 is a scheduler limit, not a limit of eight concurrent HTTP clients. Additional requests may wait in the queue. The max context setting does not guarantee 32K tokens for every concurrent request under a fixed KV cache budget.
Files and services
/opt/services/models/gemma-4-12B-it/
/opt/services/vllm/.venv/
/opt/services/monitoring/
/etc/systemd/system/gemma12b.service
/etc/gemma12b-auth.yaml # secret, DO NOT PUBLISH
/opt/services/monitoring/grafana.env # secret, DO NOT PUBLISH
Reproducible bootstrap
The internally retained homa-bootstrap-v1.sh is a portable candidate installer, not yet clean-VM certified. It is deliberately not included as a public download: the implementation reference contains host-specific assumptions and needs a dedicated security review. Review it before running, and first test in an isolated VM. It expects Debian 12 x86_64, a working NVIDIA driver/GPU with sufficient VRAM, working Docker/Compose, outbound package/model access, and an actual private IP assigned to the host. The model may require Hugging Face authorization/license acceptance.
Replace the example address with the new host's private address; do not copy an old host's address.
# Private operator workflow only, on a disposable Debian 12 VM.
# Replace PRIVATE_HOST_IP with the actual private IPv4 assigned to that VM.
# Obtain the reviewed script from an approved internal source first.
sudo env HOMA_PRIVATE_IP=PRIVATE_HOST_IP bash ./homa-bootstrap-v1.sh --preflight
sudo env HOMA_PRIVATE_IP=PRIVATE_HOST_IP bash ./homa-bootstrap-v1.sh --dry-run
# DANGEROUS: --apply installs packages, downloads artifacts and manages services.
# Explicit operator approval and an isolated VM are required.
sudo env HOMA_PRIVATE_IP=PRIVATE_HOST_IP \
HOMA_BOOTSTRAP_APPROVAL=AUTHORIZE_HOMA_BOOTSTRAP_V1 \
bash ./homa-bootstrap-v1.sh --apply
sudo env HOMA_PRIVATE_IP=PRIVATE_HOST_IP bash ./homa-bootstrap-v1.sh --verify
Optional HOMA_COMPONENT=all|inference|monitoring scopes deployment. The installer pins vLLM 0.32.0, Transformers 5.19.0, Hugging Face Hub 2.4.0, Prometheus 3.7.3, Grafana 12.2.0, and Node Exporter 1.9.1.
Safety: The script blocks collisions with unmanaged paths, ports and units; creates new per-host secrets and uses localhost for model and metrics. It may install system packages and download large dependencies when --apply is explicitly authorized. Do not run with shell tracing (bash -x) because secrets may leak. Do not blindly run on an existing production host.
On-host validation
systemctl is-active gemma12b.service
systemctl is-enabled gemma12b.service
curl -fsS http://127.0.0.1:14041/health
curl -fsS http://127.0.0.1:14041/metrics >/dev/null
# A /v1/models call requires Authorization: Bearer <KEY>
The recorded deployment passed service health, authenticated model discovery and a non-authenticated access check. A fresh deployment should also verify that unauthenticated model endpoints reject access and exercise a real completion request.
Prerequisites and reproducibility boundaries
- A fresh Debian 12 x86_64 VM or host with a working NVIDIA driver,
nvidia-smi, enough VRAM, and a verified CUDA-compatible stack. The candidate installer itself requires at least 44,000 MiB of reported GPU memory; do not infer that every consumer GPU with a similar product label meets this threshold. - A separately provisioned Docker Engine and Compose plugin, systemd, package repositories, and access to the necessary model weights. A Hugging Face account or license acceptance may be needed.
- An assigned private IPv4 address and an approved change scope. Confirm no existing services or paths will be overwritten.
- Pin a model revision explicitly for reproducibility. The candidate's default revision is
main, and version-tagged container images are not digest-pinned. A successful run today does not guarantee byte-for-byte reproducibility later. - Retain an approved rollback plan and backups/snapshots appropriate to the target environment before
--apply. No clean-VM recovery validation is recorded.
Representative authenticated API request (synthetic example)
A local operator can use the OpenAI-compatible endpoint after obtaining authorization. Set the API_KEY variable through an approved secret mechanism; do not type or print actual keys in documentation. The following test prompt is fabricated for illustration and is not a captured user request.
curl --fail-with-body --silent --show-error \
http://127.0.0.1:14041/v1/chat/completions \
-H "Authorization: Bearer ${API_KEY}" \
-H 'Content-Type: application/json' \
-d '{"model":"google/gemma-4-12B-it","messages":[{"role":"user","content":"Reply with READY."}],"max_tokens":16}'
Observed checks demonstrated an authenticated inference request. This example is not an independently executed verification of the published instructions. For health checks, HTTP 200 indicates that the endpoint responded; it does not prove that RAG, long context or recovery workloads are production-ready.
Incident and rollback notes
- Preflight and dry-run: expected to avoid mutation. Check the script's actual diff and side effects before relying on those expectations.
- Apply: alters packages, services, local model files, and monitoring state. Run on an approved disposable host only until acceptance is complete.
- Rollback: retain the previous inference route and explicitly documented, host-specific service recovery steps. No tested automated rollback is included in the evidence.
- Prohibited: never run reference scripts blindly on the recorded production host, and never modify live credentials or traffic from these instructions.