Skip to main content

Architecture and reproducible deployment

Data flow​

Boundary: The RAG application and Phoenix tracing are separate from this inference node. No public access to vLLM is enabled by the recorded configuration.

Important runtime settings​

model: google/gemma-4-12B-it
precision: bfloat16
context length: 32768 tokens
GPU memory utilization target: 0.90
max_num_seqs: 8
OpenAI-compatible API: 127.0.0.1:14041
systemd unit: gemma12b.service

max_num_seqs=8 is a scheduler limit, not a limit of eight concurrent HTTP clients. Additional requests may wait in the queue. The max context setting does not guarantee 32K tokens for every concurrent request under a fixed KV cache budget.

Files and services​

/opt/services/models/gemma-4-12B-it/
/opt/services/vllm/.venv/
/opt/services/monitoring/
/etc/systemd/system/gemma12b.service
/etc/gemma12b-auth.yaml # secret, DO NOT PUBLISH
/opt/services/monitoring/grafana.env # secret, DO NOT PUBLISH

Reproducible bootstrap​

The internally retained homa-bootstrap-v1.sh is a portable candidate installer, not yet clean-VM certified. It is deliberately not included as a public download: the implementation reference contains host-specific assumptions and needs a dedicated security review. Review it before running, and first test in an isolated VM. It expects Debian 12 x86_64, a working NVIDIA driver/GPU with sufficient VRAM, working Docker/Compose, outbound package/model access, and an actual private IP assigned to the host. The model may require Hugging Face authorization/license acceptance.

Replace the example address with the new host's private address; do not copy an old host's address.

# Private operator workflow only, on a disposable Debian 12 VM.
# Replace PRIVATE_HOST_IP with the actual private IPv4 assigned to that VM.
# Obtain the reviewed script from an approved internal source first.
sudo env HOMA_PRIVATE_IP=PRIVATE_HOST_IP bash ./homa-bootstrap-v1.sh --preflight
sudo env HOMA_PRIVATE_IP=PRIVATE_HOST_IP bash ./homa-bootstrap-v1.sh --dry-run
# DANGEROUS: --apply installs packages, downloads artifacts and manages services.
# Explicit operator approval and an isolated VM are required.
sudo env HOMA_PRIVATE_IP=PRIVATE_HOST_IP \
HOMA_BOOTSTRAP_APPROVAL=AUTHORIZE_HOMA_BOOTSTRAP_V1 \
bash ./homa-bootstrap-v1.sh --apply
sudo env HOMA_PRIVATE_IP=PRIVATE_HOST_IP bash ./homa-bootstrap-v1.sh --verify

Optional HOMA_COMPONENT=all|inference|monitoring scopes deployment. The installer pins vLLM 0.32.0, Transformers 5.19.0, Hugging Face Hub 2.4.0, Prometheus 3.7.3, Grafana 12.2.0, and Node Exporter 1.9.1.

Safety: The script blocks collisions with unmanaged paths, ports and units; creates new per-host secrets and uses localhost for model and metrics. It may install system packages and download large dependencies when --apply is explicitly authorized. Do not run with shell tracing (bash -x) because secrets may leak. Do not blindly run on an existing production host.

On-host validation​

systemctl is-active gemma12b.service
systemctl is-enabled gemma12b.service
curl -fsS http://127.0.0.1:14041/health
curl -fsS http://127.0.0.1:14041/metrics >/dev/null
# A /v1/models call requires Authorization: Bearer <KEY>

The recorded deployment passed service health, authenticated model discovery and a non-authenticated access check. A fresh deployment should also verify that unauthenticated model endpoints reject access and exercise a real completion request.

Prerequisites and reproducibility boundaries​

  1. A fresh Debian 12 x86_64 VM or host with a working NVIDIA driver, nvidia-smi, enough VRAM, and a verified CUDA-compatible stack. The candidate installer itself requires at least 44,000 MiB of reported GPU memory; do not infer that every consumer GPU with a similar product label meets this threshold.
  2. A separately provisioned Docker Engine and Compose plugin, systemd, package repositories, and access to the necessary model weights. A Hugging Face account or license acceptance may be needed.
  3. An assigned private IPv4 address and an approved change scope. Confirm no existing services or paths will be overwritten.
  4. Pin a model revision explicitly for reproducibility. The candidate's default revision is main, and version-tagged container images are not digest-pinned. A successful run today does not guarantee byte-for-byte reproducibility later.
  5. Retain an approved rollback plan and backups/snapshots appropriate to the target environment before --apply. No clean-VM recovery validation is recorded.

Representative authenticated API request (synthetic example)​

A local operator can use the OpenAI-compatible endpoint after obtaining authorization. Set the API_KEY variable through an approved secret mechanism; do not type or print actual keys in documentation. The following test prompt is fabricated for illustration and is not a captured user request.

curl --fail-with-body --silent --show-error \
http://127.0.0.1:14041/v1/chat/completions \
-H "Authorization: Bearer ${API_KEY}" \
-H 'Content-Type: application/json' \
-d '{"model":"google/gemma-4-12B-it","messages":[{"role":"user","content":"Reply with READY."}],"max_tokens":16}'

Observed checks demonstrated an authenticated inference request. This example is not an independently executed verification of the published instructions. For health checks, HTTP 200 indicates that the endpoint responded; it does not prove that RAG, long context or recovery workloads are production-ready.

Incident and rollback notes​

  • Preflight and dry-run: expected to avoid mutation. Check the script's actual diff and side effects before relying on those expectations.
  • Apply: alters packages, services, local model files, and monitoring state. Run on an approved disposable host only until acceptance is complete.
  • Rollback: retain the previous inference route and explicitly documented, host-specific service recovery steps. No tested automated rollback is included in the evidence.
  • Prohibited: never run reference scripts blindly on the recorded production host, and never modify live credentials or traffic from these instructions.