<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/">
    <channel>
        <title>Shayan Radi — Engineering Journal</title>
        <link>https://blog.shayanradi.com/blog</link>
        <description>Engineering field notes, tutorials and experimental findings.</description>
        <lastBuildDate>Thu, 19 Nov 2026 00:00:00 GMT</lastBuildDate>
        <docs>https://validator.w3.org/feed/docs/rss2.html</docs>
        <generator>https://github.com/jpmonette/feed</generator>
        <language>en</language>
        <copyright>Copyright © 2026 Shayan Radi</copyright>
        <item>
            <title><![CDATA[Deploying a self-hosted Gemma 4 inference node with vLLM and Grafana]]></title>
            <link>https://blog.shayanradi.com/blog/homaai-inference-monitoring</link>
            <guid>https://blog.shayanradi.com/blog/homaai-inference-monitoring</guid>
            <pubDate>Thu, 19 Nov 2026 00:00:00 GMT</pubDate>
            <description><![CDATA[A measured field report on self-hosted LLM inference, monitoring, security boundaries and workload limitations.]]></description>
            <content:encoded><![CDATA[<p>We deployed a BF16 Gemma 4 12B model using vLLM on a Debian 12 NVIDIA GPU host, then added local monitoring with Grafana, Prometheus, Node Exporter and an NVIDIA SMI-based GPU collector. The work focused on reproducible deployment, operational visibility and measurements rather than just a working chat endpoint.</p>
<h2 class="anchor anchorTargetStickyNavbar_SAay" id="architecture">Architecture<a href="https://blog.shayanradi.com/blog/homaai-inference-monitoring#architecture" class="hash-link" aria-label="Direct link to Architecture" title="Direct link to Architecture" translate="no">​</a></h2>
<p>The model exposes an authenticated OpenAI-compatible API on loopback. Prometheus scrapes vLLM and node-level metrics; Node Exporter's textfile collector ingests GPU memory, utilization, temperature and power. Grafana runs on a private network interface, with internal-only access.</p>
<h2 class="anchor anchorTargetStickyNavbar_SAay" id="key-implementation-decisions">Key implementation decisions<a href="https://blog.shayanradi.com/blog/homaai-inference-monitoring#key-implementation-decisions" class="hash-link" aria-label="Direct link to Key implementation decisions" title="Direct link to Key implementation decisions" translate="no">​</a></h2>
<ul>
<li class="">Model: <code>google/gemma-4-12B-it</code> with <code>bfloat16</code>, 32,768 maximum model tokens.</li>
<li class="">Runtime: vLLM <code>0.32.0</code>, systemd autostart and API-key authentication.</li>
<li class="">Metrics: vLLM <code>/metrics</code> plus Prometheus, Node Exporter and <code>nvidia-smi</code> collector.</li>
<li class="">Deployability: an explicit-approval bootstrap script with preflight, dry-run, apply and verification modes.</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_SAay" id="measured-performance">Measured performance<a href="https://blog.shayanradi.com/blog/homaai-inference-monitoring#measured-performance" class="hash-link" aria-label="Direct link to Measured performance" title="Direct link to Measured performance" translate="no">​</a></h2>
<table><thead><tr><th>Test</th><th>Observation</th></tr></thead><tbody><tr><td>Short burst of 64 concurrent requests</td><td>All 64 completed; p95 25.35 s</td></tr><tr><td>Sustained 32-worker closed-loop test (~10 min)</td><td>1,820 completions, zero observed failures</td></tr><tr><td>Sustained output throughput</td><td>291 tokens/s</td></tr><tr><td>Sustained end-to-end latency p95</td><td>10.92 s</td></tr></tbody></table>
<p>These are <strong>specific short-prompt benchmarks</strong>, not a general production traffic guarantee. Longer RAG inputs, streaming, model contention and arrival patterns need separate acceptance tests.</p>
<h2 class="anchor anchorTargetStickyNavbar_SAay" id="lessons-learned">Lessons learned<a href="https://blog.shayanradi.com/blog/homaai-inference-monitoring#lessons-learned" class="hash-link" aria-label="Direct link to Lessons learned" title="Direct link to Lessons learned" translate="no">​</a></h2>
<p>The hardest issues were practical: Python headers for Triton compilation, prebuilt FlashInfer sampling kernels, restricted access to container registries, and Linux file permissions in non-root monitoring containers. Switching to an accessible container mirror and fixing only non-secret YAML/dashboard file permissions stabilized the monitoring stack.</p>
<p>For configuration details, see the <a class="" href="https://blog.shayanradi.com/docs/homaai">HomaAi technical documentation</a> covering architecture, monitoring queries, operational checks, security boundaries and benchmark methodology.</p>
<p><strong>What is not complete:</strong> all-panel browser verification for Grafana v4, a clean-VM bootstrap acceptance run, long-context RAG tests, Phoenix-to-vLLM tracing, alerting, dashboard v5 and backend cutover.</p>
<p><strong>Hardware caveat:</strong> the collected report names an RTX 4090 while also reporting approximately 48 GiB of GPU memory. We have not resolved that inventory mismatch, so the hardware SKU should not be treated as confirmed.</p>]]></content:encoded>
            <category>LLMOps</category>
            <category>vLLM</category>
            <category>Grafana</category>
            <category>Prometheus</category>
            <category>Infrastructure</category>
        </item>
    </channel>
</rss>