Local model serving

Argus local-executor runs reach the 5070 Ti through LiteLLM, not by calling llama.cpp directly from the runner. These facts were measured on 2026-08-23 and should be updated when the homelab serving chain changes.

Serving chain

Layer	Host	Port	Name or alias	Role
Argus runner	pve-builder	n/a	backend=opencode or gateway backend=claude	Executes the harness and calls LiteLLM.
LiteLLM	pve-llm-infra / 192.168.1.28	4000	local-30b	OpenAI/Anthropic-shaped gateway alias.
LiteLLM model entry	apps/llm-infra/litellm-config.yaml	n/a	hosted_vllm/qwen3.8-27b-q3	Bridges the alias to the hosted local server.
llama-swap	pve-gpu / 192.168.1.27	8003	qwen3.8-27b-q3	Starts llama-server for the q3 Qwen3.8 model.

local-30b is the Argus-facing model id. The backing model is the Qwen3.8-27B q3 entry served by llama-swap on the 16 GB 5070 Ti. The agent alias carries chat_template_kwargs: {enable_thinking: false} so Qwen3.8's hybrid thinking is off per request (a thinking model that emits its tool call inside the reasoning block then stops is the failure class this fixes); the raw alias qwen3.8-27b-q3 keeps thinking on for interactive use.

q3 llama-swap flags

The qwen3.8-27b-q3 entry in homelab-apps gpu/llama-swap/models.yaml uses these llama-server flags:

--ctx-size 98304 \
  -fa on \
  -ctk q4_0 \
  -ctv q4_0 \
  -ngl 99 \
  -ub 512 \
  --jinja \
  --chat-template-file /etc/llama-swap/templates/qwen3.8-tolerant-system.jinja \
  --parallel 1 \
  --cont-batching \
  --spec-type draft-mtp,ngram-mod \
  --spec-draft-n-max 2 \
  --cache-reuse 256 \
  --reasoning-budget 5000

The context limit is 98,304 tokens. KV cache uses q4_0 for both keys and values. The chat template is the tolerant Qwen3.8 system template at /etc/llama-swap/templates/qwen3.8-tolerant-system.jinja. The trailing flags are native MTP speculative decoding (no draft model; the MTP heads cost no VRAM), prefix KV-cache reuse across requests, and a 5,000-token cap on chain-of-thought.

Measurements

All numbers in this section were measured on 2026-08-23 against the q3 server above.

Measurement	Value
llama-server VRAM in use	15,576 MiB of 16,303 MiB
Prompt load	61,863-token prompt plus 256 tokens in 65.8 s
Decode throughput	about 68 tok/s for a short-prompt 256-token generation
Configured context	98,304 tokens

Harness base-request sizes, also measured on 2026-08-23:

Harness shape	Prompt tokens
Claude Code, no MCP servers	29,696
Claude Code with the MetaMCP hub-build namespace, 39 tools	39,608
Claude Code with the host default MCP config	99,846
opencode base request	9,255

agents/hub-build/SKILL.md alone is about 68 KB, roughly 17,000 tokens, so skill choice and MCP namespace selection matter as much as model context.

The opencode runner configs declare limit.context 98304 for the q3 aliases and compaction: {auto, prune, reserved 10000} so a long session compacts instead of overflowing the 98,304-token context.

Deploying homelab-apps changes

Komodo only auto-deploys the llama-swap stack on push when compose.yaml changes. A change limited to models.yaml is not enough to trigger that auto-deploy, but llama-swap hot-reloads its config on a 2 s poll, so a models.yaml-only change does not need a container restart.

For a models.yaml-only q3 change:

1. Merge or push the homelab-apps change.
2. In Komodo, deploy the llama-swap stack (Resource Sync will not pick up a models.yaml-only change on its own).
3. Smoke the path through LiteLLM using the local-30b alias before sending an Argus build lane through it.