Decisions

These are settled. Do not re-open one without the owner, and do not undo one because its reason is not visible in the code you are editing — the reason is here. Owner rules are not engineering preferences and no measurement overrides them. Open questions are in the last section; everything above it is closed.

Owner rules

Serve every model at its full native context. Never divide the KV pool by an agent count. Owner, 2026-09-11. --max-model-len is a per-request ceiling on a pool the scheduler shares, so capping it buys no capacity. On 2026-09-11 a context slider set the 27B to 110,592 (the 1,595,321-token pool at util 0.95 on vLLM 0.29, divided by 16 agents) and VS Code failed on the first request ("maximum context length is 110592 tokens ... prompt contains at least 110593"), while the full pool held about 6.1 full-length requests at once. Capacity is therefore always shown as a mix ("K at full context + N at typical size"), never pool ÷ agents. GLM is the one exception, pinned below native for a VRAM reason (below).

Reasoning stays enabled on every model, in every config, benchmark and soak load. Owner, 2026-09-09. The owner's work is agentic; the reasoning is the value. A speedup measured with thinking off is not a speedup, and any inventory item whose fix is "disable thinking" is rejected. The empty-content-at-small-max_tokens symptom is a budget problem: the fix is a server-side floor, which is why glm53 carries min_output_tokens = 8192 in models.toml. The gateway raises an explicitly small budget and never lowers one or invents an absent one.

No speedup that changes model output, on any model. Owner, 2026-09-02: "we don't want to lose much quality as it causes hallucination during critical projects." This is what fixes GLM's decode ceiling at roughly 16-17 tok/s. Measured 2026-09-02, not re-measured since: top-6-of-8 router pruning gives 20.48 tok/s against 16.50 (+24%) but visibly degrades output (repetition loops, "Jupiter (planet) (planet) (planet)"), and FP8 for the BF16 dense weights would save about 10 ms/token at roughly 31 dB SNR — a real, mild loss, and no FP4-experts-plus-FP8-dense checkpoint exists anyway.

One model at a time. Co-residents are opt-in, per launch. Owner, 2026-09-16 (models.toml:299-303). Nothing auto-adds the LFM2 resident to desired.json; an operator runs servedeck start lfm2 when it is actually needed. Verified live 2026-09-18: desired.json residents is empty, lfm2-350m.service is disabled and inactive, doctor reports port 8007 not listening.

The 27B's util 0.95 is deliberate and must be made safe, not lowered. --enforce-eager is never the fix. Owner, 2026-09-11, after an Xid 31 eight minutes into a 0.95 boot on vLLM 0.27.1. Also settled the same day: run the latest vLLM and fix forward — no up-front bisect, no retreat to 0.27.1 at util 0.47. CUDA graphs off is not acceptable even as an end state (it cost +23.6% per round in the upstream report of the same fault). If a crash recurs: stop it, keep the evidence, root-cause it.

Never advertise a context that has not been prefilled successfully at that length. 2026-09-02. Both earlier GLM settings (262,144, then the first 327,680) were advertised and never tested near their limit. Applies to models.toml ctx and to every client's maxInputTokens.

No hardware purchases. Treat the kit as fixed. Owner, 2026-08-31, when a 256 GB RAM upgrade was proposed twice as the unlock for GLM's expert cache. It was not worth it even ignoring cost: GLM's best case with 256 GB was about 55 tok/s, while Flash-Next already delivers about 145 tok/s because it fits entirely in VRAM. When a model does not fit, the answer is a smaller model.

Do not change host network configuration. Owner, 2026-08-31, after an agent ran nmcli connection up to A/B a WiFi band during a slow download. Connectivity affects the whole machine and the owner may be using it: diagnose, report numbers and options, stop there. (The A/B was invalid anyway — the 5 GHz association failed, so the link never left 2.4 GHz.)

No sudo from an agent session. CONTRIBUTING.md:13-14, and a standing constraint through the v1-to-v2 cutover. Where privilege is needed, emit a copyable command for the owner. This is why kernel.yama.ptrace_scope=0 is pinned by /etc/sysctl.d/90-servedeck.conf rather than relaxed at launch the way serve.sh:106-131 did it.

Everything binds 127.0.0.1, with no auth. Settled 2026-09-12 with the v2 design. A cross-origin write to /api is refused by an Origin check; /v1 stays open for local clients.

Nothing is pushed without the owner. Standing rule. As of 2026-09-18 main is far ahead of origin/main, so the public repo matches neither the current code nor these docs. Neither is Flash-Next restarted while the owner's agents are using it.

The page's layout is the owner's. Owner, 2026-09-17, twice: a compacted v2 page was rejected ("it was fine, just little flaw, you destroyed it") and so was an unrequested redesign that followed. THE dashboard is v1's, restored byte-for-byte from tag pre-cutover-2026-09-17 and served on the v2 backend through servedeck/legacy_page.py. Do not restructure it. Fix only a flaw the owner names, and render it with Playwright (the pw/ venv in the session scratchpad; google-chrome and firefox --headless both crash silently on this box) and show the screenshot before changing layout. Small correctness fixes on the existing shape are fine.

The request-size histogram's 100-request window stays. Owner, 2026-09-17. Seeding it from vLLM's lifetime buckets so a restart would not empty it was tried and rejected ("you flawed the binning"): those buckets are 20k/50k/100k/200k, so the seeded picture showed coarse bands where watched requests give fine bins. Reverted (1afed99). An empty window after a restart is the accepted cost. Bars are capped at 20 across the drawn axis (reqstats._FINE_BARS).

Critical GLM-5.3 fixes go upstream, and everything is tracked. Owner, 2026-09-02. A genuine upstream defect becomes a minimal PR with a test that fails before and passes after; a design call becomes an issue; our own optimisations (hot-expert cache, DMA staging, grouped prefill, launcher) stay fork-only but are recorded anyway, in ~/Projects/kimi-colab-context. Targeting is per fix: generic defects to vllm-project/vllm, GLM-offload-branch specifics to ZJY0516/vllm. A small number of solid PRs beats many speculative ones, because bad PRs cost the owner reputation. vLLM's own AGENTS.md forbids pure code-agent PRs, requires a human to review every changed line and run the relevant tests, requires an AI-assisted description to state the tests run and their results, and forbids an agent from requesting the ready label — none of which its CI enforces, so run the pinned hooks locally.

Published GLM work keeps its framing. github.com/pctablet505/glm53-flash-single-gpu (public; docs MIT, src/ Apache-2.0 as vLLM-derived; verified live 2026-09-18, clean at a7389b3). Claim "the best result we know of on a single RTX PRO 6000", never "world record"; the one third-party number stays marked second-hand and unreproduced; the 16.50 tok/s at 66 hot slots versus the shipped 14.99 at 54 is stated as not a controlled A/B. The internal brief lives outside that repo and SESSION-BRIEF-* stays gitignored.

Engineering decisions

servedeck is the single control plane; the v1 llm CLI is retired, not maintained in parallel. Decided 2026-09-12 (the v2 design, in git history), cut over 2026-09-17. Two control planes sharing one mutable bash config (local_llm/.config) poisoned it three times, and llm started models outside servedeck, wrote Codex a config pointing at a raw model port with no reasoning_content mirror, and group-killed whatever held a model's port. Since 2026-09-18 ~/.local/bin/llm is a shim that prints "servedeck owns this box" and exits 2, so the habit fails loudly instead of doing damage.

One URL for every client, and main is the name that survives a switch. Every client reaches http://127.0.0.1:8010/v1; the gateway resolves main and local to whatever holds the main slot and publishes them in GET /v1/models (verified live 2026-09-18: main and local are listed alongside the served id and its aliases). Codex replays the model name from its own session state on resume --last, which produced a live 404 storm after a model rename — hence also the rule that old model names stay aliases forever, and that the gateway's 404 body lists every accepted name. No client may be pointed at a model port, because the normalisation that a chat-completions client needs (the reasoning/reasoning_content mirror) lives in the gateway.

Models run as transient systemd user units, launched by systemd-run. Decided 2026-09-12. An improvised process lifecycle (R4) produced a 67-lap crash loop; a transient unit gets the restart policy, the stop timeout and the environment from one place and leaves nothing to clean up.

Restart=always with a start-limit ceiling of 3 per 300 s. 2026-09-18, measured not stylistic: when vLLM's engine core dies the API server returns from serve_http normally and the process exits 0, so on-failure fires zero times and a model whose engine died stays down until somebody notices. systemd's own 5-per-10 s limit can never fire against RestartSec=10, so the ceiling is set explicitly — a model that cannot boot reaches failed after about 30 s instead of restarting every ten seconds forever while holding a port and taking the GPU. TimeoutStopSec=120, and flashnext and glm53 pass --shutdown-timeout 30 so the engine unwinds its pinned host memory (cudaHostUnregister on the 40 GiB KV-offload buffer, then the unlink) instead of being force-killed; systemd killing the unit first is what orphaned 40 GiB of host RAM on every restart.

No restart watchdog that is not gated on a persisted desired state. A timer was disabled on purpose after it resurrected a server the owner had deliberately stopped (2026-08-27 20:13:58) and called reset-failed while doing it, erasing the start-limit backstop: inactive is exactly what a deliberate stop leaves behind. servedeck's replacement is desired.json plus reconcile — the poll relaunches a model that desired state wants and nothing is running, three times, 60 s apart, with a notice each time, then stops.

desired.json records what an operator asked for, including the launch settings of the last READY boot (schema 3, 2026-09-18). Nothing derives it from a probe: a model that crashed must stay desired so reconcile brings it back, and a model an operator stopped must stay stopped. Before schema 3 the allocator was write-only — the tuned utilisation, context, agent count and KV-offload size lived only in a running process's argv, so a servedeck restart or reboot relaunched at registry defaults with utilisation recomputed from free VRAM (0.98 on an idle card, the value with the OOM-under-concurrency history) and nothing said so. Only a launch that became ready is recorded, so a configuration that fails to boot is never the one a reboot repeats.

models.toml is the single source of truth for every model property. Decided 2026-09-12 (R1: a model's identity had been copied into many files, and two renames caused two 404 storms). Public name, aliases, port, context, parsers, flags and env all derive from it; servedeck/models.py:render_argv computes --served-model-name, --host, --port, --max-model-len and --gpu-memory-utilization rather than letting them be written twice. GLM's --kv-cache-memory-bytes is derived from the launch ctx via kv_cache_bytes_per_token = 17200 for the same reason: a pinned product and a movable ctx are two sources of truth for one number, and the page's context control moved only one of them.

Every model launches offline. HF_HUB_OFFLINE=1 in [defaults.env], 2026-09-18. A launch then cannot depend on the network, cannot resolve a revision other than the one on disk, and needs no HF_TOKEN. This flag has moved twice — Flash-Next only, then all four unverified, then back to Flash-Next on review — and is registry-wide now only because the 2026-09-18 audit verified all four snapshots are complete on disk and servedeck doctor grew a weights (<key>) row per model, so a missing checkpoint is reported before a launch instead of surfacing as a 401 minutes into one. The cost: a genuinely missing file fails fast with a local-files-only error. The 27B and GLM have not been launched under v2 yet, so watch the first one.

A main model's utilisation is its proven value, and a card that cannot fit it is a refusal. 2026-09-18. models.toml pins flashnext 0.96, qwen27b 0.95, glm53 0.95, each with the launcher or .config line that proved it. Free-VRAM arithmetic on an idle card yields 0.98 for Flash-Next, which boots and then dies with agents in flight (measured 2026-09-03). Handing a model less than its pinned value would silently change its KV budget, the context that fits and the agent count, so compute_util refuses and names what is holding the card instead. Residents still derive utilisation from their vram_mib budget. In both paths the margin is also the launching worker's CUDA-context cushion: at least 700 MiB of the free pool is left for the process that is about to claim it, pinned by a test.

Host RAM is preflighted per model. 2026-09-18, host_ram_gib in models.toml (flashnext 95, glm53 155). There is no swap on this box and pinned pages cannot be reclaimed, so overshooting is an OOM kill of the desktop, not a slowdown; one happened on 2026-08-28. serve-opt.sh refused to launch below CPU_OFFLOAD_GB + 30 and servedeck had no equivalent until this. Two related refusals: a model may not run --cpu-offload-gb and --kv-offloading-size together, and a start is refused when the model's port already has a listener (named by pid — registry ports get squatted, :8002 and :8003 repeatedly by cvi-scratch servers).

The KV-offload reaper fails closed. 2026-09-18. vLLM unlinks /dev/shm/vllm_offload_*.mmap only in its graceful cleanup, so a SIGKILLed or GPU-losing engine leaves the whole --kv-offloading-size (40 GiB for Flash-Next) pinned in tmpfs until reboot, and every relaunch adds another under a fresh engine id. The reaper deletes a buffer only when it could read every vLLM process's and every live model's mappings; if the scan is not trusted it deletes nothing and logs why. That scan needs kernel.yama.ptrace_scope = 0 to see a sibling engine at all — the same sysctl Flash-Next's PLE handoff needs — so doctor checks it as a host row unconditionally (live value 0, pinned at boot).

GLM is pinned to 327,680 context, not its native 1,048,576. 2026-09-12, models.toml:202-205. config.json says 1,048,576, but GLM's usable ceiling is free VRAM, not the model: long-prompt activations are allocated per request and are not reserved up front. Measured 2026-09-02 (not re-measured): at 60 hot expert slots a 180k prompt died of CUDA OOM; at 54 slots a 304,800-token prompt prefilled (TTFT 205 s), and 233,670 tokens held at 48 slots. The slot cut is nearly free in decode (48 → 14.92 tok/s, 54 → 14.99, inside run-to-run spread). Raise ctx and kv_cache_bytes_per_token's product together, never one alone.

GLM's sanitize_tool_tags is on; capture is off. models.toml:213-245. The streaming tool-call parser can miss a closing tag that straddles a token boundary, and the tag then leaks into a parsed argument — captured live as list_dir({"path</arg_key>": …}), which the client rejects, after which the model invents paths. Stripping markup that is never valid argument text is a repair, not a guess. Capture is consent-only and needs SERVEDECK_GLM_CAPTURE=1 in the environment as well, because bodies contain everything the user typed; the predecessor defaulted it on, writing full bodies into a world-readable scratch directory with no retention policy.

An answerless GLM turn is not retried. glm_policies.py:496-531. A silent retry doubles GPU work on a slot running at --max-num-seqs 1, hides a floor that is set too low, and in the original converted a streaming request into a non-streaming one and rewrote the caller's chat_template_kwargs. What the operator gets instead: the upstream's own empty turn, one journal warning naming the likely cause, and a counter.

Open

The GPU power cap. Live value 375 W, set at every boot by /etc/systemd/system/nvidia-power-limit.service (nvidia-smi -pl 375, verified 2026-09-18; a cap set by hand does not survive a reboot). The card's range is 150-600 W with a 600 W factory default. What is known: the card fell off the bus (Xid 79, then Xid 154) 3 min 52 s after the cap was raised to 600 W under load on 2026-09-11, and 550 W had run about 1.5 h clean before that — but a second Xid 79/154 pair arrived on 2026-09-12 under ordinary load at a stable 450 W, and that fault is a confirmed NVIDIA GSP firmware bug on this card class (GB202, reproduced across motherboards and drivers 570-595). So the earlier conclusion "485 W is validated" is unsupported: a lower cap only lowers the odds. The cost of the current cap is measured (27B, idle box, 39,760-token cold prefill, 45 s decode windows, 2026-09-11): 585 W gives 4.48 s TTFT and 1,467 tok/s of 16-way decode, 476 W gives 5.25 s / 1,359, 388 W gives 6.01 s / 1,221, 300 W gives 7.44 s / 1,024. Single-stream decode is nearly flat (128-144 tok/s) down to 300 W; prefill is the cap-sensitive phase (Flash-Next: 14,302 tok/s at 585 W, 10,717 at 384 W, 6,594 at 227 W). What would settle it: continuous power and temperature logging, since nothing records either and the next fault will be just as ambiguous; and a check of the 12V-2x6 seating and PSU transient response, which needs the owner.

GLM's restore_reasoning. Off, and models.toml:224 says it must stay off until somebody turns it on with evidence. The win is measured and large: clients never echo reasoning_content, so GLM's template renders the in-flight assistant turn as an empty <think></think> and the model reads the thread as fresh; restoring it by tool_call id cut the amnesia mass from 3.32% to 0.00% on the decisive logprob probe, and to 0.00% in all 12 varied conversation states, matching a 4.5%-per-turn greeting rate measured independently over 3,898 real transcript turns (2026-09-02). The output-side mirror cannot substitute: it makes thinking available to a client, it cannot make a client send it back. The reason it is off — "three Xid 31 faults within 40 minutes of it first firing, after 14 hours clean" — is refuted as evidence: the owner pointed out that the 14-hour baseline was an idle period, so the comparison never held. What is left is only that this box has an active, unrelated GPU-fault problem. What would settle it: one session with it on, watching state/telemetry, under load in both arms. Note that /v1/responses gets no GLM repairs at all by design, so a Codex session on GLM would not see the fix either way.

Two Codex homes. ~/.codex/config.toml is wired to the gateway (top-level model = "main", provider servedeck, wire_api = "responses"; [profiles.*] tables are refused by the installed binaries, so the top-level form is the one that works). A second Codex home at ~/Projects/local_llm/.codex/config.toml is the one the CLI at ~/Projects/local_llm/bin/codex actually uses (CODEX_HOME points there). On 2026-09-18 it was repointed at :8010/v1 with model main, so it no longer bypasses the gateway — but it is still a second home, invisible to servedeck wire and to doctor, holding about 1 GB of live session and thread history. Converging means moving that history while no codex is running, which is why it is still open. ats-optimizer (config/app.yaml, base_url http://localhost:8001/v1) does still name a model port: decide whether it becomes a gateway client or stays deliberately pinned — it owns :8000 itself and binds 0.0.0.0, so it is not simply another client.