Architecture

One Python process holds the page, the JSON API, the /v1 gateway, the poller and reconcile. Every model runs outside it, as its own transient systemd unit. models.toml is the only place a model's identity lives.

clients ──► servedeck  (one uvicorn process, 127.0.0.1:8010)
              ├── /v1/*          gateway: name → port, policies, streaming
              ├── /api/*         control: state, mutations, wire, doctor, SSE
              ├── /api/server/*  the v1 page's controls (legacy_page.py)
              └── /              the page (servedeck/web/)
                      │ systemd-run --user --unit=model-<key> --collect
                      ▼
              model-<key>.service ──► vllm serve   (its own cgroup)

servedeck runs from ~/.config/systemd/user/servedeck.service (Type=exec, Restart=on-failure, 3 starts per 60 s, KillMode=process, TimeoutStopSec=15) and starts no model itself.

The one process, and what its death costs

Page, API, gateway, poller and reconcile share one event loop. That buys one thing worth having — no second control plane to disagree with the first — and costs a single point of failure: when the process dies every client loses the only URL it knows, while the models keep serving on their own ports and notice nothing. Recovery is servedeck coming back and re-reading reality.

Two rules keep the loop alive. Nothing in a request handler blocks: systemctl, journalctl and nvidia-smi run under asyncio.to_thread and request paths read only the poller's snapshot, because a coroutine parked on a subprocess stalls the gateway mid-stream. Reconcile runs only once our own port answers as us: _reconcile_after_bind polls /api/health until its instance matches this process's id (60 s ceiling, 0.25 s interval). uvicorn runs the ASGI lifespan before bind(), and on 2026-09-11 a servedeck that lost the port restarted 67 times in about six minutes (systemd's restart counter, 21:36 to 21:42 IST), each lap running its lifespan reconcile before any bind; a status-code check cannot tell a rival servedeck's good 200 from its own.

Registration order in create_app is load-bearing — /api/* (including legacy_page.register), then the gateway, then the page — because /v1/{path:path} and /{asset:path} are catch-alls and Starlette matches in registration order. A same-origin middleware refuses writes to /api/* from a foreign Origin, since mutations take their key in the path and no body and so are CORS simple requests any browser tab could send; /v1 is not covered and callers with no Origin are unaffected.

The registry is the source of truth

models.py loads and validates models.toml and renders one entry into a vllm serve argv and env. render_argv(util, port) is pure, which is what lets tests/test_models_golden.py compare it byte-for-byte against a legacy launcher's dry run; resolving ctx = "native" needs the hub cache and is a separate call. Utilisation lives in control.compute_util, not here — one implementation, not two: a main model gets floor2((free_mib - margin_mib) / total_mib), unless models.toml pins the util it is proven at, in which case a card that cannot fit that value is a refusal naming the holder rather than a quiet launch at a different KV budget. Derived flags follow the same rule: GLM's --kv-cache-memory-bytes is ctx_tokens * kv_cache_bytes_per_token, so context and KV cap cannot drift.

Models as transient units

units.py translates only: build an argv, run it, parse it back. Never shell=True; only model-<key> and the suite's sd-test-<name> are accepted as names. The point of transient units is cgroup separation — the model sits under the user manager, not under servedeck, so restarting or crashing servedeck cannot take it along and there is no KillMode to get wrong. v1's in-process launcher took the model down with the dashboard on every restart.

--collect is required and is a trap: it unloads a failed unit at once, after which systemctl show returns an unknown unit's defaults (ActiveState=inactive Result=success), so the naive check reads "clean stop" for every crash. Every judgement in control.py is written around that, and failure reports come from the journal, which outlives the unit.

start() preflights what otherwise costs an outage rather than a second: host RAM against host_ram_gib (no swap here, so overshooting OOM-kills the session), a refusal when --cpu-offload-gb and --kv-offloading-size would both be passed, and a refusal when the port already has a listener, naming the pid. A start that never became ready is cleaned up with units.stop, not control.stop, so the operator's recorded intent survives.

Readiness

The four journal markers (Loading weights took, GPU KV cache size:, Capturing CUDA graphs, Application startup complete.) drive the progress bar only. Truth is GET 127.0.0.1:<port>/v1/models answering 200 with one of that model's served names, so a boot that skips a marker still becomes ready. The journal follower starts before the unit, anchored two seconds in the past, because journalctl --since has one-second granularity.

Adoption

control.live() lists every model-* unit, joins each with a port probe, and marks a unit whose key the registry does not know unknown=True rather than guessing. It then adopts: a registered model with no unit whose port has a real listener (socket table) answering with that model's registered id appears as unit="(adopted)", sub_state="adopted", with the listener's pid — which is how a hand launch, or v1's server at the cutover, becomes visible without being restarted. A foreign process on a registry port is doctor's finding, not a model. control.adopt() is the operator action that writes adopted units into desired state; unknown keys are reported, never adopted, because without a spec there is no slot to file them under.

desired.json and reconcile

state/desired.json is intent, not a mirror. Schema 3 (2026-09-18):

{"version": 3, "main": "flashnext", "residents": ["lfm2"],
 "launch": {"flashnext": {"util": 0.96, "argv": {"--max-model-len": "262144",
            "--max-num-seqs": "16", "--kv-offloading-size": "40"}}}}

launch records the settings of the last boot that actually became ready, so a configuration that failed is never the one a reboot repeats. Without it the allocator was write-only: the tuned utilisation, context, agent count and offload size lived in a running process's argv and nowhere else, so the next restart relaunched at registry defaults with utilisation recomputed from an idle card (0.98 — the value with the OOM-under-concurrency history) and nothing said so. argv holds whole flags rather than named fields, so a control the page grows tomorrow persists without a schema change; a null value removes a flag. Writes come only from control.start/stop/switch/adopt, atomically (same-directory temp file, fsync, os.replace). Missing, empty, unparseable or unknown-version reads as "want nothing" with a warning, never an exception; v2 migrates to v3 in place and silently, v1 is read as main = backend if desired_state == "RUNNING", copied to desired.json.v1 and rewritten, because a file left at the old version re-warned every poll.

reconcile makes desired state true and restarts nothing — a unit that exists is left alone, ready or not, so servedeck coming up cannot kill a model 40 seconds into a 90-second boot. It never stops anything and never writes desired state: a crashed model must stay desired, a stopped one stopped. Each key it does start is started from that key's stored launch values. It blocks for as long as the boots take, hence a task created after the bind.

The poll loop

Every 2 s the poller rebuilds the /api/state document and publishes it. Its blocking half runs in a thread: the offload reaper, routes.refresh() (the pushed liveness snapshot every request path reads), unit start times, nvidia-smi total/free, desired.json, one /metrics scrape per ready model.

Publishing is thread-safe (call_soon_threadsafe); each subscriber's 256-deep queue drops its oldest event, so a slow page loses history rather than stalling the poller. The poll also closes the gap systemd cannot: _recover_desired relaunches a model desired state names while nothing is running — the state --collect plus an exit-0 engine death makes indistinguishable from "nobody wanted it" — three times, ≥60 s apart, one model per poll, with a recovering notice each time, then stops and leaves the notice standing. It relaunches with the stored launch settings, exactly as reconcile does. Until 2026-09-18 it called start() bare, so a recovery lap silently re-tuned the box: an operator restart at util 0.97 / 1 seq / 20 GiB offload came back as 0.98 / 16 / 40, because a launch that requests no utilisation gets one derived from the free VRAM of the card recovery had just emptied — and because that launch requested nothing, it was not recorded either, leaving desired.json reading 0.97 while the card ran 0.98.

Measuring the boot that just succeeded

Every boot that reaches ready is measured, and the measurement is what the capacity panel predicts with. control._record_boot re-reads the unit's journal, parses vLLM's own memory accounting (servedeck/bootfacts.py — the Free memory on device … Desired GPU memory utilization is … line and GPU KV cache size: N tokens), and appends an observation to state/measurements.json.

The stored overhead_gib is a calibration residual, util × GPU_TOTAL_GIB − weights − kv, not a sum of vLLM's printed parts. Only the residual makes the panel reproduce the engine: vLLM sizes its budget against CUDA's device total (94.97 GiB here) while capacity sizes against nvidia-smi's (95.59 GiB), and the residual absorbs that 0.62 GiB by construction. Re-basing capacity onto CUDA's total was tried and reverted — every stored overhead in the repo is fitted against the nvidia-smi basis and already absorbs the same offset, so moving the basis without re-fitting all of them broke six calibrations and improved no prediction.

Two traps, both of which make a written record unreadable rather than wrong:

• The backend tag must be the one the resolver keys on. resolve_inputs filters a repo's observations by the backend discover_models() derives from the hub cache, which here is flashnext where models.toml says build = "qwen38next". An entry tagged with the registry's value is stored, kept, and skipped at read time.
• The context must be a number. Tier 1 is an exact max_model_len match, so an observation recorded with ctx = None can never be found by it.

This existed as dead code for three weeks: append_observation had three tests and no callers, so nothing had been written since 2026-09-02 and the panel replayed a 2026-08-28 measurement of a model that had since gained MTP-3 speculative decoding and the offload connector. Its overhead had grown 4.47 → 5.62 GiB, so every prediction was 1.15 GiB optimistic — enough that the panel answered can_apply: true, findings: [] for a Flash-Next launch at util 0.95 and 262,144 tokens that the engine then refused, 2.5 minutes in, twice. With the loop closed the panel predicts 245,363 tokens where vLLM measured 245,440, and 350,040 where vLLM measured 350,043.

The KV-offload reaper

vLLM's CPU KV offload buffer is /dev/shm/vllm_offload_*.mmap, sized by --kv-offloading-size (40 GiB for Flash-Next) — host RAM, on a box with no swap. vLLM unlinks it only in its graceful cleanup(), so an engine that is SIGKILLed, crashes or loses the GPU leaves the whole buffer pinned until reboot, and each relaunch adds another under a fresh engine id. The reaper runs in control.start and in every poll (a systemd restart never passes through start), publishing an offload_reaped notice with the size freed.

It fails closed. The scan reads /proc/<pid>/maps and /proc/<pid>/fd for every process whose cmdline contains vllm, plus every pid servedeck believes belongs to a live model. If /proc cannot be listed, any one of those processes cannot be read, or servedeck thinks a model is live but cannot enumerate its cgroup's pids, the scan is trusted=False with a reason and nothing is deleted — an unreadable holder is indistinguishable from no holder, and "no holder" means "delete 40 GiB". Reading a sibling engine's maps needs kernel.yama.ptrace_scope = 0, which is also what Flash-Next's PLE handoff needs; doctor reports it as a host row. See HOST.md.

The state directory

state/, resolved from the checkout (the checkout is the deployment).

v1 leftovers, referenced by no code in servedeck/ and readable as history only: server.json (+ .bak-dead-glm53-pid214759), ack.json, extra_args.json, history.jsonl, boot_logs/, desired.json.bak-glm53-stale.

The legacy page surface

servedeck/legacy_page.py exists because the page is v1's page: v1's web/index.html + app.js + style.css as at tag pre-cutover-2026-09-17 is what the owner uses and what is served, and the owner rejected a redesign. The module builds v1's shapes from the v2 runtime — upstream, vllm, sizing, boot, supervisor, config, gpu_v1, uptimes blocks added to the v2 state document rather than substituted, a telemetry event per poll, the /api/models rail with disk sizes, /api/disk, /api/capacity/estimate and the /api/server/* controls — so the CLI, doctor and the tests keep reading the v2 document unchanged. None of the arithmetic is reimplemented: sizing calls parallelism, the estimate calls capacity and kvcalc, disk sizes come from disksize and discovery. A failure in the legacy blocks is caught so it cannot take the v2 state down.

The mutation contract

A mutation answers 202 {"accepted": true, …} and reports over SSE; the blocking Control call runs in a thread. Anything decidable from the snapshot is refused synchronously, so no client has to watch a stream to learn it was rejected: 404 unknown_model, 409 busy, 409 already_live, 409 main_slot_busy, 409 not_main_slot, 409 not_live, 409 ctx_unresolved. The reason vocabulary is exactly control.Refusal's, so a refusal decided from the snapshot and one decided against reality are indistinguishable to a client; refusals that need reality (not_enough_vram, not_enough_host_ram, conflicting_offload, port_busy, env_scan_failed, gpu_unavailable, start_failed, boot_failed) arrive as notices. Single-flight has two layers and the outer one matters: _claim sets rt.busy in the handler, immediately after a clean precheck with no await between, which is what makes check-then-claim atomic — setting it inside the task let two Applies in one tick both pass and both restart the model. _run_mutation then holds rt.lock for the whole operation and clears busy only after publishing the outcome, because clearing it first left a tick where /api/state said nothing was running while the notice stream was about to say ready.

Module map

Gone in v2 and not returning: config.py, paths.py, supervisor.py, procctl.py, phases.py, updetect.py, shellconfig.py, legacy.py, history.py, logtail.py, preflight.py, smoke.py, events.py, and registry.py (now discovery.py, because the model registry owns that name).

Debt

Two mutation API families. /api/models/{key}/*, /api/switch/{key} and /api/adopt key on the registry key, precheck into typed refusals and take no body. /api/server/{start,restart,stop,adopt} key on repo_id + backend, return untyped {"error": "…"}, and are the only route that accepts the allocator's util and argv overrides — which is why they cannot be deleted while v1's Configure panel is the page. /api/server/adopt also answers synchronously where /api/adopt is 202 + SSE. Both families share _claim/_run_mutation, so they cannot race, but they are two surfaces with one meaning.

The single point of failure: nothing serves /v1 while the process is down, and every client knows only that URL.

Unwired instrumentation. telemetry.py and xid_watch.py are written and tested, but nothing starts the sampler (the telemetry SSE event is legacy_page's strip, a different thing) and cli_gpu.py is not registered in cli.py, so servedeck gpu-log does not exist. The reason they were written — 2026-09-12, the GPU fell off the bus and nothing on the box had recorded its temperature — is still unaddressed on the running box.

needs_tty was retired from models.toml on 2026-09-18. It survives as a Model field defaulting to False, so an older registry still loads; nothing reads it and /api/state no longer publishes it.

Where the older documents went

Six documents were deleted on 2026-09-18 once this set absorbed them. They are in git history, and git log --diff-filter=D --name-only -- docs/ finds the commit; the last revision of each is git show HEAD~1:<path>. Test docstrings that cite SPEC.md §3 or REDESIGN-2026-09-12.md §2.7 mean those files at that revision.

Keys and defaults: CONFIGURATION.md. Running it: OPERATIONS.md. Why things are the way they are: DECISIONS.md.