fix(upgrade): finish the restart on hosts with no service manager

[jiunbae]

Why

muxa upgrade builds the new binaries, then asks the OS service manager to restart the daemon. When there isn't one that works, it stops at "manual restart required" — the new code is installed but the running daemon keeps serving the old build until a human remembers to restart it.

A container with systemd installed but not running hits this on every upgrade. which systemctl succeeds (the binary is right there in /usr/bin), so the path looks supported right up until the call fails:

$ systemctl --user restart muxad
Failed to connect to bus: No medium found

This bit me twice today on my own box — both times it presented as "muxad died", which is a misdiagnosis the upgrade flow shouldn't be inviting.

What

Fall back to replacing the daemon ourselves: ask the daemon on the socket this upgrade is already managing to drain, wait for it to actually let go, then start the new one detached.

This is not the pkill fallback that was removed in 68cfba4, and the difference is the whole design. That one hunted for processes named muxad and signalled whatever it found. This one addresses exactly one daemon through its own control channel, identified by the socket the user pointed us at. A muxad belonging to another socket, checkout, or user is never a candidate.

To make that possible, shutdown joins the IPC protocol:

• Routes to the same broadcast SIGTERM already fires, so the drain path is unchanged and in-flight handlers still finish.
• Opt-in: a Server without a trigger answers with an error instead of exiting, so a caller can distinguish "declined" from "old daemon that never knew the method".
• The socket is 0600, so anyone who can call it is already the user who owns the daemon — the same boundary that already lets ingest forge arbitrary agent events. It doesn't widen the trust model.
• Advertised as a hello capability, and documented in PROTOCOL.md. Additive, and the protocol is explicitly unstable pre-1.0.

Notes

The detached spawn reuses init's shell-based nohup … & disown rather than a new pre_exec + setsid: the workspace sets unsafe_code = "forbid", and setsid(1) isn't on macOS anyway. init/apply.rs already had exactly this logic, so it moved to init::util and both callers share it — no copy to drift.

Two dry-run tests asserted the string "manual restart required", which is the behavior this removes; they now pin the meaning instead. Their !contains("pkill") / !contains("SIGUSR1") assertions are deliberately kept — they're the guard against regressing 68cfba4, and this change must not.

Verification

• New unit tests: shutdown drains the daemon; a server without a trigger refuses; a refused shutdown leaves it serving.
• Drove a real daemon on a scratch socket: {"ok":true} → exits → releases the socket → logs it.
• Drove this host's actual scenario: confirmed systemctl --user restart muxad fails here, then the socket fallback replaced the daemon (pid 1305768 → 1306054).
• Safety property checked: the production daemon on a different socket kept serving 46 agents, untouched, across all of the above.
• cargo test --workspace (429 + 11 e2e), workspace clippy -D warnings, fmt --check all clean.

🤖 Generated with Claude Code

[jiunbae]

Marking draft. Five review rounds in, this is not converging, and I'd rather say so than keep patching.

What's solid

The daemon-side design is right and demonstrably works: restart over IPC makes muxad drain and exec() its own argv[0] — same pid, same argv, same socket. Verified live by swapping a binary under a running daemon: same pid came back, /proc/PID/exe went from … (deleted) to the replacement.

That design deleted whole defect classes rather than fixing them — no two-daemon overlap (one process), no lost argv, no socket rediscovery, no "does a supervisor own this?".

What isn't

The CLI cannot tell whether the daemon came back. Every "is it up?" check so far has been unable to distinguish the old daemon from the new one:

• Round 4 found the probe was connect().is_ok(), which succeeds into the listen backlog with nobody accepting — so it reported the dying daemon as back, and RestartOutcome::Failed could never fire.
• Round 5 found my fix's own justification was wrong. I claimed "a round-trip can only be answered by a process that is accepting". But Server::run's select! is random-order, so the loop can accept once more after the shutdown broadcast — the dying daemon answers the round-trip too.

The real problem underneath both: the probe has no identity. And re-exec preserves the pid, so a pid can't supply one either. Without knowing which daemon answered, every verdict built on it is a guess — which is why round 4's fix produced round 5's defects (a 3s verify that now fails the command on a daemon that starts healthily in 4s).

The fix, for whoever picks this up

Give the daemon a generation: an env var incremented across the exec (MUXA_RESTART_GENERATION), surfaced in hello. Then "did it come back" stops being a timing guess and becomes did the generation advance — the one question that actually answers it. Every timeout in the current code (3s vs 15s vs 5s, all mutually inconsistent) collapses into that.

Round-5 findings worth keeping regardless:

• NotRunning falls through to a ManualRequired message asserting "muxad is still running the previous build" when nothing was running.
• The re-exec flag is re-armable after SIGTERM.
• The capability-gate test never exercises the gate it names (the mutation survives).
• PROTOCOL.md has the drain/reply ordering inverted.

Not merging this as-is. The daemon half is worth salvaging; the CLI half needs the generation counter before it can honestly claim anything about the outcome.