Why rootless Podman under a system service with User= silently mis-routes logs and swallows READY=1 — and the two flags that fix both.
User=myuser has lingering enabled, rootless Podman finds the user D-Bus and registers a transient scope under user@1000.service. Journald and systemd both attribute events by cgroup path, so container output and READY=1 land on the wrong unit. Adding Delegate=yes to the service and --cgroups=split to podman run keeps conmon and the container inside the service's own cgroup subtree. The --log-driver=passthrough that was added as a workaround can be removed — it never fixed attribution and only disabled podman logs. This is exactly what Quadlet generates automatically.
The service is a system unit (caddy-wg0.service) that runs a rootless Podman container as User=myuser. The split is intentional: the companion socket unit (caddy-wg0.socket) binds privileged port 443 as root and passes the file descriptor into the container, while everything inside the container runs completely unprivileged.
One environmental fact makes this non-obvious: user myuser has lingering enabled. Lingering keeps the user's own systemd instance alive (user@1000.service) even when no one is logged in, and it keeps /run/user/1000 — including the D-Bus socket — mounted at all times. This is the detail that triggers the bug.
This is the service as initially written. Two keys are absent; a third was added as a failed workaround:
[Unit] Description=Caddy for wg0 BindsTo=wg-quick@wg0.service After=wg-quick@wg0.service [Service] User=myuser Group=myuser # Delegate= absent — Podman is not authorised to create sub-cgroups here Type=notify NotifyAccess=all ExecStartPre=podman volume exists caddy-wg0-data ExecStartPre=podman volume exists caddy-wg0-config ExecStartPre=podman network exists caddy-wg0 ExecStart=podman run \ --rm \ --replace \ --name=caddy-wg0 \ # --cgroups=split absent — Podman falls back to user D-Bus transient scope --volume=caddy-wg0-config:/config \ --volume=caddy-wg0-data:/data \ --volume=/home/myuser/.config/systemd/user/caddy-wg0/conf:/etc/caddy:ro \ --network=caddy-wg0 \ --log-driver=passthrough \ # ← attempted workaround; silences podman logs without fixing attribution docker.io/library/caddy ExecReload=podman exec caddy-wg0 /usr/bin/caddy reload --config /etc/caddy/Caddyfile --address unix//run/admin.sock --force
Two independent mechanisms both failed, each pointing at the same hidden root cause:
journalctl -u caddy-wg0 only shows systemd's own start/stop messages. Caddy's actual output — access logs, errors, config parsing — goes nowhere. Switching the log driver to passthrough silenced podman logs too, without helping.Type=notify, NotifyAccess=all, and --sdnotify=container, the service timed out waiting for READY=1. Switching between --sdnotify=conmon and --sdnotify=container made no difference.Both symptoms share one underlying cause: the container's processes end up in the wrong cgroup. Tracing why requires understanding what Podman actually spawns.
Podman is not a simple exec wrapper. It uses a two-process architecture: a launcher that sets everything up and exits, and two long-lived children — conmon (the container monitor) and the container's own init process.
conmon (container monitor) is a small C daemon that holds the container's stdio pipes, manages TTY, implements OOM notification, and reaps the container when it exits. Both conmon and the container are long-lived. The podman run process is only the launcher — it configures namespaces, starts conmon, and exits.
Every Linux process belongs to a cgroup. Systemd organises its units as a cgroup tree: slices are interior nodes, services are leaf nodes, processes live inside their service's cgroup. So you'd expect conmon and caddy to appear under caddy-wg0.service. They don't.
When Podman runs inside a system service as a non-root user, and that user has lingering, Podman finds /run/user/1000 and the user D-Bus socket. Rather than creating sub-cgroups under the current service (which it isn't authorised to do — Delegate=yes is absent), Podman asks the user's systemd manager to allocate a transient scope — a temporary cgroup node that lives only as long as the container. That scope is created under user@1000.service, not under caddy-wg0.service.
Journald does not rely on stdout/stderr file descriptors to determine unit ownership. For every process that writes to the journal, journald reads /proc/<PID>/cgroup, extracts the cgroup path, and maps it to a unit name. When Caddy writes a log line, journald sees the cgroup path ending in run-r3a8f7c.scope under user@1000.service — and that is the unit that receives the log entry. caddy-wg0.service is never consulted.
Switching to --log-driver passthrough only changes where the bytes flow (preventing Podman from buffering them), not which cgroup the writing process lives in. Journald still reads the cgroup path and still gets the wrong answer. As a side-effect, passthrough also disables podman logs.
The mechanism is identical. When Caddy sends READY=1 to NOTIFY_SOCKET, systemd uses SO_PEERCRED to get the sender's PID and reads /proc/<PID>/cgroup. The cgroup path leads to user@1000.service. Systemd credits that unit with the notification. caddy-wg0.service is left waiting and eventually times out — regardless of what --sdnotify flag or NotifyAccess value is set, because those only control forwarding, not attribution.
Delegate=yes alone is not sufficientDelegate=yes grants permission — it instructs the kernel to allow the service's processes to create child cgroup nodes within its subtree, without needing to call back into systemd. But granting permission does not change how Podman behaves. With --cgroups=default, Podman never uses that permission.
In rootless mode, --cgroups=default hands cgroup management to the OCI runtime (crun on Fedora). crun's rootless protocol is unconditional: it contacts the user systemd manager over D-Bus and requests a transient scope. It does this because it assumes it cannot create cgroup nodes in the system hierarchy — which is normally true. The fact that Delegate=yes has changed that assumption for this particular service is invisible to crun; it does not inspect the parent cgroup's delegation state. The result is the same transient scope under user@1000.service, regardless of whether Delegate=yes is present or not.
--cgroups=split changes who is in charge. Instead of offloading to crun, Podman itself takes responsibility for cgroup placement. It reads the current cgroup path — the service's own delegated cgroup — and directly creates two child nodes beneath it using the kernel's cgroup interface. No D-Bus call is made; no transient scope is allocated. The delegation that Delegate=yes established is finally put to use. Both conmon and the container end up exactly where they should be: inside caddy-wg0.service's subtree.
The two flags are therefore not redundant — they address different layers. Delegate=yes is a prerequisite that makes the kernel permissive; --cgroups=split is the instruction that makes Podman act on that permission instead of falling back to D-Bus.
A conceptual picture of each building block. The first two sections are expanded here; deeper detail on journald attribution and sd_notify is available in the collapsible sections below.
A cgroup (control group) is a kernel data structure that groups one or more processes together for accounting and resource control. Every process belongs to exactly one cgroup at any moment. Cgroups form a tree: the root cgroup contains child cgroups, which contain further children.
Systemd uses this tree to organise its units. Slices (.slice units) are interior nodes — they provide grouping but hold no processes directly. Services (.service) and scopes (.scope) are leaf-level nodes where processes actually live. Because journald and systemd's notification routing both derive unit ownership from a process's cgroup path, placing a process in the wrong cgroup subtree silently misdirects both mechanisms.
Fedora uses cgroup v2 (the unified hierarchy). The entire cgroup tree lives under /sys/fs/cgroup/. Each subdirectory is a cgroup node; a process's membership is visible at /proc/<PID>/cgroup as a single path like 0::/system.slice/caddy-wg0.service.
By default, only systemd is authorised to create child cgroup nodes under any unit. Delegate=yes relaxes this for a specific service — it hands ownership of the service's cgroup subtree to whatever process is running inside, letting it create child nodes freely. Without delegation, Podman cannot create sub-cgroups under caddy-wg0.service and must fall back to another approach.
That fallback is the transient scope. A scope is a lightweight systemd unit representing a group of externally-started processes (as opposed to a service, which systemd starts itself). Scopes are created programmatically via D-Bus — the caller asks systemd to track a set of PIDs as a named, temporary unit. Podman uses the user systemd manager's D-Bus to do this, because it finds the user session alive via lingering. The resulting scope appears under user@1000.service, not under caddy-wg0.service.
--cgroups=default vs --cgroups=split--cgroups=default is Podman's out-of-the-box behavior in rootless mode: it hands cgroup management to the OCI runtime — crun on Fedora. crun follows a fixed rootless protocol. It contacts the user systemd manager over D-Bus and requests a transient scope, because it has no way to create cgroup nodes in the system hierarchy as an unprivileged process. Critically, crun does this unconditionally — it does not inspect the parent cgroup to see whether delegation has been granted. Even with Delegate=yes present, --cgroups=default still goes through D-Bus and still places conmon and the container under user@1000.service.
--cgroups=split takes the OCI runtime out of the picture for cgroup placement. Podman itself reads the process's current cgroup (which, inside a delegated service, is caddy-wg0.service's cgroup) and directly creates two sibling child nodes beneath it using the kernel's cgroup interface. No D-Bus call, no transient scope. The delegation that Delegate=yes established is put to use for the first time.
The name "split" describes the resulting topology: conmon gets its own child node, and the container gets its own sibling child node — rather than both sharing one merged node. This separation is not merely cosmetic. Cgroup v2 enforces a "no internal processes" rule: a cgroup that has child sub-cgroups cannot simultaneously have processes living directly inside it. If conmon and the container shared one node and the container then needed to create cgroups for its own internal processes (for example, a container running a full init system), the kernel would refuse. Splitting them into separate sibling scopes keeps each leaf clean and avoids that constraint entirely.
Delegate=yes alone → crun still uses D-Bus → transient scope under user@1000.service ✗--cgroups=split alone → Podman tries direct cgroup creation → fails, no permission ✗caddy-wg0.service ✓
Three changes to the unit file — two additions, one removal. Together they ensure conmon and the container are placed inside caddy-wg0.service's own cgroup subtree, and that podman logs continues to work.
[Service] User=myuser Group=myuser +Delegate=yes # grant ownership of this service's cgroup subtree Type=notify NotifyAccess=all ExecStart=podman run \ --rm \ --replace \ --name=caddy-wg0 \ + --cgroups=split \ # Podman creates child cgroups directly; no D-Bus --volume=caddy-wg0-config:/config \ --volume=caddy-wg0-data:/data \ --volume=/home/myuser/.config/systemd/user/caddy-wg0/conf:/etc/caddy:ro \ --network=caddy-wg0 \ - --log-driver=passthrough \ # remove: was never fixing attribution; breaks podman logs docker.io/library/caddy
--log-driver=passthroughpassthrough bypasses Podman's log buffer and wires the container's stdout/stderr directly to the service's inherited file descriptors. It was added as an attempt to make journald attribute log lines to caddy-wg0.service — but journald's attribution is decided by reading /proc/<PID>/cgroup, not by who holds the write end of the fd. As long as the container was in the wrong cgroup, passthrough changed nothing about where logs landed. It also had a concrete downside: it disabled podman logs caddy-wg0 entirely, since Podman no longer maintains a log buffer. Once the cgroup is fixed with the two flags above, the default journald log driver attributes logs correctly, and podman logs works again.
Grants caddy-wg0.service ownership of its cgroup subtree. The kernel is instructed to allow the service's processes to create child cgroup nodes directly — without calling back into systemd or the user D-Bus. This is a necessary prerequisite, but on its own it does not change Podman's behaviour: --cgroups=split is still needed to make Podman use this permission.
Tells Podman to take cgroup placement into its own hands rather than delegating to crun. Podman creates two sibling child cgroup nodes directly inside the delegated service cgroup — one for conmon, one for the container — using the kernel interface. No D-Bus call is made. This is what actually moves the container into caddy-wg0.service's subtree.
Remove it. It was added trying to fix log attribution, but journald attributes by cgroup path, not by file descriptor lineage — so it never helped. It also disabled podman logs. With the cgroup fixed, the default journald log driver works correctly and podman logs is restored.
systemctl daemon-reload && systemctl restart caddy-wg0.socket, confirm the container landed in the right cgroup:systemctl show caddy-wg0.service -p ControlGroup -p DelegateControlGroup should read …/system.slice/caddy-wg0.service. If the path still contains user.slice, Delegate=yes may not have applied — double-check that daemon-reload ran and that the service restarted cleanly.
Quadlet's generator encodes this fix as institutional knowledge. Inspecting the generated service with systemctl --user show caddy.service reveals the two critical fields:
... ExecStart={ argv[]=/usr/bin/podman run … --cgroups=split --sdnotify=container … } ... ControlGroup=/user.slice/user-1000.slice/user@1000.service/app.slice/caddy.service Delegate=yes DelegateControllers=cpu cpuset io memory pids Type=notify NotifyAccess=all ...
Both --cgroups=split in ExecStart and Delegate=yes in the unit properties are present in every Quadlet-generated container service. Quadlet's user-mode services live natively under user@1000.service, so cgroup placement is inherently correct there — but the generator still adds Delegate=yes and --cgroups=split for proper sub-cgroup organisation (separating conmon from container) and correct sd_notify routing within its own subtree.
User=) is a hybrid that inherits the system manager's authority over the socket but delegates container lifecycle to a user context. The two-flag fix manually replicates what Quadlet does automatically, giving the system unit the delegation it needs to keep the container's cgroup within its own subtree.