mitogen_parallel — single-host role parallelism

A dotpro apply runs against one host, and Ansible runs that host's tasks serially. As the collection grows past ~20 roles, more independent work (archive extraction, templating, config writes) queues behind a single serial timeline while the machine's other cores sit idle. dotpro apply --parallel tried multi-process role fan-out and was retired: the dpkg lock capped it near 1.2x and the orchestration (shared become, interleaved output, fact races) cost more than it returned. The fast-apply work then claimed the two fat buckets a different way, single-process: async prefetch overlaps the network fetches, and the merged apt install collapses every role's package install into one dpkg transaction. This doc records what a real single-host parallel strategy would take, why it is a core-engine effort rather than a plugin, and the fallback that already works, so the idea is judged against evidence instead of relitigated.

Packaging

If it is built, this ships as its own package in the dotfilespro ecosystem (a standalone strategy-plugin collection, e.g. dotfilespro.parallel), not folded into dotfilespro.roles. The strategy is general-purpose execution-engine machinery with no tie to any role; its release cadence and its ansible-core / mitogen compatibility matrix differ from the roles'; and keeping it separate lets other Ansible users adopt it without pulling in this collection. The roles collection would depend on it optionally and fall back to mitogen_linear when it is absent, the same graceful-degradation the CLI already applies to mitogen itself.

The mechanism gap

forks: N already spreads work across N cores, but its unit is (host, task): the linear strategy dispatches one task across N hosts at once and never two tasks of one host concurrently. Intra-host parallelism means changing the unit from host to role-subgraph, which the strategy layer does not model at all.

Transport versus engine

Mitogen's transport could carry it. It keeps persistent, multiplexed contexts per host, routes concurrent RPCs, and has a fork service for child interpreters, so it would happily run two roles against one box at once. The blocker sits above the strategy plugin: ansible-core's results collector, VariableManager, handler notification, and changed/failed aggregation all assume serial per-host completion. A strategy plugin controls the dispatch loop but not that pipeline, so mitogen_parallel is a partial rewrite of the execution engine, not a drop-in strategy like mitogen_linear / mitogen_free.

The correctness landmine

Ansible tasks carry implicit data dependencies through facts and registered vars: role B's when: can depend on a fact role A set, with nothing declaring it. A scheduler cannot infer independence statically, so it must either add a surface for the user to declare which roles or blocks are independent (and own the bugs when they get it wrong) or accept nondeterministic runs. This safety gap, not implementation effort, is why no such strategy ships today.

The fallback that exists today

Map each unit of work to a 127.0.0.x pseudo-host and set forks: N: Ansible treats them as N hosts on one box, so the fork pool spreads them across cores. That is real single-host CPU parallelism with no new strategy. The costs are the same ones that retired --parallel: shared become handling, interleaved output, no fact sharing between the pseudo-hosts, and the repo-gated apt installs (terraform, kubectl, docker) still serialize on the dpkg lock. Best fit is the download-and-extract-heavy, become-light role work.

ROI

The fat buckets are already handled, and a measured cold full converge confirms there is little left to chase. Setup: 18 roles, default apply (async + merged apt install), mitogen_linear, a clean ubuntu:26.04 container. Of 146s wall clock, bucketed from profile_tasks:

• apt / dpkg: ~77s (52%), already one merged transaction behind the dpkg lock, which no strategy can parallelize.
• network: ~39s (26%), already overlapped by async prefetch.
• local CPU/IO (extraction + file writes): ~18s (12%) total, but only ~11s of it is in tasks over 0.5s (nine archive extractions across roles: gcloud, yq, k9s, helm, glab, gh, kubeseal, difftastic, the font cache). The other ~7s is 137 sub-0.5s file writes, where parallelism saves nothing because dispatch is already near-free under mitogen.

So the entire slice intra-host parallelism could touch is ~11s of disk-write-bound extraction on a cold run. Extraction does not scale linearly with cores (parallel unzips contend on disk bandwidth), so the realistic ceiling is collapsing that ~11s of serial extraction toward the longest single one (~3.4s): about a 5% cold-run win, and roughly 0% on a warm run, where every extraction is a creates:-guarded no-op. A core-engine rewrite for a 5%-cold / 0%-warm ceiling is not worth it today. If a single role's extraction ever dominates, an async block already overlaps it single-process, the cheaper lever.

Open questions

• Measured once (see ROI): ~11s of disk-bound extraction, a ~5% cold ceiling, so this stays parked. Re-measure when the role set grows enough extraction-heavy roles to move that number, since that is the only thing that changes the verdict.
• If it is, does the pseudo-host + forks fallback capture enough of it to ship despite the become / output / fact-sharing caveats, or does the remainder need a real strategy?
• What would a "declare independence" surface look like (a per-role flag, a block-level marker) that stays both safe and readable?
• Does leaning further into async (more of each role's heavy work launched as background jobs and joined late) dominate a parallel strategy on both effort and risk?