Sheltie on real repositories

We measured what Sheltie would have done on 23 popular open-source repositories over 90 days (landings from 2026-07-21 to 2026-10-19). We compared it with what each project does today, not with a strawman. This page reports what we found, including where Sheltie does not help, where it would have let breakage through, the bugs in Sheltie the study exposed, and what we could not prove.

Every footprint here is the exact one Sheltie’s adapters compute for that commit. The queue results come from replaying each project’s real arrival times through Sheltie’s scheduler. They are simulations, not production runs.

The scripts, the bench and every number are in research/real-world.

Download this report as a PDF

Summary

The study also found problems in Sheltie, fixed before these results were computed (see What the study changed in Sheltie).

The projects and how they merge

We classified 26 projects from their GitHub rulesets, Actions events and pull request timelines. Three have no GitHub pull request merge flow to compare against:

• golang/go reviews on Gerrit;
• guava is exported from Google’s internal repository;
• spring-boot’s maintainers merge locally.

Whether the queue runs more than pull request CI decides how to read the results:

Queue projects: the real queue against Sheltie

How it was measured

• Ready time is when a pull request entered the merge process:
  • for GitHub merge queues, its last AddedToMergeQueue event;
  • for bors, the last “approved by” comment;
  • for traefik, when the merge label was added;
  • for Tide, when lgtm, approved, passing presubmits and the removal of blocking labels were all in place.
• Landed is the pull request’s mergedAt. (A merge-queue commit’s own timestamp is when its build started, not when it landed.)
• CI per build is the median wall time of a queue build:
  • for GitHub merge queues, merge_group runs grouped by commit, from first start to last finish;
  • for bors, its “Duration” line;
  • for gradle, whose CI runs on TeamCity, from pending to final of the gating status GitHub Merge Queue Check Pass on each landed commit.
• Sheltie replays the same entries at the same times through the scheduler Sheltie runs (plan_epoch), with the same CI time per run:
  • Each change counts as tested on main as it was when its own pull request CI last started.
  • If anything that landed after that writes what the change reads, Sheltie rebuilds it on the new head. If it writes what the change writes, the change goes to the merge engine.
  • Otherwise it lands at once.
• Like for like: the comparison uses only pull requests the real queue never ejected, because the replay assumes every build passes.
• Rebase merges: trino merges pull requests by rebasing several commits onto main. Each such pull request counts as one change, with one footprint from its first parent to its last commit.

Waiting

Waits are median / 90th percentile / total.

Read with care:

• rust-lang/rust (bors) runs 11 CI jobs on pull requests and 82 in bors. Landing at once would skip 71 job configurations, so its 88% is not a result Sheltie could deliver as is. Also, 222 of its 440 landings are rollups that bundle 2,151 pull requests; the replay sees a rollup as one change.
• traefik’s 37 changes in 90 days never overlapped in flight. The bot already lands them within minutes, so the saving is small in absolute terms (9 h 37 min in total).
• kubernetes is not in the table. Its waits are dominated by holds and policy (up to 35 days), and Prow’s CI is not visible on GitHub, so we have no CI time to replay with.
• bevy saves least because its changes overlap often. Of 592 changes, Sheltie would send 115 to the merge engine, rebuild 92 and build 57 together with a conflicting change.

CI runs

• Real queues build every entry, and rebuild entries behind every failure. Sheltie builds a change again only when something that landed after its CI overlaps it.
• For gradle the real count is a lower bound: one build per queue entry. Builds re-run after an ejection ahead of an entry are not visible.
• The last column is the fair cost for projects whose queue runs more than pull request CI: one full CI run per change plus Sheltie’s rebuilds. It is about the same as the real queue.

What Sheltie would have done with each change:

Would landing at once have let breakage through?

Queue failures, split by what Sheltie would have done with the same pull request:

Except at trino, queue failures hit pull requests that Sheltie would land at once as often as, or more often than, those it would rebuild. At trino and gradle, about a quarter of the pull requests Sheltie would land at once failed in the queue at least once. Footprint overlap is a weak predictor of which queue build fails.

Of the 223 pull requests that failed in the queue and that Sheltie would land at once, 165 went back into the queue with no new commits and merged. Those failures were most likely flaky or caused by another entry in the same build; we did not check each one. To find the rest, we took every pull request that failed in the queue, that Sheltie would land at once, and that got new commits afterwards. For each one we compared the pull request’s own diff at the time of the failure with its final diff, and read the failed jobs and follow-up commits:

• 18 only rebased or merged main; their diff did not change.
• 14 failed checks that run only in the queue, which Sheltie would need on the pull request:
  • 10 are repository-wide checks at gradle (distribution size, accepted API list, version lists, performance baselines). Any change can break these in combination with others, and no footprint captures them.
  • 4 at bevy, whose failing jobs run only in the queue.
• 14 are other gradle test or code fixes after a queue failure. Gradle’s pull request CI is a subset of its queue CI, so the failing test most likely did not run on the pull request, but we did not identify it (gradle’s queue CI is on TeamCity, outside the GitHub API).
• 2 changed trino’s CI workflows, which are only exercised in the queue.
• 1 is a semantic conflict with main (bevy: “Fix example merge conflict”). Footprints should have caught it and did not.
• 9 have a code change whose cause we could not determine: 5 at trino, 2 at rust-analyzer, 2 at bevy.

Projects without a merge queue

How it was measured

• CI is the build and test workflows that run both on pull requests and on pushes to main. Security audits, release, CodeQL, Copilot setup, zizmor, typos, CLA and labeller workflows are left out.
• Stale at merge: something else landed between the start of the pull request’s last CI and its merge.
• Sheltie would rebuild a pull request when one of those landings writes something it reads.
• Extra CI costs each such rebuild one full pull request CI run of wall time. That is an upper bound: Sheltie reruns only affected tests.
• Persistent red episode: two or more consecutive failed runs of a CI workflow on main. The candidates are the landings since the last green run. An episode counts as “with an overlapping landing” when a candidate was stale and something that landed in its gap overlaps it.
• Chance baseline: how many episodes would show an overlapping landing if overlap had nothing to do with breaking. Each candidate is taken to overlap at that project’s overall rate.

Results

n/m means not measurable:

• keycloak runs no CI on pushes to main, so red main is invisible;
• storybook’s main CI is not on GitHub Actions;
• only 38 of okhttp’s 147 pull requests had matching CI suites.

All persistent red episodes, by what their candidates looked like:

• stale base, overlapping landing: Sheltie rebuilds before landing: 83
• tested on the current main: not an interaction: 64
• stale base, no overlap: Sheltie lands it too: 27
• direct push or untested change in range: 2

• Stale merges are the norm, but they do not explain red main. Across 176 persistent episodes, an overlapping landing was in range 83 times, against 85.8 expected by chance. We found no evidence that Sheltie’s rebuilds would have prevented these breakages.
• 64 episodes started at changes tested on the current main. Flaky tests, infrastructure and environment changes break main without any interaction; vitest alone had 26.
• 27 episodes had stale changes with no overlap. If any of them was a real interaction, Sheltie’s footprints would have missed it too.
• Sheltie would cost CI here. It would rebuild 6–87% of pull requests. The largest shares come from coarse footprints: vscode’s root install script puts the whole repository in every read set, and keycloak’s Maven build plugins do the same for its test suite.

Adapter coverage

A footprint is leaky when the change reads something the adapter cannot model exactly:

• a build script;
• a proc macro;
• an install script;
• a build plugin;
• a file outside the module graph.

Its read set then widens to everything that leak could reach. Leaky does not mean serial, but it makes rebuilds more likely. This is the share of landings with a leaky footprint after the fixes below:

The causes that remain are mostly deliberate caution:

• Code that runs at build time: Cargo build scripts and proc macros, npm install scripts, Maven plugins that run code, Python setup.py extensions.
• Languages an adapter does not read:
  • ruff’s Python packaging;
  • pydantic’s Rust core;
  • prometheus’ web UI;
  • files under test/ read at run time.
• rust-lang/rust (95%): syn cannot parse the unstable syntax the compiler uses, so Sheltie claims nothing about those crates.
• bevy: a wasm32-only include_str! of a file that CI generates.

What the study changed in Sheltie

What we cannot prove

What was verified, and how

Reproduce it

research/real-world/README.md lists the steps: survey, collection, footprints, analyses and table generation. results/tables.md holds the tables above, generated from results/*.json.