Synthetic test fixture for the aftergrid report renderer. The company, people, and numbers are fictional. The attestation flag and the doubled path below are artifacts of the fixture.

Draft. Publication approval has not been verified. Saved evidence passed validation; SQL was not rerun. Recorded reviews are listed below. (attestation 0: source unverified_note is not trusted)

New users who got the onboarding checklist came back more often

For: Non-technical product owner · From: Dana Okafor · Generated 2026-07-20T14:05:00Z · Revision 1

Answer

Yes. New users who saw the onboarding checklist came back within a week more often than new users who did not, and the difference is big enough to keep the checklist.

The one thing that would change this

This is a fair comparison only because the two groups were assigned at random. If the checklist had gone to a particular kind of user (for example only iPhone users), the difference could be about those users, not the checklist. And the gap is one experiment's measurement: the 3 pp bar is a rule the team agreed in advance for acting on it, not a promise that the gap stays that size.

Decision it informs

Whether to keep the onboarding checklist for all new users or remove it. Dana owns the decision. Before the experiment, the team agreed the checklist earns its place if it lifts the share of new users who come back within a week by at least 3 pp.

Evidence

Cause and effect

New users who saw the checklist came back within a week more often than new users who did not: 34.8% compared with 28.5%.

Two horizontal bars. Checklist: 34.8% of 624 new users came back within seven days. Old onboarding: 28.5% of 653. The checklist bar is longer by 6.3 pp.

The numbers behind this chart

Users who saw the checklist came back more often: 34.8% vs 28.5%
arm	signups	retained	retained 7d rate
checklist	624	217	34.8%
control	653	186	28.5%

Who is counted: every person who signed up between the first of June and the twelfth of July (New York time) and was randomly given one of the two onboardings, 1,277 people in all. "Came back" means they opened the app on at least one of the seven days after the day they signed up.

Compared with what: the people who were randomly given the old onboarding at the same time.

The gap is 6.3 pp, above the 3 pp the team agreed on before the experiment. Out of 624 people who saw the checklist, 217 came back. Out of 653 who saw the old onboarding, 186 came back.

Who was counted, and how many came back
Onboarding	New users	Came back within 7 days	Share who came back
checklist	624	217	34.8%
control	653	186	28.5%

Left out: people who signed up before the first of June or after the twelfth of July were not part of the experiment and are not counted.

Limits: "came back" says nothing about what they did in the app. Six weeks of early-summer signups; a different season could behave differently.

Who is counted: Every user who signed up between 2026-06-01 and 2026-07-12 (New York time) and was randomly assigned to one of the two onboardings; 1,277 users in total.
Compared with: Users assigned the checklist compared with users assigned the old onboarding, assigned at random when they signed up.
When: 2026-06-01 to 2026-07-12 (America/New_York)

Who was left out, and the limits of this claim

Users who signed up before 2026-06-01 or after 2026-07-12 were not part of the experiment and are not counted.
"Came back" means opened the app on any of the next seven days; it says nothing about what they did in the app.
Six weeks of signups in early summer; a different season could behave differently.

How this was calculated

ref:retention_by_arm.checklist.retained_7d_rate = 34.8%
ref:retention_by_arm.control.retained_7d_rate = 28.5%
ref:retention_by_arm.checklist.signups = 624
ref:retention_by_arm.control.signups = 653
derived:lift = 6.3 pp
ext:keep_threshold = 3 pp
Definition retained_7d version 2 (metric, recorded as approved)
lift is the difference of after ref:retention_by_arm.checklist.retained_7d_rate against baseline ref:retention_by_arm.control.retained_7d_rate: Checklist rate minus control rate, in percentage points.
keep_threshold is a target from a document dated 2026-05-29: Experiment plan for the onboarding checklist: the checklist is worth its maintenance cost if it lifts 7-day retention by at least 3 percentage points.

Question or flag this claim (reference: fnd_7k2m9q4w1xzb r1 c1)

Pattern, not proof

The difference shows up on phones and not on the web: 8.7 pp on mobile against -1.3 pp on the web.

The same comparison, phones and web separately
Where they signed up	Onboarding	New users	Share who came back
mobile	checklist	476	37.0%
mobile	control	460	28.3%
web	checklist	148	27.7%
web	control	193	29.0%

This split was not planned before the experiment. It is a follow-up look, so treat it as a lead to test, not a settled fact. On phones, 37.0% of checklist users came back against 28.3% of the others. On the web the two groups are small, so the web number moves a lot with a handful of people: 27.7% against 29.0%.

Who is counted: The same users, split by whether they signed up on the iPhone or Android app (mobile) or in a browser (web).
Compared with: Checklist versus old onboarding, inside each platform group. This split was not planned before the experiment; it is a follow-up look. (a follow-up look, not planned before the analysis)
When: 2026-06-01 to 2026-07-12 (America/New_York)

Who was left out, and the limits of this claim

Nothing was left out.
This split was decided after seeing the overall result, so it is a lead to test, not a settled fact.
The web groups are small, so the web number moves a lot with a handful of users.

How this was calculated

ref:retention_by_platform_arm.mobile_checklist.retained_7d_rate = 37.0%
ref:retention_by_platform_arm.mobile_control.retained_7d_rate = 28.3%
ref:retention_by_platform_arm.web_checklist.retained_7d_rate = 27.7%
ref:retention_by_platform_arm.web_control.retained_7d_rate = 29.0%
derived:mobile_lift = 8.7 pp
derived:web_lift = -1.3 pp
Definition retained_7d version 2 (metric, recorded as approved)
mobile_lift is the difference of after ref:retention_by_platform_arm.mobile_checklist.retained_7d_rate against baseline ref:retention_by_platform_arm.mobile_control.retained_7d_rate
web_lift is the difference of after ref:retention_by_platform_arm.web_checklist.retained_7d_rate against baseline ref:retention_by_platform_arm.web_control.retained_7d_rate

Question or flag this claim (reference: fnd_7k2m9q4w1xzb r1 c2)

How we checked

These are separate facts. Passing checks mean the arithmetic and the references hold; they do not by themselves make the conclusion right.

✓Checks passed No user is counted twice. · The two groups are about the same size, as random assignment should produce. · The count of users who came back matches the approved retained_7d definition computed a second way. · The checklist arm is at least 3 percentage points ahead with at least 500 users in each arm.
!Definition retained_7d, version 2, recorded as approved (approval recorded by dana-okafor on 2026-05-28; a recorded lifecycle is not a verified approval).
✓Method review Recorded 2026-09-15 by fixture: hand review by the exemplar author (CrimsonLotus, claude-fable-5-1).
×Publication approval None verified. This is a draft.
✓Data Retained inputs are kept: artifact replay and analysis rerun are possible.
–Covers Signups from June 1 to July 12, 2026, and their app opens through July 19. Nothing after July 19 is included.

No user is counted twice (Check unique_users).
The two groups are about the same size, as random assignment should produce (Check arm_balance).
The count of people who came back matches the approved definition of "came back within a week" computed a second way (Check retained_7d_reconcile).
Definition used: retained_7d v2. An approval by Dana dated the twenty-eighth of May is recorded in the team's records; this draft has not verified it.
Data: a retained copy of the users and app-open events for the experiment window. Anyone can replay these numbers from the saved results and rerun the queries against the retained copy.
Method review: recorded by the exemplar author. No human has approved this Finding for publication; it is a draft.

What would change our mind

If the checklist group were not at least 3 pp ahead of the other group, with at least five hundred people in each group, the checklist did not help enough to keep (Check falsifier_lift, expected to pass).

On a re-check, the main claim is re-tested on the same six weeks of signups with the same definition; it holds if the gap is still at least 3 pp. It cannot be re-tested if the definition changes or if there is no longer a group without the checklist. The phone-versus-web claim is a judgement for the next experiment, not a number to re-test; Dana decides.

Question or flag this Finding

Email Dana Okafor

If the button does nothing on this device, write to dana@loop.example and quote fnd_7k2m9q4w1xzb r1. Your message carries only these ids, never the numbers.

Appendix

Queries: retention_by_arm (primary comparison), retention_by_week_arm (stability by signup week), retention_by_platform_arm (exploratory split), under queries/.
Results: retention_by_arm, retention_by_week_arm, retention_by_platform_arm, under results/.
Retained inputs: inputs/users.csv, inputs/events.csv, hashes in the manifest.
Parameters: analytical timezone America/New_York; window first of June to twelfth of July.