From 37e1e9b169dda56ad1f2b399e86ba0b6d71f0576 Mon Sep 17 00:00:00 2001 From: Yashika Khurana Date: Fri, 25 Sep 2026 14:10:23 -0700 Subject: [PATCH 1/3] Document the is_holdback flag for long-term holdbacks Adds a Holdbacks page under Experiment Workflow > Designing covering what the flag does, how the rolling analysis window is built, when the first results appear, and the cases where setting the flag has no effect. Also corrects the Jetstream configuration page, which recommended setting start_date and end_date by hand for holdbacks. That is what the flag replaces, and a value there overrides what Experimenter sends. --- docs/data-analysis/jetstream/configuration.md | 6 +- docs/workflow/holdbacks.md | 90 +++++++++++++++++++ sidebars.js | 5 ++ 3 files changed, 98 insertions(+), 3 deletions(-) create mode 100644 docs/workflow/holdbacks.md diff --git a/docs/data-analysis/jetstream/configuration.md b/docs/data-analysis/jetstream/configuration.md index c34837601..0f87dbd53 100644 --- a/docs/data-analysis/jetstream/configuration.md +++ b/docs/data-analysis/jetstream/configuration.md @@ -115,9 +115,9 @@ enrollment_query = """ """ # You can override either or both of the start_date and end_date. -# In conjunction with a custom enrollment query, this can be useful for holdbacks, -# since you don't really care about the period of time before your client upgrades -# to the new version. +# Do not set these on a holdback that has the is_holdback flag set: Experimenter +# supplies the enrolment and end dates for those, and a value here overrides it. +# See /workflow/holdbacks. start_date = "2020-01-01" end_date = "2020-12-31" diff --git a/docs/workflow/holdbacks.md b/docs/workflow/holdbacks.md new file mode 100644 index 000000000..6f5369418 --- /dev/null +++ b/docs/workflow/holdbacks.md @@ -0,0 +1,90 @@ +--- +id: holdbacks +title: Holdbacks +slug: /workflow/holdbacks +--- + +How long-term holdbacks are analyzed automatically using the `is_holdback` flag. + +## What a holdback is + +A holdback keeps a slice of the population on the old experience after a feature ships, +so you can keep measuring the feature's impact over a long period. Unlike a normal +experiment, a holdback enrolls continuously and does not have a planned end date. + +That continuous enrollment is what makes holdbacks awkward to analyze. Jetstream expects +an enrollment period followed by an observation period, and a holdback never closes +enrollment, so there is no window to analyze. Historically the way round this was to +hand-write `enrollment_period` and `end_date` into the experiment's Jetstream config and +update them by hand whenever you wanted fresher numbers. + +## The `is_holdback` flag + +Setting the **is_holdback** flag on an experiment replaces that manual configuration. +Experimenter then synthesizes an analysis window for Jetstream and moves it forward once +a week, so results accumulate on their own. + +The window works like this: + +- The **observation period** is the last 21 days of the window. +- Everything between the experiment's start date and the beginning of that observation + period counts as **enrollment**. There must be at least 7 days of it. +- The window ends on the date of the most recent weekly rerun, not on today's date. The + two are the same on the day a rerun fires and differ on other days. Anchoring on the + rerun is deliberate: anchoring on today would shift the window by a day every time the + API is read, instead of stepping cleanly once a week. + +Each weekly rerun pushes the window out by another 7 days, so the analysis covers more +time as the holdback runs and you get a cumulative read rather than a fixed snapshot. + +Jetstream computes only the **Overall** period for a flagged holdback. There are no +weekly or 28-day breakdowns, because a holdback is long-running by nature. + +## When the first results appear + +The first rerun fires once the experiment is **28 days old** — 21 days of observation +plus the 7-day minimum enrollment — and then on every 7th day after the start date. + +Two details are worth knowing: + +- Eligible days are days where the age of the experiment in days is an exact multiple of + 7. This is counted from the start date, not from the previous run, and there is no + catch-up. If an eligible day is missed, the next opportunity is a week later. +- The job runs daily at **03:00 UTC**. The flag needs to be set before that time on an + eligible day for that day's run to pick it up. + +## When the flag does nothing + +The flag only has an effect while a holdback is **still enrolling**. The rolling +enrollment window is the whole mechanism, so it is skipped for an experiment that has +either: + +- an end date, or +- an enrollment end date. + +In that case the experiment is marked as a holdback but nothing else changes, and there +is no warning. If you have set the flag and no results are appearing, check those two +dates first. + +## Do not configure the dates by hand + +Once the flag is on, Experimenter owns the enrollment and end dates. Anything in the +experiment's Jetstream config that sets them will override what Experimenter sends and +pin the analysis back to a fixed window, which quietly defeats the flag. + +That means, in the experiment's `.toml` in +[metric-hub](https://github.com/mozilla/metric-hub/tree/main/jetstream): + +- do not set `enrollment_period` or `end_date` +- avoid a custom `enrollment_query` with a hard-coded date range, which caps enrollment in + SQL and has the same effect + +See [Jetstream configuration](/data-analysis/jetstream/configuration) for the config +format itself. + +## Migrating an existing holdback + +Holdbacks that were set up before this feature existed are being migrated in stages, and +the owner is contacted before their experiment is changed. If you own one, you will hear +from the Experimentation team first. After the migration, leave the Jetstream config +alone — the flag takes over from there. diff --git a/sidebars.js b/sidebars.js index 98224dacf..f979deb97 100644 --- a/sidebars.js +++ b/sidebars.js @@ -48,6 +48,11 @@ module.exports = { type: "doc", label: "Firefox Labs", id: "workflow/firefox-labs" + }, + { + type: "doc", + label: "Holdbacks", + id: "workflow/holdbacks" } ] }, From cefd62484aadb62dcf5a9ed50372949f0346e971 Mon Sep 17 00:00:00 2001 From: Yashika Khurana Date: Fri, 25 Sep 2026 14:16:13 -0700 Subject: [PATCH 2/3] Clarify enrollment_query guidance and add client-impact note A custom enrollment query does not interfere with the flag, per review on metric-hub#1685; only enrollment_period and end_date override what Experimenter sends. Softened that bullet from a prohibition to a note about the cohort staying fixed if the query hard-codes a date range. Adds a section stating the flag is analysis-side only, which is the question owners have asked most, and a note that seeing effects change over time still needs a manual analysis. --- docs/workflow/holdbacks.md | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/docs/workflow/holdbacks.md b/docs/workflow/holdbacks.md index 6f5369418..658f31104 100644 --- a/docs/workflow/holdbacks.md +++ b/docs/workflow/holdbacks.md @@ -38,7 +38,16 @@ Each weekly rerun pushes the window out by another 7 days, so the analysis cover time as the holdback runs and you get a cumulative read rather than a fixed snapshot. Jetstream computes only the **Overall** period for a flagged holdback. There are no -weekly or 28-day breakdowns, because a holdback is long-running by nature. +weekly or 28-day breakdowns, because a holdback is long-running by nature. If you want +to see how an effect changes over time, for example a novelty effect that tapers off, +that still needs a manual analysis. + +## It does not affect enrolled clients + +Setting the flag is an analysis-side change only. The recipe that clients fetch is built +separately and contains none of this — no flag, no synthesized dates — so targeting, +bucketing and enrollment are untouched, including for clients already in the holdback. +Nothing about the experiment itself changes; only how its results are computed. ## When the first results appear @@ -76,8 +85,12 @@ That means, in the experiment's `.toml` in [metric-hub](https://github.com/mozilla/metric-hub/tree/main/jetstream): - do not set `enrollment_period` or `end_date` -- avoid a custom `enrollment_query` with a hard-coded date range, which caps enrollment in - SQL and has the same effect + +The rest of the config is fine to keep. A custom `enrollment_query`, for example, +changes which rows the analysis reads, which is a normal thing to want and does not +interfere with the flag. Do note that if such a query has a hard-coded date range, the +set of enrolled clients stays fixed at that range: the window will keep moving forward +but it will not pick up anyone who enrolled later. See [Jetstream configuration](/data-analysis/jetstream/configuration) for the config format itself. From 78609051819003769e4dd999e3dbfeb43f0eab92 Mon Sep 17 00:00:00 2001 From: Yashika Khurana Date: Fri, 25 Sep 2026 14:44:34 -0700 Subject: [PATCH 3/3] Reword informal phrasing in the holdbacks page --- docs/workflow/holdbacks.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/workflow/holdbacks.md b/docs/workflow/holdbacks.md index 658f31104..e270ff6d9 100644 --- a/docs/workflow/holdbacks.md +++ b/docs/workflow/holdbacks.md @@ -12,9 +12,9 @@ A holdback keeps a slice of the population on the old experience after a feature so you can keep measuring the feature's impact over a long period. Unlike a normal experiment, a holdback enrolls continuously and does not have a planned end date. -That continuous enrollment is what makes holdbacks awkward to analyze. Jetstream expects +That continuous enrollment is what makes holdbacks difficult to analyze. Jetstream expects an enrollment period followed by an observation period, and a holdback never closes -enrollment, so there is no window to analyze. Historically the way round this was to +enrollment, so there is no window to analyze. Historically the workaround was to hand-write `enrollment_period` and `end_date` into the experiment's Jetstream config and update them by hand whenever you wanted fresher numbers. @@ -79,7 +79,8 @@ dates first. Once the flag is on, Experimenter owns the enrollment and end dates. Anything in the experiment's Jetstream config that sets them will override what Experimenter sends and -pin the analysis back to a fixed window, which quietly defeats the flag. +pin the analysis back to a fixed window, which disables the flag's behavior +without any warning. That means, in the experiment's `.toml` in [metric-hub](https://github.com/mozilla/metric-hub/tree/main/jetstream):