docs: document the is_holdback flag for long-term holdbacks

docs/data-analysis/jetstream/configuration.md

@@ -115,9 +115,9 @@ enrollment_query = """
 """
 
 # You can override either or both of the start_date and end_date.
-# In conjunction with a custom enrollment query, this can be useful for holdbacks,
-# since you don't really care about the period of time before your client upgrades
-# to the new version.
+# Do not set these on a holdback that has the is_holdback flag set: Experimenter
+# supplies the enrolment and end dates for those, and a value here overrides it.
+# See /workflow/holdbacks.
 start_date = "2020-01-01"
 end_date = "2020-12-31"
 

docs/workflow/holdbacks.md

@@ -0,0 +1,104 @@
+---
+id: holdbacks
+title: Holdbacks
+slug: /workflow/holdbacks
+---
+
+How long-term holdbacks are analyzed automatically using the `is_holdback` flag.
+
+## What a holdback is
+
+A holdback keeps a slice of the population on the old experience after a feature ships,
+so you can keep measuring the feature's impact over a long period. Unlike a normal
+experiment, a holdback enrolls continuously and does not have a planned end date.
+
+That continuous enrollment is what makes holdbacks difficult to analyze. Jetstream expects
+an enrollment period followed by an observation period, and a holdback never closes
+enrollment, so there is no window to analyze. Historically the workaround was to
+hand-write `enrollment_period` and `end_date` into the experiment's Jetstream config and
+update them by hand whenever you wanted fresher numbers.
+
+## The `is_holdback` flag
+
+Setting the **is_holdback** flag on an experiment replaces that manual configuration.
+Experimenter then synthesizes an analysis window for Jetstream and moves it forward once
+a week, so results accumulate on their own.
+
+The window works like this:
+
+- The **observation period** is the last 21 days of the window.
+- Everything between the experiment's start date and the beginning of that observation
+  period counts as **enrollment**. There must be at least 7 days of it.
+- The window ends on the date of the most recent weekly rerun, not on today's date. The
+  two are the same on the day a rerun fires and differ on other days. Anchoring on the
+  rerun is deliberate: anchoring on today would shift the window by a day every time the
+  API is read, instead of stepping cleanly once a week.
+
+Each weekly rerun pushes the window out by another 7 days, so the analysis covers more
+time as the holdback runs and you get a cumulative read rather than a fixed snapshot.
+
+Jetstream computes only the **Overall** period for a flagged holdback. There are no
+weekly or 28-day breakdowns, because a holdback is long-running by nature. If you want
+to see how an effect changes over time, for example a novelty effect that tapers off,
+that still needs a manual analysis.
+
+## It does not affect enrolled clients
+
+Setting the flag is an analysis-side change only. The recipe that clients fetch is built
+separately and contains none of this — no flag, no synthesized dates — so targeting,
+bucketing and enrollment are untouched, including for clients already in the holdback.
+Nothing about the experiment itself changes; only how its results are computed.
+
+## When the first results appear
+
+The first rerun fires once the experiment is **28 days old** — 21 days of observation
+plus the 7-day minimum enrollment — and then on every 7th day after the start date.
+
+Two details are worth knowing:
+
+- Eligible days are days where the age of the experiment in days is an exact multiple of
+  7. This is counted from the start date, not from the previous run, and there is no
+  catch-up. If an eligible day is missed, the next opportunity is a week later.
+- The job runs daily at **03:00 UTC**. The flag needs to be set before that time on an
+  eligible day for that day's run to pick it up.
+
+## When the flag does nothing
+
+The flag only has an effect while a holdback is **still enrolling**. The rolling
+enrollment window is the whole mechanism, so it is skipped for an experiment that has
+either:
+
+- an end date, or
+- an enrollment end date.
+
+In that case the experiment is marked as a holdback but nothing else changes, and there
+is no warning. If you have set the flag and no results are appearing, check those two
+dates first.
+
+## Do not configure the dates by hand
+
+Once the flag is on, Experimenter owns the enrollment and end dates. Anything in the
+experiment's Jetstream config that sets them will override what Experimenter sends and
+pin the analysis back to a fixed window, which disables the flag's behavior
+without any warning.
+
+That means, in the experiment's `.toml` in
+[metric-hub](https://github.com/mozilla/metric-hub/tree/main/jetstream):
+
+- do not set `enrollment_period` or `end_date`
+
+The rest of the config is fine to keep. A custom `enrollment_query`, for example,
+changes which rows the analysis reads, which is a normal thing to want and does not
+interfere with the flag. Do note that if such a query has a hard-coded date range, the
+set of enrolled clients stays fixed at that range: the window will keep moving forward
+but it will not pick up anyone who enrolled later.
+
+See [Jetstream configuration](/data-analysis/jetstream/configuration) for the config
+format itself.
+
+## Migrating an existing holdback
+
+Holdbacks that were set up before this feature existed are being migrated in stages, and
+the owner is contacted before their experiment is changed. If you own one, you will hear
+from the Experimentation team first. After the migration, leave the Jetstream config
+alone — the flag takes over from there.

sidebars.js

@@ -48,6 +48,11 @@ module.exports = {
               type: "doc",
               label: "Firefox Labs",
               id: "workflow/firefox-labs"
+            },
+            {
+              type: "doc",
+              label: "Holdbacks",
+              id: "workflow/holdbacks"
             }
           ]
         },