From 2580d02cc54113ff40036a77493b66a46cb2d688 Mon Sep 17 00:00:00 2001 From: Jared Lockhart <119884+jaredlockhart@users.noreply.github.com> Date: Thu, 30 Jul 2026 11:59:29 -0400 Subject: [PATCH] docs: add a Requesting QA page and link to it Because * The process for filing a QA request to get an experiment, rollout, or feature tested was documented only in an external Google Doc. * Four places in the docs linked at that doc or the raw Jira create URL without explaining the process. This commit * Adds a self-contained Requesting QA page covering issue types, prerequisites, field guidance, timing, and FAQ. * Adds it to the Testing & QA sidebar category. * Converts the four scattered links to point at the new page. --- docs/getting-started/for-experiment-owners.md | 2 +- .../technical-reference/feature-definition.md | 2 +- docs/workflow/experiments.md | 2 +- docs/workflow/qa-requests.md | 122 ++++++++++++++++++ docs/workflow/risk-mitigation.mdx | 2 +- sidebars.js | 1 + 6 files changed, 127 insertions(+), 4 deletions(-) create mode 100644 docs/workflow/qa-requests.md diff --git a/docs/getting-started/for-experiment-owners.md b/docs/getting-started/for-experiment-owners.md index aeef4294b..bbad179b7 100644 --- a/docs/getting-started/for-experiment-owners.md +++ b/docs/getting-started/for-experiment-owners.md @@ -17,7 +17,7 @@ Responsibilities include: - Align on the design of the experiment by attending a [data science office hours](https://www.google.com/url?q=https://docs.google.com/document/d/1dH-aG8IsYtq6881_Q_cyEtmxli0bK7nuVcUD-5D7q-s/edit%23&sa=D&source=calendar&ust=1646684514330583&usg=AOvVaw0O5Sbz3fHVx2rFDcl3DpNg) on [Wednesday and Thurday](https://mozilla-hub.atlassian.net/wiki/spaces/DATA/pages/6849684/Office+Hours). The goal is whenever possible to answer all questions possible within office hours. - If follow up work is needed from data science, then we need to file a [Data Org Jira ticket](https://mozilla-hub.atlassian.net/jira/software/c/projects/DO/boards/269). - Implementing the experiment in [Nimbus console AKA Experimenter](https://experimenter.services.mozilla.com/) -- Creating the [QA "Nimbus/Remote delivery" Jira ticket](https://mozilla-hub.atlassian.net/secure/CreateIssueDetails!init.jspa?pid=10212&issuetype=11290) (linked in experiment brief) +- Creating the QA "Nimbus/Remote delivery" Jira ticket (linked in experiment brief). See [Requesting QA](/workflow/qa-requests). - Launching the experiment, ending enrollment, ending experiment (the [experimentation workflow](/workflow/overview)) - Working with the experiment team (in slack at #ask-experimenter) to identify a set of reviewers trained and authorized to review experiments in your feature area. The experiment team will train them on how to review. diff --git a/docs/technical-reference/feature-definition.md b/docs/technical-reference/feature-definition.md index 086bde3e7..d05235c2a 100644 --- a/docs/technical-reference/feature-definition.md +++ b/docs/technical-reference/feature-definition.md @@ -28,7 +28,7 @@ Choose the guide for your platform: After landing a new feature, it is recommended to go through QA before running experiments or rollouts. This provides an extra layer of stability and can surface limitations early. -- See [this document](https://docs.google.com/document/d/1oz1YyaaBI-oHUDsktWA-dLtX7WzhYqs7C121yOPKo2w/edit) for steps on how to file a QA request. Use the `Feature-Configuration` label in Jira. ([Example](https://mozilla-hub.atlassian.net/browse/QA-1785)) +- See [Requesting QA](/workflow/qa-requests#field-guidance-functional-feature) for steps on how to file a QA request. Use the `Feature-Configuration` label in Jira. ([Example](https://mozilla-hub.atlassian.net/browse/QA-1785)) - If you have documentation about the feature's configuration, link it to the QA ticket — this helps with test plan and test case creation. Common questions QA will ask: diff --git a/docs/workflow/experiments.md b/docs/workflow/experiments.md index a2e9dd143..d7ab8fa09 100644 --- a/docs/workflow/experiments.md +++ b/docs/workflow/experiments.md @@ -118,7 +118,7 @@ Once your design is validated: 1. **Enable your feature for experimentation** using the [Nimbus feature API](/technical-reference/feature-definition). If your feature supports it, add an [exposure event](/advanced/feature-variables#recording-exposure-events) so analysis can distinguish users who actually encountered the change from those who were enrolled but never saw it. 2. **Write your experiment** in [Experimenter](https://experimenter.services.mozilla.com/nimbus/) and link to your experiment brief. -3. **QA your experiment** by putting it into [Preview](/workflow/testing) and either self-testing or filing a [QA Jira ticket](https://mozilla-hub.atlassian.net/secure/CreateIssueDetails!init.jspa?pid=10212&issuetype=11290). +3. **QA your experiment** by putting it into [Preview](/workflow/testing) and either self-testing or [filing a QA Jira ticket](/workflow/qa-requests). 4. **Review risks** — experiments make remote changes to the experience of live users, often millions of people. Experimenter will ask you to assess risk in several categories: brand risk, messaging risk, partner-related risk, revenue impact, and AI/ML risk. Review and mitigate [risks](/workflow/risk-mitigation) before you launch. ## Launch, monitor, learn diff --git a/docs/workflow/qa-requests.md b/docs/workflow/qa-requests.md new file mode 100644 index 000000000..24bf755d8 --- /dev/null +++ b/docs/workflow/qa-requests.md @@ -0,0 +1,122 @@ +--- +id: qa-requests +title: Requesting QA +slug: /workflow/qa-requests +--- + +How to file a QA request so QA can test your experiment, rollout, or feature before launch. + +The canonical process lives in a Google Doc: [How to file a QA request](https://docs.google.com/document/d/1oz1YyaaBI-oHUDsktWA-dLtX7WzhYqs7C121yOPKo2w/edit). This page reproduces it so you do not have to leave the docs site. + +## What a QA request is + +QA is a soft sign-off before launch. Feature testing alone is not enough for experiments or rollouts. A QA passthrough verifies that the feature turns on and off through Nimbus, that it works across the desired platforms and important locales, that the targeting expression enrolls only the intended users, and that the telemetry the analysis depends on is gathered while enrolled. + +If your work requires QA, you file a Jira ticket in the Quality Assurance (QA) project to start QA involvement. For the rules on when you may skip QA for an experiment or rollout, see [QA Sign-off](/workflow/risk-mitigation#qa-sign-off). + +## Issue types + +The QA project (pid 10212) has three issue types. Pick the one that matches your work. + +| Issue type | When to use | Create form | +| --- | --- | --- | +| Nimbus/Remote delivery (`11290`) | Experiments, rollouts, and messaging delivered through Nimbus or Remote Settings, including pre-flips through the Nimbus Secure Experiments collection. This is the type experiment owners file. | [Create](https://mozilla-hub.atlassian.net/secure/CreateIssueDetails!init.jspa?pid=10212&issuetype=11290) | +| Functional (`11461`) | Feature-configuration testing before you experiment, and general feature work. | [Create](https://mozilla-hub.atlassian.net/secure/CreateIssueDetails!init.jspa?pid=10212&issuetype=11461) | +| Embedded QA (`10064`) | Projects that will be in development for at least three release cycles. Identical fields to Functional. | [Create](https://mozilla-hub.atlassian.net/secure/CreateIssueDetails!init.jspa?pid=10212&issuetype=10064) | + +## Before you file a Nimbus/Remote delivery request + +- A Data Science (DS) Jira ticket if your experiment or rollout needs analysis. The DS team assesses feasibility on their [board](https://mozilla-hub.atlassian.net/jira/software/c/projects/DS/boards/258). This ticket is not required if no data analysis is needed. +- An Experimenter recipe (Nimbus page). The delivery team uses it to confirm the experiment or rollout can run on the current infrastructure, and QA uses it to test the recipe. See the [experiment workflow overview](/workflow/overview) to create one. + +## How to file + +1. Go to [jira.mozilla.com](https://mozilla-hub.atlassian.net). +2. Click **Create** at the top of the page. +3. In the first dropdown, select the **Quality Assurance (QA)** project. +4. In the second dropdown, select the issue type from the three above. + +Each field has a role in QA work. Fields are marked by importance: + +- **Mandatory**: required to start QA. If these are missing, QA will follow up with questions and will not assign an owner until the details are provided. +- **Recommended**: not blocking, but they speed up the process. +- **Optional**: self-explanatory. + +After you fill in the fields for your issue type, click **Create**. An email is sent to the QA team. + +## Field guidance: Nimbus/Remote delivery + +Use this form for experiments and rollouts. Depending on complexity, QA needs one to three days for testing (documenting, questions, test case preparation, execution, and sign-off). + +| Field | Importance | Guidance | +| --- | --- | --- | +| Summary | Mandatory | Short description of the experiment or rollout. Example: "QA for holdback experiment for new MR1 onboarding on about:welcome". | +| Reporter | Mandatory | Usually set to your account. Change it if you are filing on someone else's behalf. | +| Priority | Mandatory | How urgent and important the work is. | +| Link to Nimbus recipe / Experiment Brief / Design document | Recommended | Add any or all of: the Nimbus recipe (Experimenter page), the Experiment Brief, and the Design Document. Providing these at filing time helps ensure a QA contact is assigned. | +| Assignee | Optional | Leave as is unless a specific QA has worked on your prior experiments. | +| Due Date | Optional | The date by which you want testing done, or the tentative launch date. | +| Attachment | Optional | Use if the experiment is in a custom build or add-on. If the build lives elsewhere, note the location in the description instead. | +| Channels | Optional | The Firefox channel and version your experiment targets. If unset, QA sets these from the linked Nimbus recipe. | +| Bugs to be filed at | Optional | Where QA should log bugs. If unset, QA finds an appropriate Bugzilla component or uses Shield: Shield Studies. | +| Contacts: Feature owner(s) + engineer(s) | Optional | People who can answer questions after the ticket is assigned. | +| Description | Optional | Additional details not already in the Nimbus/Experimenter ticket. | +| Delivery mechanism | Optional | Dropdown for how the experiment or rollout is delivered, usually Nimbus. | +| Collaborators, Labels | Optional | QA fills these in (QA adds a QA:Experiment or QA:Rollout label after assignment). | + +## Field guidance: Functional (feature) + +Use this form for feature-configuration testing before you experiment. Add the `Feature-Configuration` label. See [QA-1785](https://mozilla-hub.atlassian.net/browse/QA-1785) for an example ticket. If you have documentation about the feature's configuration, link it to the ticket so QA can build the test plan and cases. + +Key Mandatory fields: + +| Field | Guidance | +| --- | --- | +| Feature name | The name of your feature. Example: "Normandy: Don't unenroll users when prefs change". | +| Summary | Short description of the request. | +| Target release | The Firefox version you plan to release in. | +| Priority | How urgent and important the work is. | +| Product | The platform your feature is on. | +| Shipping Method | Normal train release, System Add-on (Balrog or other delivery), or Other (web page or anything else). | +| Relevant Links | Links to tracking bugs or anything that helps QA. | +| Link to Technical Documentation | Documentation, designs, UX flows, or feature technical documentation. | +| Feature Owners | Who QA should contact with issues or requests. | +| Reporter | Usually your account. Change it if filing for someone else. | +| Engineering team | Select your team from the dropdown so QA can route the ticket. | + +Recommended fields include Description (work in scope, work not in scope, other details), Bugwork (when the work is only verifying bug fixes to a released feature), Engineers, EPM, and Product Manager. + +## Timing + +- Nimbus/Remote delivery: file at least two days before launch for a low-complexity request, about one week before for a medium or high-complexity request. If QA lacks sufficient detail, the deadline may slip. +- Functional: file by the Friday before the target Nightly cycle begins. + +QA cannot start without a due date, or at least a Firefox version number, because the work cannot be prioritized. You may still file early and add the date later. + +## FAQ + +**This experiment or feature already had a round of QA. Do I need a new one?** +Yes if the targeting changed or a new feature configuration was added, or if the previous QA was a different issue type (for example, the first was Functional and the new one is a rollout). Not necessarily for a code change on the feature (self-testing can cover it) or a change to the targeted version or channel (a risk the team can accept). + +**Can I reuse an old QA ticket?** +Only if it has not been marked Resolved. If it is still open and changes occur, update the ticket and notify the assigned QA. + +**Can I file one ticket for both a feature and an experiment?** +No. They are treated differently: separate QA documentation, tests, and sign-offs. File one ticket for each. + +**Can I file a QA request before I know I need it?** +Yes. If you are planning an experiment or rollout but the timeline or some details are not final, file the request and update the ticket once the timeline and details are known. + +**Can I reserve QA time by filing in advance?** +No. QA prioritizes tickets by how actionable they are and by bandwidth. + +## Contacts + +- Nimbus/Remote delivery: Slack [#ask-experimenter](https://mozilla.slack.com/archives/CF94YGE03) or @sescalante. +- Functional: dte-leads on Slack. + +## Examples + +- [QA-5522](https://mozilla-hub.atlassian.net/browse/QA-5522) (Nimbus/Remote delivery) +- [QA-5474](https://mozilla-hub.atlassian.net/browse/QA-5474) (Nimbus/Remote delivery) +- [QA-1785](https://mozilla-hub.atlassian.net/browse/QA-1785) (Functional feature-configuration) diff --git a/docs/workflow/risk-mitigation.mdx b/docs/workflow/risk-mitigation.mdx index ff63f3ed5..fda6d5836 100644 --- a/docs/workflow/risk-mitigation.mdx +++ b/docs/workflow/risk-mitigation.mdx @@ -62,7 +62,7 @@ If your product involves encryption of any type - like a VPN - it is important t Feature testing alone is not enough for experiments or rollouts. A QA passthrough will also ensure that the feature is successfully turned on/off through Nimbus, that it works on all desired platforms and most important locales, that the targeting expression works as expected (only users that meet the criteria will be enrolled in the experiment/rollout), and that the telemetry that we're basing the analysis on is successfully gathered while enrolled. -Unless you are 100% certain you don't need QA - [file a QA "Nimbus/Remote delivery" ticket here](https://mozilla-hub.atlassian.net/secure/CreateIssueDetails!init.jspa?pid=10212&issuetype=11290). You can use [this document](https://docs.google.com/document/d/1oz1YyaaBI-oHUDsktWA-dLtX7WzhYqs7C121yOPKo2w/edit#heading=h.s50iqavbblol) as a guide when filing the ticket to ensure that you pass all the relevant information that you can to QA. Going through QA is recommended, as they often find critical edge cases that were missed. +Unless you are 100% certain you don't need QA, follow [Requesting QA](/workflow/qa-requests) to file a Nimbus/Remote delivery ticket. That page walks through the fields so you pass all the relevant information to QA. Going through QA is recommended, as they often find critical edge cases that were missed. #### When You May Skip QA diff --git a/sidebars.js b/sidebars.js index 329a4649d..ec4afd7ad 100644 --- a/sidebars.js +++ b/sidebars.js @@ -57,6 +57,7 @@ module.exports = { type: "category", label: "Testing & QA", items: [ + "workflow/qa-requests", "workflow/testing", "platform-guides/desktop/desktop-feature-api-testing", "platform-guides/android/android-preview-testing",