Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/getting-started/for-experiment-owners.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Responsibilities include:
- Align on the design of the experiment by attending a [data science office hours](https://www.google.com/url?q=https://docs.google.com/document/d/1dH-aG8IsYtq6881_Q_cyEtmxli0bK7nuVcUD-5D7q-s/edit%23&sa=D&source=calendar&ust=1646684514330583&usg=AOvVaw0O5Sbz3fHVx2rFDcl3DpNg) on [Wednesday and Thurday](https://mozilla-hub.atlassian.net/wiki/spaces/DATA/pages/6849684/Office+Hours). The goal is whenever possible to answer all questions possible within office hours.
- If follow up work is needed from data science, then we need to file a [Data Org Jira ticket](https://mozilla-hub.atlassian.net/jira/software/c/projects/DO/boards/269).
- Implementing the experiment in [Nimbus console AKA Experimenter](https://experimenter.services.mozilla.com/)
- Creating the [QA "Nimbus/Remote delivery" Jira ticket](https://mozilla-hub.atlassian.net/secure/CreateIssueDetails!init.jspa?pid=10212&issuetype=11290) (linked in experiment brief)
- Creating the QA "Nimbus/Remote delivery" Jira ticket (linked in experiment brief). See [Requesting QA](/workflow/qa-requests).
- Launching the experiment, ending enrollment, ending experiment (the [experimentation workflow](/workflow/overview))
- Working with the experiment team (in slack at #ask-experimenter) to identify a set of reviewers trained and authorized to review experiments in your feature area. The experiment team will train them on how to review.

Expand Down
2 changes: 1 addition & 1 deletion docs/technical-reference/feature-definition.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Choose the guide for your platform:

After landing a new feature, it is recommended to go through QA before running experiments or rollouts. This provides an extra layer of stability and can surface limitations early.

- See [this document](https://docs.google.com/document/d/1oz1YyaaBI-oHUDsktWA-dLtX7WzhYqs7C121yOPKo2w/edit) for steps on how to file a QA request. Use the `Feature-Configuration` label in Jira. ([Example](https://mozilla-hub.atlassian.net/browse/QA-1785))
- See [Requesting QA](/workflow/qa-requests#field-guidance-functional-feature) for steps on how to file a QA request. Use the `Feature-Configuration` label in Jira. ([Example](https://mozilla-hub.atlassian.net/browse/QA-1785))
- If you have documentation about the feature's configuration, link it to the QA ticket — this helps with test plan and test case creation.

Common questions QA will ask:
Expand Down
2 changes: 1 addition & 1 deletion docs/workflow/experiments.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ Once your design is validated:

1. **Enable your feature for experimentation** using the [Nimbus feature API](/technical-reference/feature-definition). If your feature supports it, add an [exposure event](/advanced/feature-variables#recording-exposure-events) so analysis can distinguish users who actually encountered the change from those who were enrolled but never saw it.
2. **Write your experiment** in [Experimenter](https://experimenter.services.mozilla.com/nimbus/) and link to your experiment brief.
3. **QA your experiment** by putting it into [Preview](/workflow/testing) and either self-testing or filing a [QA Jira ticket](https://mozilla-hub.atlassian.net/secure/CreateIssueDetails!init.jspa?pid=10212&issuetype=11290).
3. **QA your experiment** by putting it into [Preview](/workflow/testing) and either self-testing or [filing a QA Jira ticket](/workflow/qa-requests).
4. **Review risks** — experiments make remote changes to the experience of live users, often millions of people. Experimenter will ask you to assess risk in several categories: brand risk, messaging risk, partner-related risk, revenue impact, and AI/ML risk. Review and mitigate [risks](/workflow/risk-mitigation) before you launch.

## Launch, monitor, learn
Expand Down
122 changes: 122 additions & 0 deletions docs/workflow/qa-requests.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
---
id: qa-requests
title: Requesting QA
slug: /workflow/qa-requests
---

How to file a QA request so QA can test your experiment, rollout, or feature before launch.

The canonical process lives in a Google Doc: [How to file a QA request](https://docs.google.com/document/d/1oz1YyaaBI-oHUDsktWA-dLtX7WzhYqs7C121yOPKo2w/edit). This page reproduces it so you do not have to leave the docs site.

## What a QA request is

QA is a soft sign-off before launch. Feature testing alone is not enough for experiments or rollouts. A QA passthrough verifies that the feature turns on and off through Nimbus, that it works across the desired platforms and important locales, that the targeting expression enrolls only the intended users, and that the telemetry the analysis depends on is gathered while enrolled.

If your work requires QA, you file a Jira ticket in the Quality Assurance (QA) project to start QA involvement. For the rules on when you may skip QA for an experiment or rollout, see [QA Sign-off](/workflow/risk-mitigation#qa-sign-off).

## Issue types

The QA project (pid 10212) has three issue types. Pick the one that matches your work.

| Issue type | When to use | Create form |
| --- | --- | --- |
| Nimbus/Remote delivery (`11290`) | Experiments, rollouts, and messaging delivered through Nimbus or Remote Settings, including pre-flips through the Nimbus Secure Experiments collection. This is the type experiment owners file. | [Create](https://mozilla-hub.atlassian.net/secure/CreateIssueDetails!init.jspa?pid=10212&issuetype=11290) |
| Functional (`11461`) | Feature-configuration testing before you experiment, and general feature work. | [Create](https://mozilla-hub.atlassian.net/secure/CreateIssueDetails!init.jspa?pid=10212&issuetype=11461) |
| Embedded QA (`10064`) | Projects that will be in development for at least three release cycles. Identical fields to Functional. | [Create](https://mozilla-hub.atlassian.net/secure/CreateIssueDetails!init.jspa?pid=10212&issuetype=10064) |

## Before you file a Nimbus/Remote delivery request

- A Data Science (DS) Jira ticket if your experiment or rollout needs analysis. The DS team assesses feasibility on their [board](https://mozilla-hub.atlassian.net/jira/software/c/projects/DS/boards/258). This ticket is not required if no data analysis is needed.
- An Experimenter recipe (Nimbus page). The delivery team uses it to confirm the experiment or rollout can run on the current infrastructure, and QA uses it to test the recipe. See the [experiment workflow overview](/workflow/overview) to create one.

## How to file

1. Go to [jira.mozilla.com](https://mozilla-hub.atlassian.net).
2. Click **Create** at the top of the page.
3. In the first dropdown, select the **Quality Assurance (QA)** project.
4. In the second dropdown, select the issue type from the three above.

Each field has a role in QA work. Fields are marked by importance:

- **Mandatory**: required to start QA. If these are missing, QA will follow up with questions and will not assign an owner until the details are provided.
- **Recommended**: not blocking, but they speed up the process.
- **Optional**: self-explanatory.

After you fill in the fields for your issue type, click **Create**. An email is sent to the QA team.

## Field guidance: Nimbus/Remote delivery

Use this form for experiments and rollouts. Depending on complexity, QA needs one to three days for testing (documenting, questions, test case preparation, execution, and sign-off).

| Field | Importance | Guidance |
| --- | --- | --- |
| Summary | Mandatory | Short description of the experiment or rollout. Example: "QA for holdback experiment for new MR1 onboarding on about:welcome". |
| Reporter | Mandatory | Usually set to your account. Change it if you are filing on someone else's behalf. |
| Priority | Mandatory | How urgent and important the work is. |
| Link to Nimbus recipe / Experiment Brief / Design document | Recommended | Add any or all of: the Nimbus recipe (Experimenter page), the Experiment Brief, and the Design Document. Providing these at filing time helps ensure a QA contact is assigned. |
| Assignee | Optional | Leave as is unless a specific QA has worked on your prior experiments. |
| Due Date | Optional | The date by which you want testing done, or the tentative launch date. |
| Attachment | Optional | Use if the experiment is in a custom build or add-on. If the build lives elsewhere, note the location in the description instead. |
| Channels | Optional | The Firefox channel and version your experiment targets. If unset, QA sets these from the linked Nimbus recipe. |
| Bugs to be filed at | Optional | Where QA should log bugs. If unset, QA finds an appropriate Bugzilla component or uses Shield: Shield Studies. |
| Contacts: Feature owner(s) + engineer(s) | Optional | People who can answer questions after the ticket is assigned. |
| Description | Optional | Additional details not already in the Nimbus/Experimenter ticket. |
| Delivery mechanism | Optional | Dropdown for how the experiment or rollout is delivered, usually Nimbus. |
| Collaborators, Labels | Optional | QA fills these in (QA adds a QA:Experiment or QA:Rollout label after assignment). |

## Field guidance: Functional (feature)

Use this form for feature-configuration testing before you experiment. Add the `Feature-Configuration` label. See [QA-1785](https://mozilla-hub.atlassian.net/browse/QA-1785) for an example ticket. If you have documentation about the feature's configuration, link it to the ticket so QA can build the test plan and cases.

Key Mandatory fields:

| Field | Guidance |
| --- | --- |
| Feature name | The name of your feature. Example: "Normandy: Don't unenroll users when prefs change". |
| Summary | Short description of the request. |
| Target release | The Firefox version you plan to release in. |
| Priority | How urgent and important the work is. |
| Product | The platform your feature is on. |
| Shipping Method | Normal train release, System Add-on (Balrog or other delivery), or Other (web page or anything else). |
| Relevant Links | Links to tracking bugs or anything that helps QA. |
| Link to Technical Documentation | Documentation, designs, UX flows, or feature technical documentation. |
| Feature Owners | Who QA should contact with issues or requests. |
| Reporter | Usually your account. Change it if filing for someone else. |
| Engineering team | Select your team from the dropdown so QA can route the ticket. |

Recommended fields include Description (work in scope, work not in scope, other details), Bugwork (when the work is only verifying bug fixes to a released feature), Engineers, EPM, and Product Manager.

## Timing

- Nimbus/Remote delivery: file at least two days before launch for a low-complexity request, about one week before for a medium or high-complexity request. If QA lacks sufficient detail, the deadline may slip.
- Functional: file by the Friday before the target Nightly cycle begins.

QA cannot start without a due date, or at least a Firefox version number, because the work cannot be prioritized. You may still file early and add the date later.

## FAQ

**This experiment or feature already had a round of QA. Do I need a new one?**
Yes if the targeting changed or a new feature configuration was added, or if the previous QA was a different issue type (for example, the first was Functional and the new one is a rollout). Not necessarily for a code change on the feature (self-testing can cover it) or a change to the targeted version or channel (a risk the team can accept).

**Can I reuse an old QA ticket?**
Only if it has not been marked Resolved. If it is still open and changes occur, update the ticket and notify the assigned QA.

**Can I file one ticket for both a feature and an experiment?**
No. They are treated differently: separate QA documentation, tests, and sign-offs. File one ticket for each.

**Can I file a QA request before I know I need it?**
Yes. If you are planning an experiment or rollout but the timeline or some details are not final, file the request and update the ticket once the timeline and details are known.

**Can I reserve QA time by filing in advance?**
No. QA prioritizes tickets by how actionable they are and by bandwidth.

## Contacts

- Nimbus/Remote delivery: Slack [#ask-experimenter](https://mozilla.slack.com/archives/CF94YGE03) or @sescalante.
- Functional: dte-leads on Slack.

## Examples

- [QA-5522](https://mozilla-hub.atlassian.net/browse/QA-5522) (Nimbus/Remote delivery)
- [QA-5474](https://mozilla-hub.atlassian.net/browse/QA-5474) (Nimbus/Remote delivery)
- [QA-1785](https://mozilla-hub.atlassian.net/browse/QA-1785) (Functional feature-configuration)
2 changes: 1 addition & 1 deletion docs/workflow/risk-mitigation.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ If your product involves encryption of any type - like a VPN - it is important t

Feature testing alone is not enough for experiments or rollouts. A QA passthrough will also ensure that the feature is successfully turned on/off through Nimbus, that it works on all desired platforms and most important locales, that the targeting expression works as expected (only users that meet the criteria will be enrolled in the experiment/rollout), and that the telemetry that we're basing the analysis on is successfully gathered while enrolled.

Unless you are 100% certain you don't need QA - [file a QA "Nimbus/Remote delivery" ticket here](https://mozilla-hub.atlassian.net/secure/CreateIssueDetails!init.jspa?pid=10212&issuetype=11290). You can use [this document](https://docs.google.com/document/d/1oz1YyaaBI-oHUDsktWA-dLtX7WzhYqs7C121yOPKo2w/edit#heading=h.s50iqavbblol) as a guide when filing the ticket to ensure that you pass all the relevant information that you can to QA. Going through QA is recommended, as they often find critical edge cases that were missed.
Unless you are 100% certain you don't need QA, follow [Requesting QA](/workflow/qa-requests) to file a Nimbus/Remote delivery ticket. That page walks through the fields so you pass all the relevant information to QA. Going through QA is recommended, as they often find critical edge cases that were missed.

#### When You May Skip QA

Expand Down
1 change: 1 addition & 0 deletions sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ module.exports = {
type: "category",
label: "Testing & QA",
items: [
"workflow/qa-requests",
"workflow/testing",
"platform-guides/desktop/desktop-feature-api-testing",
"platform-guides/android/android-preview-testing",
Expand Down
Loading