You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
CTRL sends email via plain SMTP with a username and password today. That path is pointed at Mailgun in UAT (from #715, after SendGrid's free trial expired), and Mailgun's free tier caps at around 100 emails a day, which isn't enough for real study volumes. We would rather use Garvan IT's Microsoft 365 tenant than move onto a paid third-party tier, so this issue is about doing that.
Following ITHELP-22014 and a follow-up chat, our on-prem SMTP relay only accepts connections from inside Garvan's internal network. CTRL runs on GCP so it's not reachable from there. Technically we could set up a site-to-site VPN or Google Cloud Interconnect to bridge GCP and the internal network, but IT would rather not add that infrastructure. Someone would have to maintain the tunnel long-term, any outage on it would take out CTRL email, and it broadens the exposure of an internal-only system to a cloud-hosted app.
The recommendation from IT is to use Garvan IT's Microsoft 365 tenant with OAuth2 for SMTP. Microsoft disabled password-based SMTP AUTH by default for M365 tenants back in 2022, so OAuth2 is the way to authenticate against M365 for sending mail. IT will set up the tenant side (app registration in Entra, mailbox scoping for ctrl-noreply@garvan.org.au, DNS if needed) once CTRL supports the OAuth flow.
Current flow: application/backend/src/utils/mailer.ts calls nodemailer.createTransport with host, port, username, password from config.smtp, and the four controllers that send email (AuthController, UsersController, ParticipantsController, MailerController) all use that transport. The call sites don't change. Everything behind them gets reworked.
Rework application/backend/src/utils/mailer.ts to sit behind a small provider abstraction. Two variants at launch: smtp-basic (what we have today, kept as a supported option so other orgs deploying CTRL can still use plain SMTP against their own relays, or SendGrid, Mailgun, Google App Password on smtp.gmail.com, or whatever else they run) and m365-oauth (new, Microsoft-specific OAuth2 SMTP using @azure/msal-node). Direction confirmed on 21 August after team discussion on slack.
Change the config schema in application/backend/src/config.ts from the fixed smtp: {host, port, username, password} block to a discriminated union on a provider field, so the shape can differ per variant.
Add @azure/msal-node as a backend dependency. MSAL handles token acquisition, in-memory cache, expiry-based re-acquisition, concurrent-request dedup, and Microsoft-specific behaviour (extended token lifetimes via ext_expires_in, correlation IDs, throttling headers) out of the box.
Update .helm/ctrl/values.yaml to match the new config shape so deployers pick their provider at deploy time.
Add a deployer docs page at docs/email.md covering both variants.
For the m365-oauth variant, the flow is: MSAL's ConfidentialClientApplication takes tenantId, clientId, and clientSecret from config, and MSAL's acquireTokenByClientCredential handles the OAuth POST + response parsing + in-memory cache + expiry refresh + concurrent-request dedup. Standard OAuth 2 client credentials flow under the covers, no user involved, no refresh token. The token is handed to nodemailer's auth: { type: 'OAuth2', user, accessToken } shape, which uses it in the SASL XOAUTH2 exchange during the SMTP AUTH step against smtp.office365.com:587. Everything else about the send (nodemailer, MIME payload, controllers, templates) stays the same as today.
For Garvan's M365 setup, the config values are Garvan's tenantId, the app registration's clientId and clientSecret, plus smtp.office365.com:587. Non-Microsoft deployers use smtp-basic (see multi-tenant section below). A generic OAuth variant becomes a follow-up when a real second-provider use case appears.
Token cache and refresh: MSAL maintains an in-memory cache per backend process. When the cached token nears expiry, MSAL re-acquires from the token endpoint. Concurrent sends share a single in-flight refresh handled by MSAL internally so we don't hit the token endpoint N times per burst. Nothing is written to disk or the database, so a process restart just fetches a fresh token. The client secret itself lives in the existing ctrl-config K8s Secret alongside the SMTP password we use today, so no new secret-storage path is needed. Rotating the client secret will require re-encrypting the values in the deploy-config repo, same pattern as any other SMTP password change.
Multi-tenant
Non-Garvan orgs deploying CTRL keep using the smtp-basic provider unchanged. That path covers self-hosted SMTP relays, SendGrid, Mailgun, Amazon SES via SMTP creds, and Google Workspace via App Password. The new m365-oauth variant is Microsoft-only. If a downstream deployer ever needs OAuth for their own provider (JWT-bearer for Google Workspace service accounts, refresh-token for delegated setups, or a custom client-credentials-capable OAuth server), that becomes a separate provider variant added later, sitting next to m365-oauth in the discriminated union on provider.
What IT needs to do on their side
Once the code lands and the tenant setup begins:
App registration in Entra for CTRL with the Mail.Send application permission.
Grant admin consent.
Generate a client secret and hand it over securely.
Authorize the app to send from ctrl-noreply@garvan.org.au specifically. Application permission Mail.Send on its own only allows sends from HVE-type accounts, so a regular mailbox has to be scoped in either through Application Access Policies or RBAC for Applications depending on what Garvan uses tenant-wide. This scoping is what unlocks the ability to send from that mailbox, not just what restricts it.
Provision ctrl-noreply@garvan.org.au itself as an alias (confirmed by @ignatiusm and @mhebrard).
Confirm Authenticated SMTP is enabled for the mailbox.
Follow-up questions IT asked, and our answers
From the ITHELP-22014 ticket:
Other Garvan teams doing this? Not a blocker for our implementation, out of scope for this issue.
Are the GCP egress IPs dedicated to CTRL or shared? Doesn't affect the M365 OAuth path since we won't be going through Garvan's internal network. Not blocking the implementation.
How many emails per week from ctrl-noreply? Below 10k a week based on a rough count of current sends (invites, password resets, OTPs, contact-us) and discussion with @mhebrard. Well below the volume that would need a dedicated HVE account. A regular mailbox with the Application Access Policy or RBAC scoping described in the "What IT needs to do" section is enough. Can be revisited later if volumes grow.
Does CTRL support OAuth today? Not yet. Only plain SMTP with a password. This issue is about adding the OAuth flow so we can use M365.
Replies needed on ctrl-noreply@garvan.org.au? Leaning toward an alias since the app only needs to send, not receive. See the team-decision section below.
Still open on IT's side, worth checking with them:
Application permissions or delegated. Application permissions seems like the natural fit for an unattended service, but IT's confirmation is what decides.
Application Access Policies or RBAC for Applications for the mailbox scoping. Whichever Garvan uses tenant-wide.
Whether SMTP AUTH is enabled tenant-wide, or needs enabling per-mailbox for ctrl-noreply. If it is disabled tenant-wide by policy, we would need to pivot to Microsoft Graph sendMail which is a bigger code change.
Submitted as new IT ticket ITHELP-27087 (references ITHELP-22014), with @ignatiusm and @mhebrard requested as watchers.
Folded into this work
Other issues that live in the same code region or are directly forced by the M365 path:
Closes Change the Sender email address #802 (mangled sender address). M365 will refuse to send from a hostname-shaped From, so the sender rework is forced by the M365 path.
Closes SMTP error crashes backend #719 (SMTP errors crash the backend). Same code region as the transport rework, and M365 adds new failure modes (bad token, expired token, XOAUTH2 rejected) that we have to handle anyway. Per @ignatiusm's review, SMTP error crashes backend #719 stays open because SMTP failure handling remains relevant for the smtp-basic path going forward. Addressing partial improvements here for the M365 path (bad token, expired token, XOAUTH2 rejected), but not closing SMTP error crashes backend #719 as part of this PR.
Kept out on purpose
Related but going into follow-up PRs after this one lands, so this PR stays reviewable:
Original two-option framing (superseded, see Update below)
Two ways this can go, either works.
Option A: use a personal Microsoft 365 Developer Program sandbox tenant for local dev, to test the OAuth path end-to-end as I build it. The sandbox only exists in my local config.json5 during dev and never touches deployed CTRL. When IT provisions the real Garvan tenant, the transition into UAT and prod is just a handful of config values (tenantId, clientId, clientSecret, sender address) swapping in. No code change on our side.
Option B: wait for IT to provision Garvan's tenant first, then write and test against it directly. Most of the code work (provider abstraction, config schema, helm changes, sender and failure-handling reworks) can still proceed in parallel while waiting. The difference is when the OAuth path gets verified against Microsoft, either continuously in a sandbox or all at once when Garvan's tenant is ready.
Worth flagging, as @mhebrard has raised before with other services, that we already have parts of CTRL that depend on personal accounts and moving those to prod has been painful. Option A is designed not to be, since the sandbox never touches deployed CTRL and only a handful of config values change at deploy time. If the team would rather skip personal accounts entirely, Option B works too.
Update. Option A is off the table. Microsoft tightened M365 Developer Program eligibility in early 2025 and signup with a personal MSA is now declined without a paid Visual Studio subscription, verified Microsoft Graph API history, or Microsoft Partner Network membership. See Q3 for details.
Going with Option B. Build the provider abstraction, config schema refactor, helm values changes, sender rework (#802), and the MSAL wiring now, against unit tests and a mocked MSAL client. End-to-end verification against real Microsoft servers happens once IT provisions the Garvan tenant. This still gets the bulk of the PR reviewable while IT's side is set up in parallel.
Nice-to-have if IT is willing: a scoped throwaway dev app registration inside Garvan's real M365 tenant with Mail.Send limited to one test mailbox via Application Access Policy. Would let me verify end-to-end during build against the exact target we deploy to. Not blocking, just faster feedback.
Questions for the team
Consolidating the calls I'd like @ignatiusm and @mhebrard to weigh in on before I start writing code, so nothing has to be scrolled back to:
Mailbox type for ctrl-noreply@garvan.org.au.Leaning toward an alias since the app only needs to send. Any concerns with alias over a shared mailbox? Confirmed as alias by both @ignatiusm and @mhebrard.
Development order.Option A (sandbox during dev) or Option B (wait for IT to provision Garvan's tenant first). Either works, defer to team preference.Confirmed Option A (sandbox during dev) per @ignatiusm's review. Microsoft 365 Developer Program is free (no cost), sandbox tenant renewable every 90 days.
Update: Microsoft tightened M365 Developer Program eligibility in early 2025. Signup with a personal MSA is now declined unless you have a paid Visual Studio subscription, verified Microsoft Graph API history, or Microsoft Partner Network membership. Option A is blocked for me personally. @mhebrard also flagged wanting clarification on the two-option framing, which is now moot given the pivot.
Falling back to Option B. All code work (provider abstraction, config schema, helm changes, sender rework, MSAL wiring) proceeds now against unit tests and a mocked MSAL client. End-to-end verification against real Microsoft servers happens when IT provisions the Garvan tenant.
Nice-to-have if IT is open to it: a scoped throwaway dev app registration inside Garvan's real M365 tenant, Mail.Send limited to one test mailbox via Application Access Policy. Would let me verify end-to-end during build against the exact target we deploy to. Not blocking, but faster feedback if available.
IT follow-ups.For the still-open items in the section above (application vs delegated, Access Policies vs RBAC, SMTP AUTH state), would you prefer @mhebrard chases them via the ITHELP-22014 ticket, or should I reach out to Tony directly? Resolved. Per @ignatiusm's suggestion and @mhebrard's confirmation, opened a new ticket ITHELP-27087 referencing ITHELP-22014, with @ignatiusm and @mhebrard requested as watchers. Ticket covers all three still-open items plus the nice-to-have dev app registration ask.
CTRL sends email via plain SMTP with a username and password today. That path is pointed at Mailgun in UAT (from #715, after SendGrid's free trial expired), and Mailgun's free tier caps at around 100 emails a day, which isn't enough for real study volumes. We would rather use Garvan IT's Microsoft 365 tenant than move onto a paid third-party tier, so this issue is about doing that.
Following ITHELP-22014 and a follow-up chat, our on-prem SMTP relay only accepts connections from inside Garvan's internal network. CTRL runs on GCP so it's not reachable from there. Technically we could set up a site-to-site VPN or Google Cloud Interconnect to bridge GCP and the internal network, but IT would rather not add that infrastructure. Someone would have to maintain the tunnel long-term, any outage on it would take out CTRL email, and it broadens the exposure of an internal-only system to a cloud-hosted app.
The recommendation from IT is to use Garvan IT's Microsoft 365 tenant with OAuth2 for SMTP. Microsoft disabled password-based SMTP AUTH by default for M365 tenants back in 2022, so OAuth2 is the way to authenticate against M365 for sending mail. IT will set up the tenant side (app registration in Entra, mailbox scoping for
ctrl-noreply@garvan.org.au, DNS if needed) once CTRL supports the OAuth flow.Reference IT sent over: https://learn.microsoft.com/en-us/exchange/mail-flow-best-practices/oauth-high-volume-mails-m365
What we're changing
Current flow:
application/backend/src/utils/mailer.tscallsnodemailer.createTransportwithhost,port,username,passwordfromconfig.smtp, and the four controllers that send email (AuthController,UsersController,ParticipantsController,MailerController) all use that transport. The call sites don't change. Everything behind them gets reworked.Rework
application/backend/src/utils/mailer.tsto sit behind a small provider abstraction. Two variants at launch:smtp-basic(what we have today, kept as a supported option so other orgs deploying CTRL can still use plain SMTP against their own relays, or SendGrid, Mailgun, Google App Password onsmtp.gmail.com, or whatever else they run) andm365-oauth(new, Microsoft-specific OAuth2 SMTP using@azure/msal-node). Direction confirmed on 21 August after team discussion on slack.Change the config schema in
application/backend/src/config.tsfrom the fixedsmtp: {host, port, username, password}block to a discriminated union on aproviderfield, so the shape can differ per variant.Add
@azure/msal-nodeas a backend dependency. MSAL handles token acquisition, in-memory cache, expiry-based re-acquisition, concurrent-request dedup, and Microsoft-specific behaviour (extended token lifetimes viaext_expires_in, correlation IDs, throttling headers) out of the box.Update
.helm/ctrl/values.yamlto match the new config shape so deployers pick their provider at deploy time.Add a deployer docs page at
docs/email.mdcovering both variants.For the
m365-oauthvariant, the flow is: MSAL'sConfidentialClientApplicationtakestenantId,clientId, andclientSecretfrom config, and MSAL'sacquireTokenByClientCredentialhandles the OAuth POST + response parsing + in-memory cache + expiry refresh + concurrent-request dedup. Standard OAuth 2 client credentials flow under the covers, no user involved, no refresh token. The token is handed to nodemailer'sauth: { type: 'OAuth2', user, accessToken }shape, which uses it in the SASL XOAUTH2 exchange during the SMTP AUTH step againstsmtp.office365.com:587. Everything else about the send (nodemailer, MIME payload, controllers, templates) stays the same as today.For Garvan's M365 setup, the config values are Garvan's
tenantId, the app registration'sclientIdandclientSecret, plussmtp.office365.com:587. Non-Microsoft deployers usesmtp-basic(see multi-tenant section below). A generic OAuth variant becomes a follow-up when a real second-provider use case appears.Token cache and refresh: MSAL maintains an in-memory cache per backend process. When the cached token nears expiry, MSAL re-acquires from the token endpoint. Concurrent sends share a single in-flight refresh handled by MSAL internally so we don't hit the token endpoint N times per burst. Nothing is written to disk or the database, so a process restart just fetches a fresh token. The client secret itself lives in the existing
ctrl-configK8s Secret alongside the SMTP password we use today, so no new secret-storage path is needed. Rotating the client secret will require re-encrypting the values in the deploy-config repo, same pattern as any other SMTP password change.Multi-tenant
Non-Garvan orgs deploying CTRL keep using the
smtp-basicprovider unchanged. That path covers self-hosted SMTP relays, SendGrid, Mailgun, Amazon SES via SMTP creds, and Google Workspace via App Password. The newm365-oauthvariant is Microsoft-only. If a downstream deployer ever needs OAuth for their own provider (JWT-bearer for Google Workspace service accounts, refresh-token for delegated setups, or a custom client-credentials-capable OAuth server), that becomes a separate provider variant added later, sitting next tom365-oauthin the discriminated union onprovider.What IT needs to do on their side
Once the code lands and the tenant setup begins:
Mail.Sendapplication permission.ctrl-noreply@garvan.org.auspecifically. Application permissionMail.Sendon its own only allows sends from HVE-type accounts, so a regular mailbox has to be scoped in either through Application Access Policies or RBAC for Applications depending on what Garvan uses tenant-wide. This scoping is what unlocks the ability to send from that mailbox, not just what restricts it.ctrl-noreply@garvan.org.auitself as an alias (confirmed by @ignatiusm and @mhebrard).Follow-up questions IT asked, and our answers
From the ITHELP-22014 ticket:
ctrl-noreply? Below 10k a week based on a rough count of current sends (invites, password resets, OTPs, contact-us) and discussion with @mhebrard. Well below the volume that would need a dedicated HVE account. A regular mailbox with the Application Access Policy or RBAC scoping described in the "What IT needs to do" section is enough. Can be revisited later if volumes grow.ctrl-noreply@garvan.org.au? Leaning toward an alias since the app only needs to send, not receive. See the team-decision section below.Still open on IT's side, worth checking with them:
ctrl-noreply. If it is disabled tenant-wide by policy, we would need to pivot to Microsoft GraphsendMailwhich is a bigger code change.Submitted as new IT ticket ITHELP-27087 (references ITHELP-22014), with @ignatiusm and @mhebrard requested as watchers.
Folded into this work
Other issues that live in the same code region or are directly forced by the M365 path:
From, so the sender rework is forced by the M365 path.Closes SMTP error when sending emails in Dev #909 (SMTP error when sending emails in Dev #909 will be closed by @ignatiusm's PR More strict email address parsing (to match nodemailer) #914 (MAIL FROM 501 Bad sender address syntaxin dev). Same root cause as Change the Sender email address #802, so it resolves for free.More strict email address parsing) when it merges. Dropping from this issue's scope.Closes SMTP error crashes backend #719 (SMTP errors crash the backend). Same code region as the transport rework, and M365 adds new failure modes (bad token, expired token, XOAUTH2 rejected) that we have to handle anyway.Per @ignatiusm's review, SMTP error crashes backend #719 stays open because SMTP failure handling remains relevant for thesmtp-basicpath going forward. Addressing partial improvements here for the M365 path (bad token, expired token, XOAUTH2 rejected), but not closing SMTP error crashes backend #719 as part of this PR.Kept out on purpose
Related but going into follow-up PRs after this one lands, so this PR stays reviewable:
application/common/src/emails/. What Email formatting and template organisation #761 asks for is custom-themes-and-logos loading across templates, which is a whole separate feature.Development approach
Original two-option framing (superseded, see Update below)
Two ways this can go, either works.
Option A: use a personal Microsoft 365 Developer Program sandbox tenant for local dev, to test the OAuth path end-to-end as I build it. The sandbox only exists in my local
config.json5during dev and never touches deployed CTRL. When IT provisions the real Garvan tenant, the transition into UAT and prod is just a handful of config values(tenantId, clientId, clientSecret, sender address)swapping in. No code change on our side.Option B: wait for IT to provision Garvan's tenant first, then write and test against it directly. Most of the code work (provider abstraction, config schema, helm changes, sender and failure-handling reworks) can still proceed in parallel while waiting. The difference is when the OAuth path gets verified against Microsoft, either continuously in a sandbox or all at once when Garvan's tenant is ready.
Worth flagging, as @mhebrard has raised before with other services, that we already have parts of CTRL that depend on personal accounts and moving those to prod has been painful. Option A is designed not to be, since the sandbox never touches deployed CTRL and only a handful of config values change at deploy time. If the team would rather skip personal accounts entirely, Option B works too.
Update. Option A is off the table. Microsoft tightened M365 Developer Program eligibility in early 2025 and signup with a personal MSA is now declined without a paid Visual Studio subscription, verified Microsoft Graph API history, or Microsoft Partner Network membership. See Q3 for details.
Going with Option B. Build the provider abstraction, config schema refactor, helm values changes, sender rework (#802), and the MSAL wiring now, against unit tests and a mocked MSAL client. End-to-end verification against real Microsoft servers happens once IT provisions the Garvan tenant. This still gets the bulk of the PR reviewable while IT's side is set up in parallel.
Nice-to-have if IT is willing: a scoped throwaway dev app registration inside Garvan's real M365 tenant with
Mail.Sendlimited to one test mailbox via Application Access Policy. Would let me verify end-to-end during build against the exact target we deploy to. Not blocking, just faster feedback.Questions for the team
Consolidating the calls I'd like @ignatiusm and @mhebrard to weigh in on before I start writing code, so nothing has to be scrolled back to:
Mailbox type for
ctrl-noreply@garvan.org.au.Leaning toward an alias since the app only needs to send. Any concerns with alias over a shared mailbox?Confirmed as alias by both @ignatiusm and @mhebrard.Scope confirmation.
Confirming User-client: contact us form is not study specific: include study name drop down, or allow admins to set study-specific email address. #440 and Email formatting and template organisation #761 stay out for follow-up PRs, and Change the Sender email address #802, SMTP error when sending emails in Dev #909, SMTP error crashes backend #719 close as part of this PR.Per @ignatiusm's review:smtp-basicpath going forward.Development order.
Option A (sandbox during dev) or Option B (wait for IT to provision Garvan's tenant first). Either works, defer to team preference.Confirmed Option A (sandbox during dev) per @ignatiusm's review. Microsoft 365 Developer Program is free (no cost), sandbox tenant renewable every 90 days.Update: Microsoft tightened M365 Developer Program eligibility in early 2025. Signup with a personal MSA is now declined unless you have a paid Visual Studio subscription, verified Microsoft Graph API history, or Microsoft Partner Network membership. Option A is blocked for me personally. @mhebrard also flagged wanting clarification on the two-option framing, which is now moot given the pivot.
Falling back to Option B. All code work (provider abstraction, config schema, helm changes, sender rework, MSAL wiring) proceeds now against unit tests and a mocked MSAL client. End-to-end verification against real Microsoft servers happens when IT provisions the Garvan tenant.
Nice-to-have if IT is open to it: a scoped throwaway dev app registration inside Garvan's real M365 tenant,
Mail.Sendlimited to one test mailbox via Application Access Policy. Would let me verify end-to-end during build against the exact target we deploy to. Not blocking, but faster feedback if available.IT follow-ups.
For the still-open items in the section above (application vs delegated, Access Policies vs RBAC, SMTP AUTH state), would you prefer @mhebrard chases them via the ITHELP-22014 ticket, or should I reach out to Tony directly?Resolved. Per @ignatiusm's suggestion and @mhebrard's confirmation, opened a new ticket ITHELP-27087 referencing ITHELP-22014, with @ignatiusm and @mhebrard requested as watchers. Ticket covers all three still-open items plus the nice-to-have dev app registration ask.