Skip to content

integration tests against sandbox aws account#712

Open
sid88in wants to merge 2 commits into
masterfrom
tests/aws-integration-tests
Open

integration tests against sandbox aws account#712
sid88in wants to merge 2 commits into
masterfrom
tests/aws-integration-tests

Conversation

@sid88in
Copy link
Copy Markdown
Owner

@sid88in sid88in commented May 31, 2026

Live AWS integration tests

This is an opt-in test suite that exercises the plugin's live AWS code
paths (the ones the AWS SDK v3 migration in #686
most affects) against a real AWS account. It is complementary to:

  • npm test — unit tests (src/__tests__), no AWS.
  • npm run test:e2e — offline CloudFormation-synthesis tests (e2e/), no AWS.

Unlike those, the integration suite costs money and needs credentials, so it
never runs in the default CI or as part of npm test / npm run test:e2e /
npm run test:all. It is gated behind the APPSYNC_PLUGIN_INTEGRATION
environment variable and, when that is unset, every suite resolves to
describe.skip and the run exits green.

What it proves

Each scenario maps to a live command and the SDK call it validates:

Tier Command Live SDK call(s) Notes
A appsync evaluate --template EvaluateMappingTemplate no deploy
A appsync evaluate --type/--field EvaluateCode no deploy
B serverless info ListApiKeys, GetGraphqlApi, DescribeStackResources get-api-keys
B appsync get-introspection GetIntrospectionSchema
B appsync env set / env get Put/GetGraphqlApiEnvironmentVariables
B appsync logs FilterLogEvents (CloudWatch Logs) log group from fixture logging
B credential/region proof deploy + live read the headline #686 test
C appsync flush-cache FlushApiCache caching billed hourly
D appsync domain create ListCertificates (ACM, us-east-1 pin) + CreateDomainName
D appsync domain assoc GetApiAssociation (+ NotFoundException path) + AssociateApi
D appsync domain create-record ListHostedZonesByName + ChangeResourceRecordSets + GetChange (poll → INSYNC)

The credential/region proof is the most valuable test: it deploys with an
explicit --region (and optional --aws-profile), asserts the API actually
landed in that region (by reading the stack's AppSync API ARN), confirms a live
command pointed at that region succeeds, and confirms the same command pointed
at a different region fails to find the API — demonstrating that the live
commands honor the region resolved from the Serverless provider, not the bare
default credential chain.

Tiers and gating

Tier Switch (in addition to APPSYNC_PLUGIN_INTEGRATION=1) Cost profile
A — evaluate none negligible (a few AppSync requests)
B — minimal deploy none cents (no hourly charge; 1-day log retention)
C — caching APPSYNC_PLUGIN_INTEGRATION_CACHING=1 hourly caching instance while it exists
D — custom domain APPSYNC_PLUGIN_INTEGRATION_DOMAIN + _HOSTED_ZONE_ID minimal (reuses existing zone + cert)

Tiers are independently skippable: with only credentials set you get A and B;
caching and domain stay skipped until you opt in.

Running it

# Tiers A + B (cheapest useful run)
APPSYNC_PLUGIN_INTEGRATION=1 \
APPSYNC_PLUGIN_INTEGRATION_REGION=us-west-2 \
AWS_PROFILE=my-sandbox \
npm run test:integration

# Add the caching tier
APPSYNC_PLUGIN_INTEGRATION=1 \
APPSYNC_PLUGIN_INTEGRATION_REGION=us-west-2 \
APPSYNC_PLUGIN_INTEGRATION_CACHING=1 \
AWS_PROFILE=my-sandbox \
npm run test:integration

# Full credential/region proof with an explicit profile whose default region
# differs from the test region
APPSYNC_PLUGIN_INTEGRATION=1 \
APPSYNC_PLUGIN_INTEGRATION_REGION=us-west-2 \
APPSYNC_PLUGIN_INTEGRATION_PROFILE=my-sandbox \
npm run test:integration

Environment variables

Variable Required Description
APPSYNC_PLUGIN_INTEGRATION yes Master switch; must be 1.
APPSYNC_PLUGIN_INTEGRATION_REGION no Test region (default us-west-2). Choose something other than us-east-1 so the ACM us-east-1 pin and the region proof are meaningful.
APPSYNC_PLUGIN_INTEGRATION_PROFILE no Named profile for the deploy/commands; also enables the profile half of the credential proof. Falls back to the default credential chain.
APPSYNC_PLUGIN_INTEGRATION_OTHER_REGION no Region used by the negative half of the region proof (default: the opposite of the test region).
APPSYNC_PLUGIN_INTEGRATION_EXPECTED_ACCOUNT_ID no If set, the profile proof asserts the deployed API's account matches it.
APPSYNC_PLUGIN_INTEGRATION_CACHING no 1 to run the caching tier.
APPSYNC_PLUGIN_INTEGRATION_DOMAIN no Domain name for the custom-domain tier (e.g. it.example.com).
APPSYNC_PLUGIN_INTEGRATION_HOSTED_ZONE_ID no Route53 hosted zone id for that domain.
APPSYNC_PLUGIN_INTEGRATION_CERT_ARN no ISSUED ACM cert ARN (us-east-1). If omitted, the plugin discovers a matching cert via ListCertificates.
SERVERLESS_BIN no Path to a Serverless binary (e.g. a v4 install). Defaults to the repo's v3.

Standard AWS credential variables (AWS_PROFILE, AWS_ACCESS_KEY_ID, OIDC
AWS_ROLE_ARN, …) are honored as usual.

Typecheck only

npm run test:integration:typecheck   # tsc -p tsconfig.integration.json

Teardown and leaked-resource recovery

Reliable teardown is the suite's top priority:

  • Every resource is named with a unique per-run id (appsync-plugin-it-<ts>-<rand>)
    and tagged appsync-plugin-integration: <run id>.
  • Deploy tiers tear down with serverless remove in afterAll (runs even on
    failure). The custom-domain tier additionally deletes its non-CloudFormation
    resources (Route53 record → API association → domain name) in reverse order.
    The ACM certificate is reused, never created, and never deleted.
  • If a run is interrupted, recover leaks with the standalone sweeper:
APPSYNC_PLUGIN_INTEGRATION=1 \
APPSYNC_PLUGIN_INTEGRATION_REGION=us-west-2 \
AWS_PROFILE=my-sandbox \
[APPSYNC_PLUGIN_INTEGRATION_DOMAIN=it.example.com] \
npm run test:integration:sweep

The sweeper deletes AppSync APIs tagged by the suite, CloudFormation stacks
named with the run-id prefix, and (if a domain is configured) a leaked custom
domain name. It is idempotent and safe to re-run. CloudFormation stack deletes
are asynchronous — verify completion in the console.

Least-privilege IAM policy

The actions actually used by the suite are below. Note that a serverless deploy is itself a CloudFormation operation that creates an AppSync API, an
API key, a CloudWatch log group + logging role, and an S3 deployment bucket;
fully constraining a deploy role is involved, so in a throwaway sandbox account
many teams simply use a broader deploy role. The policy below is the scoped
target.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AppSync",
      "Effect": "Allow",
      "Action": [
        "appsync:CreateGraphqlApi",
        "appsync:DeleteGraphqlApi",
        "appsync:UpdateGraphqlApi",
        "appsync:GetGraphqlApi",
        "appsync:ListGraphqlApis",
        "appsync:StartSchemaCreation",
        "appsync:GetSchemaCreationStatus",
        "appsync:GetIntrospectionSchema",
        "appsync:CreateApiKey",
        "appsync:DeleteApiKey",
        "appsync:ListApiKeys",
        "appsync:CreateDataSource",
        "appsync:DeleteDataSource",
        "appsync:UpdateDataSource",
        "appsync:CreateResolver",
        "appsync:DeleteResolver",
        "appsync:CreateFunction",
        "appsync:DeleteFunction",
        "appsync:FlushApiCache",
        "appsync:CreateApiCache",
        "appsync:DeleteApiCache",
        "appsync:EvaluateCode",
        "appsync:EvaluateMappingTemplate",
        "appsync:GetGraphqlApiEnvironmentVariables",
        "appsync:PutGraphqlApiEnvironmentVariables",
        "appsync:TagResource",
        "appsync:UntagResource",
        "appsync:ListTagsForResource",
        "appsync:CreateDomainName",
        "appsync:DeleteDomainName",
        "appsync:GetDomainName",
        "appsync:ListDomainNames",
        "appsync:AssociateApi",
        "appsync:DisassociateApi",
        "appsync:GetApiAssociation"
      ],
      "Resource": "*"
    },
    {
      "Sid": "CloudFormation",
      "Effect": "Allow",
      "Action": [
        "cloudformation:CreateStack",
        "cloudformation:UpdateStack",
        "cloudformation:DeleteStack",
        "cloudformation:DescribeStacks",
        "cloudformation:DescribeStackResources",
        "cloudformation:DescribeStackEvents",
        "cloudformation:GetTemplate",
        "cloudformation:ListStacks",
        "cloudformation:ValidateTemplate",
        "cloudformation:CreateChangeSet",
        "cloudformation:DescribeChangeSet",
        "cloudformation:ExecuteChangeSet",
        "cloudformation:DeleteChangeSet"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Logs",
      "Effect": "Allow",
      "Action": [
        "logs:FilterLogEvents",
        "logs:CreateLogGroup",
        "logs:DeleteLogGroup",
        "logs:PutRetentionPolicy",
        "logs:DescribeLogGroups",
        "logs:TagResource"
      ],
      "Resource": "*"
    },
    {
      "Sid": "IamForDeploy",
      "Effect": "Allow",
      "Action": [
        "iam:CreateRole",
        "iam:DeleteRole",
        "iam:GetRole",
        "iam:PassRole",
        "iam:PutRolePolicy",
        "iam:DeleteRolePolicy",
        "iam:AttachRolePolicy",
        "iam:DetachRolePolicy",
        "iam:TagRole"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DeploymentBucket",
      "Effect": "Allow",
      "Action": [
        "s3:CreateBucket",
        "s3:DeleteBucket",
        "s3:ListBucket",
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:PutBucketPolicy",
        "s3:GetBucketPolicy",
        "s3:PutBucketTagging",
        "s3:GetEncryptionConfiguration",
        "s3:PutEncryptionConfiguration"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DomainTierOnly",
      "Effect": "Allow",
      "Action": [
        "acm:ListCertificates",
        "route53:ListHostedZonesByName",
        "route53:ChangeResourceRecordSets",
        "route53:GetChange"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Identity",
      "Effect": "Allow",
      "Action": ["sts:GetCallerIdentity"],
      "Resource": "*"
    }
  ]
}

If you skip the custom-domain tier (the default), the DomainTierOnly statement
and the domain-related AppSync actions are unnecessary.

Serverless v4 note

The suite spawns the serverless binary from node_modules (override with
SERVERLESS_BIN), so it can be pointed at a Serverless v4 install. Two caveats,
unverified here:

  • v4 may require SERVERLESS_ACCESS_KEY / a license and suppression of login
    prompts (the wrappers already disable telemetry and interactive setup).
  • v4 resolves credentials as SDK v3 objects, so the v2-style getPromise /
    expireTime branch in the plugin's resolveCredentials will not fire (it
    degrades gracefully). Re-confirm the credential/region proof under v4.

Summary by CodeRabbit

  • Tests

    • Added an opt-in live AWS integration test suite with gated tiers (evaluate, deploy, caching, custom domain), automated best-effort cleanup/sweeper, and helpers for running Serverless commands and credential gating.
  • Documentation

    • Added comprehensive integration testing docs: how to opt in, required env, teardown behavior, and least-privilege IAM guidance.
  • Chores

    • Added a manual CI workflow to run the integration suite and new npm scripts to run, typecheck, and sweep integration tests.

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 31, 2026

Review Change Stack

📝 Walkthrough

Walkthrough

Adds an opt-in live AWS integration test system: environment gating, run-id/project provisioning, Serverless/AWS helpers, four gated integration suites (evaluate, deploy, caching, domain), a leaked-resource sweeper, Jest/TS config and npm scripts, plus a manual GitHub Actions workflow and documentation.

Changes

Live AWS Integration Testing Infrastructure

Layer / File(s) Summary
Configuration and test gating
integration/helpers/config.ts, integration/helpers/gate.ts
Environment parsing determines whether integration tests run and provides gating wrappers that toggle describe/describe.skip per tier.
Test project provisioning & run-id
integration/helpers/run-id.ts, integration/helpers/project.ts
Generates unique run IDs, creates temporary Serverless/AppSync projects (schema, resolver, serverless.yml), symlinks the plugin, and returns cleanup callbacks.
Serverless CLI & AWS helpers
integration/helpers/sls.ts, integration/helpers/aws.ts
Runs Serverless CLI with region/profile defaults and robust error wrapping; builds AWS SDK clients, parses ARNs, discovers leaked resources, and deletes AppSync/CloudFormation/domain resources.
Integration test suites (Tiers A–D)
integration/eval.integration.test.ts, integration/deploy.integration.test.ts, integration/caching.integration.test.ts, integration/domain.integration.test.ts
Tier A: evaluate mapping/templates (no deploy); Tier B: deploy + live AppSync commands with region/account proof; Tier C: caching flush-cache test; Tier D: custom domain create/assoc/record.
Leaked resource cleanup and sweeper
integration/sweeper.ts
Idempotent sweeper that finds tagged/prefixed leaked APIs and stacks and attempts deletion, with optional custom-domain cleanup.
Jest, tsconfig, and npm scripts
jest.integration.config.ts, jest.integration.setup.ts, tsconfig.integration.json, package.json
Dedicated Jest config for integration suite (long timeouts, serialized execution, globalSetup), an integration tsconfig, and scripts: test:integration, test:integration:typecheck, test:integration:sweep.
GitHub Actions workflow and docs
.github/workflows/integration.yml, doc/integration-tests.md
Manual workflow_dispatch workflow using OIDC to assume an AWS role, scoped environment and concurrency, and an always-run sweep; docs describe tiers, env vars, least-privilege IAM policy, and teardown/sweeper behavior.

Sequence Diagram

sequenceDiagram
  participant CI as GitHub Actions
  participant Runner as Workflow Runner
  participant Build as Build Step
  participant Tests as Integration Tests
  participant AWS as AWS Services
  participant Sweeper as Resource Sweeper
  CI->>Runner: workflow_dispatch
  Runner->>Build: checkout & setup (Node, npm ci)
  Build->>Tests: npm run test:integration (jest)
  Tests->>AWS: Tier A: evaluate (no deploy)
  Tests->>AWS: Tier B: deploy & live CLI commands
  Tests->>AWS: Tier C: caching commands
  Tests->>AWS: Tier D: domain provisioning
  Runner->>Sweeper: npm run test:integration:sweep (always)
  Sweeper->>AWS: delete leaked AppSync APIs
  Sweeper->>AWS: delete leaked CloudFormation stacks
  Sweeper->>AWS: cleanup custom domains
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Suggested reviewers

  • AlexHladin

Poem

🐰 Hopped in with a run-id bright,

I spin up schemas through the night,
I flush the cache and check the logs,
then sweep stray stacks from clouded bogs — hooray! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 57.89% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'integration tests against sandbox aws account' clearly describes the main change: adding integration tests for a real AWS account.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch tests/aws-integration-tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@sid88in sid88in requested a review from AlexHladin May 31, 2026 00:23
@sid88in sid88in self-assigned this May 31, 2026
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🧹 Nitpick comments (2)
.github/workflows/integration.yml (1)

52-52: 💤 Low value

Pin aws-actions/configure-aws-credentials to a commit SHA (ideally consistently across workflows).

integration.yml uses a floating tag (aws-actions/configure-aws-credentials@v4); since this action is commonly used for OIDC role assumption, pinning to a full commit SHA reduces supply-chain risk. Also, other workflows here already use floating tags (e.g., actions/checkout@v4 / actions/setup-node@v4 in the same file, actions/checkout@v6 / actions/setup-node@v6 in ci.yml, actions/checkout@v5 / actions/setup-node@v5 and softprops/action-gh-release@v2 in release.yml), so consider adopting a repo-wide pinning policy rather than pinning only this one action.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/integration.yml at line 52, Replace the floating action
tag aws-actions/configure-aws-credentials@v4 with a pinned full commit SHA for
that action (and update any other workflow usages to the same pinned SHA for
consistency); locate the line referencing
aws-actions/configure-aws-credentials@v4 in the workflow and change it to the
corresponding commit SHA from the action's repository, and consider applying the
same pinning policy to other actions like actions/checkout and
actions/setup-node across the repository so all workflows use fixed SHAs rather
than floating tags.
integration/helpers/sls.ts (1)

80-93: ⚡ Quick win

Set an explicit maxBuffer for execFileSync.

execFileSync defaults to a 1 MB stdout/stderr buffer. A deploy/remove with verbose CloudFormation output can exceed this, causing an ENOBUFS kill that surfaces as a spurious SlsCommandError rather than the real deploy result.

♻️ Proposed fix
     const stdout = execFileSync(SERVERLESS_BIN, fullArgs, {
       cwd: options.cwd,
       timeout: options.timeout ?? 240_000,
+      maxBuffer: 64 * 1024 * 1024,
       env: {
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@integration/helpers/sls.ts` around lines 80 - 93, The execFileSync call that
runs Serverless (using SERVERLESS_BIN, fullArgs, and options) does not set
maxBuffer and can overflow the default 1MB; update the options object passed to
execFileSync in this helper (where stdio and env are set) to include a larger
maxBuffer (e.g., 10*1024*1024 or 50*1024*1024) so verbose CloudFormation output
won’t cause an ENOBUFS/SlsCommandError; keep the other env/stdio settings intact
and ensure you apply the new maxBuffer to the same options object used by
execFileSync.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/integration.yml:
- Line 42: Replace the default actions/checkout@v4 behavior so it does not
persist GITHUB_TOKEN into .git/config: update the checkout step referencing
actions/checkout@v4 to include persist-credentials: false (so the job
authenticates via OIDC only and no credentials are written to the repo config).

In `@doc/integration-tests.md`:
- Line 49: Update the Tier D env var entry so it references the full canonical
name instead of the truncated suffix: replace the `_HOSTED_ZONE_ID` fragment
used alongside `APPSYNC_PLUGIN_INTEGRATION_DOMAIN` with the complete
`APPSYNC_PLUGIN_INTEGRATION_HOSTED_ZONE_ID` variable name (matching the
documented canonical name used elsewhere) so the table entry and any examples
use `APPSYNC_PLUGIN_INTEGRATION_DOMAIN` +
`APPSYNC_PLUGIN_INTEGRATION_HOSTED_ZONE_ID`.

In `@integration/helpers/aws.ts`:
- Around line 167-173: The current check uses client.send(new
ListDomainNamesCommand({ maxResults: 25 })) and only inspects domainNameConfigs
for a match, which misses domains on later pages; replace this single-call
lookup with a paginated loop (similar to findLeakedApis/findLeakedStacks) that
repeatedly calls ListDomainNamesCommand using nextToken until all pages are
consumed or a domainName match is found, then set exists accordingly and only
return when no pages remain and no match was found; update the logic around
domainNameConfigs/exists to use the paginated result.
- Around line 174-183: The catch block around client.send(new
GetApiAssociationCommand({ domainName })) / DisassociateApiCommand currently
swallows all errors; change it so only the NotFoundException is ignored and all
other errors are propagated (or rethrown) so real failures surface.
Specifically, in the catch for the try containing GetApiAssociationCommand and
DisassociateApiCommand, detect whether err is an Error and err.name ===
'NotFoundException' and return/do-nothing only in that case; for any other error
(including non-Error values) rethrow (or throw a new Error preserving the
original) so permission/throttling/other failures are not silently swallowed.

In `@integration/sweeper.ts`:
- Around line 46-91: The sweeper currently swallows per-resource errors in
main() (during deleteApi, cleanupDomainName, deleteStack) so process exits 0
even when deletions fail; add failure tracking inside main: introduce a boolean
or counter (e.g., hadFailures or failureCount) and set it when any catch block
catches an error for deleteApi, cleanupDomainName, or deleteStack, then after
all work if hadFailures is true set process.exitCode = 1 (or throw an Error) so
the CI run fails; update log messages unchanged but ensure the catch blocks for
those functions update the shared failure flag.

---

Nitpick comments:
In @.github/workflows/integration.yml:
- Line 52: Replace the floating action tag
aws-actions/configure-aws-credentials@v4 with a pinned full commit SHA for that
action (and update any other workflow usages to the same pinned SHA for
consistency); locate the line referencing
aws-actions/configure-aws-credentials@v4 in the workflow and change it to the
corresponding commit SHA from the action's repository, and consider applying the
same pinning policy to other actions like actions/checkout and
actions/setup-node across the repository so all workflows use fixed SHAs rather
than floating tags.

In `@integration/helpers/sls.ts`:
- Around line 80-93: The execFileSync call that runs Serverless (using
SERVERLESS_BIN, fullArgs, and options) does not set maxBuffer and can overflow
the default 1MB; update the options object passed to execFileSync in this helper
(where stdio and env are set) to include a larger maxBuffer (e.g., 10*1024*1024
or 50*1024*1024) so verbose CloudFormation output won’t cause an
ENOBUFS/SlsCommandError; keep the other env/stdio settings intact and ensure you
apply the new maxBuffer to the same options object used by execFileSync.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: d2c43fc7-53d4-4131-a416-0926f5acc0f3

📥 Commits

Reviewing files that changed from the base of the PR and between 6bbd830 and e7ee83e.

📒 Files selected for processing (17)
  • .github/workflows/integration.yml
  • doc/integration-tests.md
  • integration/caching.integration.test.ts
  • integration/deploy.integration.test.ts
  • integration/domain.integration.test.ts
  • integration/eval.integration.test.ts
  • integration/helpers/aws.ts
  • integration/helpers/config.ts
  • integration/helpers/gate.ts
  • integration/helpers/project.ts
  • integration/helpers/run-id.ts
  • integration/helpers/sls.ts
  • integration/sweeper.ts
  • jest.integration.config.ts
  • jest.integration.setup.ts
  • package.json
  • tsconfig.integration.json

Comment thread .github/workflows/integration.yml
Comment thread doc/integration-tests.md Outdated
Comment thread integration/helpers/aws.ts Outdated
Comment thread integration/helpers/aws.ts
Comment thread integration/sweeper.ts
Sibling to the offline e2e suite, exercising the plugin's live AWS code
paths (post AWS-SDK-v3 migration in #686) against a real account. Opt-in
via APPSYNC_PLUGIN_INTEGRATION=1; skips cleanly and is excluded from
npm test / test:e2e / test:all and default CI.

- Tier A: appsync evaluate (VTL + JS), no deploy
- Tier B: minimal deploy -> info/get-introspection/env/logs + the
  credential/region proof for #686
- Tier C: caching/flush-cache (gated)
- Tier D: custom domain + Route53 + ACM (gated, off by default)
- run-id tagging + serverless remove teardown + standalone sweeper
- doc/integration-tests.md, optional OIDC workflow (manual/scheduled)
@sid88in sid88in force-pushed the tests/aws-integration-tests branch from e7ee83e to 498318e Compare May 31, 2026 05:01
@sid88in
Copy link
Copy Markdown
Owner Author

sid88in commented May 31, 2026

Screenshot 2026-05-30 at 10 02 43 PM

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/integration.yml:
- Around line 29-32: The workflow's concurrency.group currently uses
integration-${{ github.ref }} which only serializes runs per ref; update the
concurrency.group to use a constant or environment-scoped account identifier so
runs are serialized per AWS account (e.g., replace integration-${{ github.ref }}
with a fixed key or integration-${{ env.AWS_ACCOUNT_ID }} / integration-${{
secrets.AWS_ACCOUNT_ID }}), keeping the concurrency block and cancel-in-progress
behavior intact; modify the workflow to ensure AWS account ID is provided to the
job environment (or use a fixed literal) so the concurrency.group truly enforces
one live suite per account.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: f8bc5b82-7328-4faa-a328-72fb471576c1

📥 Commits

Reviewing files that changed from the base of the PR and between e7ee83e and 498318e.

📒 Files selected for processing (17)
  • .github/workflows/integration.yml
  • doc/integration-tests.md
  • integration/caching.integration.test.ts
  • integration/deploy.integration.test.ts
  • integration/domain.integration.test.ts
  • integration/eval.integration.test.ts
  • integration/helpers/aws.ts
  • integration/helpers/config.ts
  • integration/helpers/gate.ts
  • integration/helpers/project.ts
  • integration/helpers/run-id.ts
  • integration/helpers/sls.ts
  • integration/sweeper.ts
  • jest.integration.config.ts
  • jest.integration.setup.ts
  • package.json
  • tsconfig.integration.json
✅ Files skipped from review due to trivial changes (1)
  • doc/integration-tests.md
🚧 Files skipped from review as they are similar to previous changes (11)
  • jest.integration.setup.ts
  • integration/eval.integration.test.ts
  • package.json
  • tsconfig.integration.json
  • integration/sweeper.ts
  • integration/helpers/run-id.ts
  • integration/caching.integration.test.ts
  • integration/helpers/config.ts
  • integration/deploy.integration.test.ts
  • integration/helpers/project.ts
  • integration/helpers/sls.ts

Comment on lines +29 to +32
concurrency:
# Never run two live suites against the same account at once.
group: integration-${{ github.ref }}
cancel-in-progress: false
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Concurrency group keyed by github.ref doesn't enforce the stated account-wide serialization.

The comment states this should "Never run two live suites against the same account at once," but group: integration-${{ github.ref }} only serializes runs originating from the same ref. Two dispatches from different branches/tags map to different groups and will run concurrently against the same AWS account. Use a constant (or environment-scoped) group key to actually serialize per account.

🔒 Proposed fix
 concurrency:
   # Never run two live suites against the same account at once.
-  group: integration-${{ github.ref }}
+  group: integration
   cancel-in-progress: false
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
concurrency:
# Never run two live suites against the same account at once.
group: integration-${{ github.ref }}
cancel-in-progress: false
concurrency:
# Never run two live suites against the same account at once.
group: integration
cancel-in-progress: false
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/integration.yml around lines 29 - 32, The workflow's
concurrency.group currently uses integration-${{ github.ref }} which only
serializes runs per ref; update the concurrency.group to use a constant or
environment-scoped account identifier so runs are serialized per AWS account
(e.g., replace integration-${{ github.ref }} with a fixed key or integration-${{
env.AWS_ACCOUNT_ID }} / integration-${{ secrets.AWS_ACCOUNT_ID }}), keeping the
concurrency block and cancel-in-progress behavior intact; modify the workflow to
ensure AWS account ID is provided to the job environment (or use a fixed
literal) so the concurrency.group truly enforces one live suite per account.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
integration/helpers/aws.ts (1)

119-130: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

DELETE_FAILED stacks are not swept and will leak.

The StackStatusFilter lists terminal create/update/rollback states but omits DELETE_FAILED. If a previous deleteStack (or serverless remove) left a stack in DELETE_FAILED, findLeakedStacks won't return it, so the sweeper silently skips it and the billable resources persist.

🛡️ Proposed fix
         StackStatusFilter: [
           'CREATE_COMPLETE',
           'UPDATE_COMPLETE',
           'ROLLBACK_COMPLETE',
           'UPDATE_ROLLBACK_COMPLETE',
           'CREATE_FAILED',
           'ROLLBACK_FAILED',
+          'DELETE_FAILED',
         ],
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@integration/helpers/aws.ts` around lines 119 - 130, The StackStatusFilter
passed to ListStacksCommand inside findLeakedStacks (where ListStacksCommand and
variables StackSummaries/NextToken are used) omits the DELETE_FAILED status so
stacks in DELETE_FAILED are not returned and leak; update the StackStatusFilter
array used in the ListStacksCommand call to include 'DELETE_FAILED' so those
stacks are returned and can be swept. Ensure only the StackStatusFilter array is
changed (add 'DELETE_FAILED') and keep pagination/NextToken handling unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@integration/helpers/aws.ts`:
- Around line 119-130: The StackStatusFilter passed to ListStacksCommand inside
findLeakedStacks (where ListStacksCommand and variables StackSummaries/NextToken
are used) omits the DELETE_FAILED status so stacks in DELETE_FAILED are not
returned and leak; update the StackStatusFilter array used in the
ListStacksCommand call to include 'DELETE_FAILED' so those stacks are returned
and can be swept. Ensure only the StackStatusFilter array is changed (add
'DELETE_FAILED') and keep pagination/NextToken handling unchanged.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4dead1ef-fa9d-4919-968d-a27ea52ae650

📥 Commits

Reviewing files that changed from the base of the PR and between 498318e and 2ce9bf0.

📒 Files selected for processing (5)
  • .github/workflows/integration.yml
  • doc/integration-tests.md
  • integration/helpers/aws.ts
  • integration/helpers/sls.ts
  • integration/sweeper.ts
✅ Files skipped from review due to trivial changes (1)
  • doc/integration-tests.md
🚧 Files skipped from review as they are similar to previous changes (2)
  • .github/workflows/integration.yml
  • integration/helpers/sls.ts

# Extra guard: only runs from the canonical repo, never a fork.
if: github.repository == 'sid88in/serverless-appsync-plugin'
runs-on: ubuntu-latest
timeout-minutes: 45
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sid88in how about 5 minutes? 45 looks like too big value.

# into .git/config, since npm ci + the suite run third-party code.
persist-credentials: false

- uses: actions/setup-node@v4
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sid88in use please v6


- uses: actions/setup-node@v4
with:
node-version: 20
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sid88in aws lambda supports Node.js 22, and 24, let's avoid 20.

@AlexHladin
Copy link
Copy Markdown
Collaborator

@sid88in before merging this PR please check that integration folder will not be included in npm build.

Option 1: Update .npmignore (preferred)

# In .npmignore (add these lines)
integration/
jest.integration.config.ts
jest.integration.setup.ts
tsconfig.integration.json
.github/
doc/
__tests__/
e2e/

Option 2: Use files field in package.json (most explicit)

json{
  "files": [
    "lib/",
    "dist/",
    "README.md",
    "LICENSE"
  ]
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants