[iankhou]

Problem statement

Addresses an incorrect CloudTrail attribution reported in #2084.

After a stack deployment fails, the diagnoser fetches CloudTrail events from the failure window and attributes errored events to the stack by matching the calling principal's identity (role names, function names, physical IDs) against the stack's resources.

Sometimes this results in an unrelated event, since those identities can be shared with resources outside of the stack. An example is when we use Role.fromRoleArn, which brings in a Role that could be created in a different stack (or not in a stack at all).

Reproduction of the problem

A first stack creates a role, ct-repro-shared-role, and a Lambda, ct-repro-unrelated-fn, that runs on it and is denied dynamodb:DescribeTable:

const noise = new cdk.Stack(app, 'CtReproUnrelated');
const role = new iam.Role(noise, 'SharedRole', {
  roleName: 'ct-repro-shared-role',
  assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com'),
  managedPolicies: [iam.ManagedPolicy.fromAwsManagedPolicyName('service-role/AWSLambdaBasicExecutionRole')],
});
new lambda.Function(noise, 'NoisyFn', {
  functionName: 'ct-repro-unrelated-fn',
  runtime: lambda.Runtime.NODEJS_20_X,
  handler: 'index.handler',
  role,
  code: lambda.Code.fromInline(`
const { DynamoDBClient, DescribeTableCommand } = require('@aws-sdk/client-dynamodb');
exports.handler = async () => {
  try { await new DynamoDBClient({}).send(new DescribeTableCommand({ TableName: 'ct-repro-some-table' })); }
  catch (e) { return e.name; }
};`),
});

A second stack imports that role as a Lambda's execution role, and fails on an unrelated resource. Nothing in this stack ever assumes the role during the deployment:

const failing = new cdk.Stack(app, 'CtReproFailing');
const imported = iam.Role.fromRoleArn(failing, 'Imported',
  `arn:aws:iam::${process.env.CDK_DEFAULT_ACCOUNT}:role/ct-repro-shared-role`, { mutable: false });
new lambda.Function(failing, 'MyFn', {
  runtime: lambda.Runtime.NODEJS_20_X,
  handler: 'index.handler',
  role: imported,
  code: lambda.Code.fromInline('exports.handler = async () => "ok";'),
});
// Fails at provisioning time: the VPC does not exist.
new ec2.CfnSecurityGroup(failing, 'BrokenSg', { groupDescription: 'x', vpcId: 'vpc-0123456789abcdef0' });

Deploy CtReproUnrelated, invoke ct-repro-unrelated-fn a few times, then deploy CtReproFailing within the next few minutes.

Before: the output attributes ct-repro-unrelated-fn's calls to MyFn, as the session ARN assumed-role/ct-repro-shared-role/ct-repro-unrelated-fn shows:

CtReproFailing/BrokenSg  (AWS::EC2::SecurityGroup BrokenSg)
  Resource handler returned message: "The vpc ID 'vpc-0123456789abcdef0' does not exist (Service: Ec2, Status Code: 400,
  Request ID: 2f7b82da-1df4-4bb9-87ed-e84b4cfee25b) (SDK Attempt Count: 1)" (RequestToken:
  5818f420-9d18-dd07-8f39-5aacafcc2431, HandlerErrorCode: InvalidRequest)
Source Location: <anonymous> (/private/tmp/ct-repro/app.js:43:1)
                 ...node internals...

   MyFn6F8F742F (via Role): ResourceNotFoundException on logs.amazonaws.com:CreateLogStream — The specified log group does not exist.
   MyFn6F8F742F (via Role): AccessDenied on dynamodb.amazonaws.com:DescribeTable — User: arn:aws:sts::123456789012:assumed-role/ct-repro-shared-role/ct-repro-unrelated-fn is not authorized to perform: dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-west-2:123456789012:table/ct-repro-some-table because no identity-based policy allows the dynamodb:DescribeTable action (3 occurrences, earliest shown)

   CloudTrail delivery can lag up to 15 minutes behind the failure; re-run `cdk diagnose` in a few minutes to check for additional control-plane errors (e.g. AccessDenied).

After: cdk diagnose, run once CloudTrail had delivered six of ct-repro-unrelated-fn's AccessDenied events inside the failure window, reports no CloudTrail errors for the stack:

❌ Stack CtReproFailing:
Resource updates failed:
CtReproFailing/BrokenSg  (AWS::EC2::SecurityGroup BrokenSg)
  Resource handler returned message: "The vpc ID 'vpc-0123456789abcdef0' does not exist (Service: Ec2, Status Code: 400,
  Request ID: ac6618bf-4042-4eb6-97b3-dc0bd5e0c904) (SDK Attempt Count: 1)" (RequestToken:
  80ba9477-6c66-8880-81a7-5937e20fb417, HandlerErrorCode: InvalidRequest)
Source Location: <anonymous> (/private/tmp/ct-repro/app.js:43:1)
                 ...node internals...

Changes

Stack identities are now the roles and Lambda functions the stack operation involved: the ones currently in the stack (DescribeStackResources), plus the ones that appear in the operation's stack events. The events cover resources the operation created and then rolled back, which the stack no longer lists. cdk deploy and cdk diagnose build this list the same way, and the CloudTrail investigation no longer reads the template.

Roles the stack only references (imported roles) are therefore no longer stack identities. This means we ignore the CloudTrail lookups for them. Physical IDs of other resource types (buckets, stages, etc.) are no longer identities either, since only roles and functions make calls. Calls by a stack resource running on an imported role still correlate, through the resource's own identity (e.g. a Lambda's session name is its function name). A custom resource's backing function in the stack is matched as a regular Lambda function, so its errors are now labeled with the function rather than the custom resource.

CloudTrail events are matched by the resource that made the control-plane call. For example, when a Lambda function makes a call, CloudTrail records that resource in userIdentity.inScopeOf.credentialsIssuedTo. If this field is present, the event belongs to the stack only if that resource does; the role it ran under is ignored. A function outside the stack therefore cannot have its calls attributed to the stack just because it shares a role with it, even a role the stack created. Events without inScopeOf (e.g. a service assuming a role with a generated session name) still match on the caller's role and session ARNs, but only against identities the stack owns (no imported roles).

After a recent failure, we suggest re-running cdk diagnose because CloudTrail may not have delivered the events yet. We now skip that suggestion if CloudTrail has already delivered any event from after the sweep window. CloudTrail does not guarantee delivery in order, so this is a good guess rather than a guarantee.

                    Errored CloudTrail event
                               │
 BEFORE                        ▼
   caller strings:  session ARN + role ARN + inScopeOf resource
   stack identities: physical IDs, ServiceToken functions,
                     roles referenced in the template (incl. imported)
   → match if ANY caller string matches ANY identity

 AFTER                         ▼
               has inScopeOf.credentialsIssuedTo?
                 ┌─────── yes ───┴─── no ───────┐
                 ▼                              ▼
   match only that resource       match session ARN + role ARN
                 └──────────────┬───────────────┘
                                ▼
   stack identities: roles + Lambda functions in the stack,
                     and in the operation's stack events   (no imported roles)

 Repro event: assumed-role/ct-repro-shared-role/ct-repro-unrelated-fn
   BEFORE: role "ct-repro-shared-role" is imported by MyFn  → attributed to MyFn ✗
   AFTER:  inScopeOf = function:ct-repro-unrelated-fn, not in stack → excluded ✓

Risks

Trade-offs

Errors from a service using an imported role for a stack resource (like Firehose or CodeBuild assuming a role the stack imported, with a service-generated session name) are no longer surfaced. They cannot be told apart from another stack's use of the same role, and the diagnoser prefers "no events found" over an incorrect attribution. The same applies to custom resources whose provider function lives outside the stack.

Known limitations

Events without inScopeOf can still be attributed to the stack when:

• a role the stack created is assumed by something outside the stack (e.g. the role is exported, or its trust policy allows other principals);
• the stack is the bootstrap stack (CDKToolkit), whose deploy and exec roles are used by every other app deploying in the same window.

Testing

• Unit tests
• Manual verification with reproduction stacks
• Re-ran the validation stacks from feat(toolkit-lib): surface CloudTrail control-plane errors for failed resources #1676, deployed concurrently in one account. cdk diagnose attributed each root cause to the same resource as before (the handler function for the custom-resource stacks, the execution role for the VPC/ENI stack), found nothing for the negative case, and attributed no events across stacks. The re-run suggestion was dropped once later events had been delivered.

Checklist

• This change contains a major version upgrade for a dependency and I confirm all breaking changes are addressed
  • Release notes for the new version:

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[maintainer-for-aws]

Automated review

A maintainer will still review this — treat the notes below as a starting point.

📌 Commit b95d72f

This PR fixes a correctness bug in toolkit-lib's post-deployment stack diagnoser, which correlates CloudTrail control-plane errors against a failed cdk deploy stack. Previously the diagnoser built its set of stack identities broadly — including imported roles referenced in the template — so an unrelated resource sharing a role could have its errors wrongly attributed to the failing stack. The change re-anchors matching on principals actually seen in the operation's stack events plus the roles/functions currently in the stack, prefers CloudTrail's inScopeOf.credentialsIssuedTo resource when matching, drops the template-parsing machinery, and suppresses the "re-run diagnose" latency hint once CloudTrail has caught up. The change is internal to toolkit-lib (no public API, schema, or message-format break), its fallback paths fail safe as commented, and it carries matching test coverage across all three touched source files, including the shared-role exclusion, inScopeOf precedence, replaced/rolled-back principals, and the DescribeStackResources failure path. No issues found on this change.

🔴 0 blocking · 🟡 0 recommended · ⚪ 0 optional

---

Generated automatically. React 👍 or 👎 to tell us whether this review helped, so we can improve these reviews.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.01%. Comparing base (611e948) to head (b95d72f).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2120      +/-   ##
==========================================
- Coverage   92.05%   92.01%   -0.04%     
==========================================
  Files          82       82              
  Lines       13195    13195              
  Branches     1894     1893       -1     
==========================================
- Hits        12146    12142       -4     
- Misses       1011     1015       +4     
  Partials       38       38              

Flag	Coverage Δ
suite.unit	92.01% <ø> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[rittneje]

@iankhou Why would roles be relevant to the stack failure other than the ones used by a custom resource provider? I see a few different flaws here:

1. A role not used by the deployment might be used concurrently during the window by something else (for example, by an actual Lambda invocation). That should not count.
2. An imported role is used for a custom resource provider within the stack. That should count.
3. A pre-existing custom resource provider is referenced within the stack. That should count.

Of course, the Achilles heel of this entire design is that it is not generally possible to correlate CloudTrail events with a particular stack deployment, which is why I do not understand why it is part of cdk deploy at all, let alone enabled by default.