Add CTST tests for storage usage reporting API by delthas · Pull Request #2340 · scality/Zenko

delthas · 2026-03-02T15:20:02Z

Summary

Add end-to-end CTST tests for the new GET /instance/{uuid}/reporting/usage Pensieve route
Tests cover: StorageManager happy path (200 + structure validation), unauthorized Keycloak user (403), multi-account presence, and metric accuracy (objectsTotal/bytesTotal strict equality)
Extend managementAPIRequest with optional username/password params for testing different Keycloak users
Extend prepareMetricsScenarios with objectSize/objectCount options for uploading non-empty test objects

bert-e · 2026-03-02T15:20:07Z

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-03-02T15:20:40Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

delthas · 2026-03-02T15:30:59Z

Depends on https://github.com/scality/pensieve-api/pull/351 and https://github.com/scality/vault2/pull/189

solution/deps.yaml

tests/ctst/steps/reporting/storageUsageReporting.ts

tests/ctst/common/hooks.ts

tests/ctst/steps/reporting/storageUsageReporting.ts

tests/ctst/features/reporting/StorageUsageReporting.feature

bert-e · 2026-03-10T11:05:13Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following reviewers are expecting changes from the author, or must review again:

@DarkIsDude

bert-e · 2026-03-11T09:32:50Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following reviewers are expecting changes from the author, or must review again:

@DarkIsDude

tests/ctst/common/common.ts

tests/ctst/steps/reporting/storageUsageReporting.ts

tests/ctst/common/common.ts

Issue: ZENKO-5202

.github/scripts/end2end/run-e2e-ctst.sh

tests/ctst/common/common.ts

tests/ctst/steps/reporting/storageUsageReporting.ts

tests/ctst/common/utils.ts

tests/ctst/features/reporting/StorageUsageReporting.feature

francoisferrand · 2026-03-12T09:03:05Z

tests/ctst/features/reporting/StorageUsageReporting.feature

+
+        Examples:
+          | locationName |
+          | us-east-1    |


missing tests for "AccountOwner" persona (and multiple accounts): to verify he can see only his own account only

missing tests for "StorageReporter" persona

AccountOwner: The /reporting/usage endpoint doesn't support per-account permissions — authorization is all-or-nothing (403 fails fast before any account filtering). An AccountOwner test would behave identically to StorageManager (200 with all accounts) or get 403, neither of which tests account-scoped visibility. We'd need per-account filtering in pensieve-api first.

StorageReporter: No StorageReporter Keycloak role exists in the test environment yet. On it

Actually, for StorageReporter this needs a new user, which is a bit cumbersome in the current version of Zenko. This will change in https://github.com/scality/cli-testing/pull/96 / #2341

I suggest keeping this aside for now, and adding the user + tests after that Zenko PR is merged. Will be easier. I tested it end to end on my end.

we can work incrementally, but:

we need these extra tests to complete the work; so if we don't do this yet, the topic (and EPIC) stays open for longer.... esp. since most of the work was done so that we could have non-admin users with this capability...

what is the ETA for that PR/change ? If this is a couple of hours or maybe days, why not ; but if there is no ETA we should really consider an alternative, less optimal, approach (i.e what is the actual cost of this "cumbersome-ness") and/or put more focus on that prerequisite change

in that case need to "split" the current ticket, so we have a followup (and put it in sprint)

The PR #2341 already has 2 approves, just a few remaining comments. Planned to be merged by early next week.

Ticket created at https://scality.atlassian.net/browse/ZENKO-5219 , assigned, added to sprint.

tests/ctst/steps/reporting/storageUsageReporting.ts

Replace the three scalar World keys (identityNameForScenario, identityTypeForScenario, accountNameForScenario) with a typed SavedIdentity[] array. This makes multiple identities naturally queryable and eliminates manual additionalAccountNames tracking. - Add SavedIdentity interface and getSavedIdentities/getSavedIdentity helpers - saveIdentityInformation now appends to the array - useSavedIdentity uses getSavedIdentity() (defaults to last entry) - Migrate direct getSaved reads in bucket-policies and iam-policies - Simplify "{int} additional accounts" step - Drop additional accounts After cleanup hook Issue: ZENKO-5202

Use the existing data_consumer Keycloak user (which has the DataConsumer role, not StorageManager) instead of introducing a dedicated norights user. This removes the KeycloakUsernameNoRights world parameter and reuses the existing DataConsumerUsername parameter. Issue: ZENKO-5202

Call prepareMetricsScenarios directly in the Before hook instead of going through an unnecessary indirection layer. Issue: ZENKO-5202

Add optional size parameter to putObject and uploadSetup so callers can pass the object size directly rather than routing it through the World saved-state dictionary. Issue: ZENKO-5202

Issue: ZENKO-5202

… groups Collapse permission scenarios into a single parameterized Scenario Outline with role/expectedStatus examples. Separate content tests (structure, multi-account, metrics) into their own group. Replace two hardcoded When steps with a single parameterized step using a role-to-username lookup. Issue: ZENKO-5202

Disable camelcase rule for role-to-username map keys (must match Keycloak usernames). Shorten step names and parameter names to stay within 120-char line limit. Issue: ZENKO-5202

DarkIsDude

Can you check the CI ?

The roleToWorldParam lookup resolved role names to ctst_-prefixed usernames (e.g. ctst_storage_manager) which are not valid for direct Keycloak token acquisition. Pass the role string directly instead, matching how managementAPIRequest works elsewhere. Issue: ZENKO-5202

tests/ctst/features/reporting/StorageUsageReporting.feature

francoisferrand · 2026-03-13T08:51:25Z

tests/ctst/features/reporting/StorageUsageReporting.feature

+        Examples:
+          | role             | expectedStatus |
+          | storage_manager  | 200            |
+          | data_consumer    | 403            |


what are really these "values" : the names of keycloak users? the keycloak role? the IAM role?

overal, I think this may actually be the name of the keycloak user, as defined in keycloak_config.json file ?

we should probably rename these to make it clear what this is this → storage_manager_identity, etc...

the is a fundamental inconsistency here, as we have hard-coded references here while we pass a KeycloakUsername in the Zenko World... To make/keep the tests portable, we will need to either handle some mapping in the world, or just create "our own" identities in the test (and just require a keycloak user to do this setup). Not sure how far we can go in this PR (certainly not the whole cleanup), but worth reviewing and checking we don't introduce more implicit coupling than there already is...

Addressed: the Examples values are now World parameter keys (StorageManagerUsername, DataConsumerUsername) rather than raw Keycloak usernames, and the column is labeled persona. The step resolves them via this.parameters[persona] with a fallback for local dev.

Reverted to raw Keycloak usernames (storage_manager, data_consumer). Here's why using World parameters doesn't work:

Two separate user populations exist in Keycloak:

Base users (storage_manager, data_consumer, storage_account_owner) — created by importing keycloak_config.json during cluster setup. These have the instanceIds attribute (set inline in the JSON), which the management API (pensieve) requires for token validation.

CTST users (ctst_storage_manager, ctst_data_consumer, etc.) — created by seedKeycloak.sh during test setup. These do not have instanceIds set. They're used for ARWWI (AssumeRoleWithWebIdentity) flows in S3/IAM tests, where instanceIds isn't needed.

World parameters point to the wrong population:

StorageManagerUsername → ctst_storage_manager (no instanceIds → 401 from management API)

DataConsumerUsername → ctst_data_consumer (same issue)

KeycloakUsername → storage_manager (works, but only exists for one user)

There's no World parameter for the base data_consumer username.

The base usernames have no env var source either:

storage_manager is hardcoded in keycloak_config.json line 81. OIDC_USERNAME in end2end.yaml is a separate duplicate of the same value — it's used for keycloak_user.json (admin user template) and the -norights user, not for keycloak_config.json.

data_consumer and storage_account_owner are hardcoded in keycloak_config.json with no env var at all.

So there's already duplication between end2end.yaml (OIDC_USERNAME: 'storage_manager') and keycloak_config.json ("username": "storage_manager"). Adding World parameters would just be a third copy of the same hardcoded strings with no actual decoupling.

Properly fixing this would mean either: (a) templating all usernames in keycloak_config.json through env vars, or (b) setting instanceIds on the ctst_ users too. Both are out of scope for this PR.

I think for users we need a proper refactoring, the env vars + json config + ctst thing + cli testing is a bit all over the place. Maybe in a separate PR later?

The SavedIdentity refactor changed useSavedIdentity to read from an array, but getSavedIdentity returns undefined when no identity was saved. This crashed CountItems which calls useSavedIdentity via 'the operation finished without error' without prior identity setup. Match old behavior by returning early when no identity exists. Issue: ZENKO-5202

- Rename "role" to "persona" in feature file and step definitions - Replace hardcoded Keycloak usernames with World parameter keys - Add Given step for persona setup, decoupled from When action - Split into "tries to retrieve" (no status check) and "retrieves" (asserts 200) - Add generic "Then the http response code is {int}" step in common - Remove section comments from feature file Issue: ZENKO-5202

Kubernetes events expire after 1 hour by default. In CI, kind export logs often runs 1-2 hours after the test window, by which time the relevant events have already been garbage-collected. This was a concrete gap during the Azure archive flakiness investigation (PR #2340): the failing tests ran around 06:36 but kind export logs didn't execute until ~08:22 — nearly 2 hours later. All Kubernetes events from the failure window (pod scheduling, restarts, liveness probe failures, OOM kills) had expired and were unavailable for diagnosis. Bump --event-ttl to 4h via a kubeadm ClusterConfiguration patch in the Kind cluster config, ensuring events survive for the entire duration of CI jobs. Issue: ZENKO-5217

World parameter keys (StorageManagerUsername, DataConsumerUsername) resolve to ctst_-prefixed usernames which lack the instanceIds Keycloak attribute required by the management API. Use the base Keycloak usernames directly. Issue: ZENKO-5202

scality deleted a comment from bert-e Mar 2, 2026

delthas force-pushed the improvement/ZENKO-5202/usage-reporting-route branch 5 times, most recently from 3f908f3 to eee9043 Compare March 4, 2026 21:06

delthas requested review from a team, SylvainSenechal and benzekrimaha March 4, 2026 21:08

delthas force-pushed the improvement/ZENKO-5202/usage-reporting-route branch 6 times, most recently from 42df742 to 1ef0c43 Compare March 5, 2026 18:39

delthas requested a review from DarkIsDude March 6, 2026 10:27

DarkIsDude requested changes Mar 9, 2026

View reviewed changes

scality deleted a comment from bert-e Mar 9, 2026

delthas force-pushed the improvement/ZENKO-5202/usage-reporting-route branch 2 times, most recently from 54b4acc to 4e692d3 Compare March 10, 2026 11:05

scality deleted a comment from bert-e Mar 10, 2026

francoisferrand requested a review from DarkIsDude March 11, 2026 09:30

delthas force-pushed the improvement/ZENKO-5202/usage-reporting-route branch from 4e692d3 to 0885b8e Compare March 11, 2026 09:32

scality deleted a comment from bert-e Mar 11, 2026

DarkIsDude reviewed Mar 11, 2026

View reviewed changes

Add CTST tests for storage usage reporting API

8b861f7

Issue: ZENKO-5202

delthas force-pushed the improvement/ZENKO-5202/usage-reporting-route branch from 0885b8e to 02d9543 Compare March 11, 2026 14:37

Bump pensieve-api

77e25de

Issue: ZENKO-5202

delthas force-pushed the improvement/ZENKO-5202/usage-reporting-route branch from 02d9543 to 77e25de Compare March 12, 2026 10:42

francoisferrand reviewed Mar 12, 2026

View reviewed changes

delthas added 6 commits March 12, 2026 15:09

Remove prepareStorageUsageReportingScenarios wrapper function

20fabe4

Call prepareMetricsScenarios directly in the Before hook instead of going through an unnecessary indirection layer. Issue: ZENKO-5202

Pass objectSize directly to putObject instead of saving to World

ac81793

Add optional size parameter to putObject and uploadSetup so callers can pass the object size directly rather than routing it through the World saved-state dictionary. Issue: ZENKO-5202

Put async function on same line as When/Then step registration

3407bb0

Issue: ZENKO-5202

delthas requested review from DarkIsDude and francoisferrand March 12, 2026 16:34

Fix lint errors in storage usage reporting steps

12e1e09

Disable camelcase rule for role-to-username map keys (must match Keycloak usernames). Shorten step names and parameter names to stay within 120-char line limit. Issue: ZENKO-5202

DarkIsDude requested changes Mar 12, 2026

View reviewed changes

francoisferrand reviewed Mar 13, 2026

View reviewed changes

delthas added 2 commits March 13, 2026 10:13

delthas mentioned this pull request Mar 13, 2026

Increase Kind apiserver event TTL from 1h to 4h #2351

Open

Conversation

delthas commented Mar 2, 2026

Summary

Uh oh!

bert-e commented Mar 2, 2026

Hello delthas,

Uh oh!

bert-e commented Mar 2, 2026

Waiting for approval

Uh oh!

delthas commented Mar 2, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bert-e commented Mar 10, 2026

Waiting for approval

Uh oh!

bert-e commented Mar 11, 2026

Waiting for approval

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

delthas Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

DarkIsDude left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

delthas Mar 12, 2026 •

edited

Loading