OCSDV-453 (PR-E): carry per-parameter filters into demand execution data syncs - #41
Merged
Merged
Conversation
This was referenced Aug 12, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Propagates per-parameter filters through demand execution data syncs and prevents oversized EFS mount paths.
Changes:
- Preserves input/output filters and forwards them to sync requests.
- Adds explicit input/output deletion semantics and tests.
- Introduces compact, validated EFS volume naming.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
uv.lock |
Updates the pinned core dependency commit. |
src/.../context_manager.py |
Propagates filters and integrates EFS naming checks. |
src/.../naming.py |
Adds compact volume naming and path-budget validation. |
test/.../test_context_manager.py |
Tests filter propagation and volume naming. |
test/.../test_naming.py |
Tests naming and budget helpers. |
test/.../test_scaffolding.py |
Updates expected generated volume names. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| temporary_request_payload_path=temporary_request_payload_path, | ||
| size_only=self.configuration.output_data_sync_configuration.size_only, | ||
| force=self.configuration.output_data_sync_configuration.force, | ||
| filter_config=param.filter_config, |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #41 +/- ##
==========================================
- Coverage 90.92% 90.81% -0.11%
==========================================
Files 26 27 +1
Lines 1465 1491 +26
Branches 141 145 +4
==========================================
+ Hits 1332 1354 +22
- Misses 96 98 +2
- Partials 37 39 +2
🚀 New features to boost your workflow:
|
rpmcginty
force-pushed
the
feature/OCSDV-453-demand-execution-sync-filters
branch
from
August 20, 2026 03:54
c635ed4 to
ea41d29
Compare
| new_resolvable = Resolvable(local=local.as_posix(), remote=param.remote_value) | ||
| # Carry the param's include/exclude forward. Only the local path is being rewritten | ||
| # here, so rebuilding the Resolvable without the filters would silently drop them | ||
| # before they ever reach the pre-execution data sync requests. |
There was a problem hiding this comment.
Nit: I don't find this comment necessary, the docstrings above already stated the reason
Comment on lines
+629
to
+631
| # Carry the param's include/exclude forward, for the same reason as the input rewrite: | ||
| # dropping them here would leave post_execution_data_sync_requests with nothing to | ||
| # apply, since it reads the filters back off these params. |
There was a problem hiding this comment.
Nit: Same thing here, docstring immediate above already spelled out the reasoning
| @@ -799,11 +864,21 @@ def generate_batch_job_builder( # noqa: C901 | |||
| vol_configurations.append(BatchEFSConfiguration(tmp_mount_point, read_only=False)) | |||
| logger.info("Constructing BatchJobBuilder instance...") | |||
| assert demand_execution.execution_platform.aws_batch is not None | |||
There was a problem hiding this comment.
Nit: this isn't in the diff hence a nit, assert here 1) poses a risk of being bypassed when python runs in -o, 2) returns an AssertionError. Suggest we use if conditions on the implementation side and raise ValueError or TypeError
| ValueError: If the longest volume name combined with the job definition name | ||
| exceeds the budget. | ||
| """ | ||
| if not volume_names: |
There was a problem hiding this comment.
vol_configurations seems to always contain scratch_mount_point and shared_mount_point. volume_names is downstream of that, so it doesn't seem like volume_names can ever be None.
This if not volume_names: condition looks to be unreachable or should be unreachable, suggest we remove it or raise an error instead of returning
Collaborator
Author
There was a problem hiding this comment.
No volumes means no EFS mount, which is a valid scenario (albeit not one that the demand execution expects). I would push back on the claim that this should be unreachable.
Comment on lines
+324
to
+327
| # The volume name leads with the mount path basename and ends in a hash over the | ||
| # file system / access point / mount path. The hash cannot be hardcoded here because | ||
| # moto generates the ids, so assert the shape and the invariants instead. The name | ||
| # must stay short: it ends up inside an efs-utils TLS state path that openssl caps |
There was a problem hiding this comment.
Nit: I think this is an unnecessary comment in test, this was well-explained on the implementation side
schristinelin
left a comment
schristinelin
left a comment
There was a problem hiding this comment.
Overall looks good, minor question about some conditions. Can we prompt AI to clean up the in-line comments so they don't repeat what's already stated in the docstrings/module-level comments? I'm personally finding them to be very distracting when trying to read the codes, and a bit concerned about verbosity as the repo grows and more comments get added
rpmcginty
force-pushed
the
feature/OCSDV-453-demand-execution-sync-filters
branch
2 times, most recently
from
August 20, 2026 20:39
ab444a0 to
891a6ae
Compare
schristinelin
approved these changes
Aug 20, 2026
Base automatically changed from
feature/OCSDV-452-prepare-batch-filters
to
main
August 21, 2026 03:27
Filters set on a demand execution param previously died before reaching the data sync layer: both parameter-rewrite functions rebuilt their Resolvable / Uploadable from scratch, keeping only local and remote. - update_demand_execution_parameter_inputs: preserve include/exclude rather than rebuilding Resolvable(local=, remote=) from scratch. Only the local path is being rewritten. - update_demand_execution_parameter_outputs: same fix. Not in the original scope, but post_execution_data_sync_requests reads filters back off these params, so wiring the output path is a no-op without it. - pre_execution_data_sync_requests: pass filter_config, and set delete=False. The input destination is a cross-execution cache, and `delete` is the caller's gate on the destructive filters + mirroring interaction that OCSDV-452 deliberately left open. - post_execution_data_sync_requests: pass filter_config through. Outputs keep delete at its default, so the destination is mirrored to the filtered subset; documented as a Warning rather than left implicit. Documents on the output path that output filtering means "do not upload", NOT "keep locally": retain_source_data=False, sync_local_to_s3 ends with remove_path, and cleanup_working_dir defaults to True, so excluded outputs leave EFS regardless. Also notes on the input path that filters bound what is transferred, not what is present: the shared-cache key is sha256(remote_value) and does not vary with filters, a known deferred gap from OCSDV-452. Tests cover propagation into both pre- and post-execution requests, both isolate_inputs branches, unfiltered control cases, and pin both delete decisions explicitly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Demand execution batch jobs failed before the container ran:
CannotStartContainerError: ... Failed to create self-signed client-side
certificate ... File name too long
Not a NAME_MAX/PATH_MAX limit. The volume name flows through a chain of
generated names into an OpenSSL buffer:
batch job definition name -> ECS task family
-> docker volume name ecs-<family>-<rev>-<volume>-<20 hex>
-> efs-utils TLS state dir
-> openssl CA database /var/run/efs/<state dir>/database/index.txt
openssl guards that path with a hard-coded 256 byte stack buffer (BSIZE in
apps/lib/apps.c), capping the database path at 246 chars and, back up the
chain, capping family + revision + volume name at 139. A 50 char volume name
("dev-de-core-opt-fsap-0acb9f234e1b57786-scratch-vol") alongside a 92 char job
definition name came to 143 -- over by 4.
Adds handlers/demand/naming.py, holding the Batch/EFS specific arithmetic:
* build_efs_volume_name(...) -- leads with the mount path basename
(scratch/shared/tmp) and takes uniqueness from a hash over the file system,
access point and mount path rather than spelling them out. 50 -> 16 chars.
* check_ecs_volume_component_budget(...) -- raises while the job definition is
being built rather than minutes later at container start, where the error
names nothing responsible.
The generic string shortening primitive it builds on, condense_str, lives in
aibs-informatics-core (utils.tools.strtools) since nothing about it is
Batch or EFS specific.
Capping execution_type would also have cleared the budget, but pushes an
OpenSSL buffer size onto callers; a descriptive execution type should not break
EFS mounting. A 40 char execution_type now fits at 129/139 where a 20 char one
previously failed at 143.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comment cleanup (@schristinelin, @pbishwakarma) -- the recurring note was that inline comments restate what the code or the docstring above already says: - context_manager.py: condense the volume-name comment from five lines to two; drop the two "carry the param's include/exclude forward" comments, which repeat the docstrings directly above them. - test_context_manager.py: cut the volume-name comment to the one fact that isn't obvious (moto generates the ids, so the hash can't be a literal). It also pointed at aibs_informatics_aws_lambda.common.naming, which does not exist (@copilot). - naming.py: move the budget derivation out of the ECS_VOLUME_COMPONENT_BUDGET comment and into the module docstring, where the chain it belongs to is already described. Correctness: - naming.py: restrict the volume-name sanitizer to ASCII (@copilot). str.isalnum is Unicode-aware, so a basename like "données" passed sanitization and produced a name ECS rejects. Regression test added. - context_manager.py: replace the aws_batch `assert` with an explicit ValueError (@schristinelin) -- asserts are stripped under `python -O` and AssertionError is the wrong type for a rejected input. Also drops the Sphinx :func:/:data: roles from naming.py docstrings. Docs here render with mkdocstrings (google style), which does not process reST roles and emits the prefix literally -- same issue @njmei raised in aws-utils. Not changed: the `if not volume_names: return` guard in check_ecs_volume_component_budget (@schristinelin). It is unreachable from the current call site, but it is not merely defensive -- no volumes means no EFS mount, so nothing derives an efs-utils path and the budget does not apply even to an over-long job definition name. Removing it broke test__no_volumes_is_not_an_error, which pins exactly that. Added a comment saying so instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rpmcginty
force-pushed
the
feature/OCSDV-453-demand-execution-sync-filters
branch
from
August 21, 2026 03:27
891a6ae to
d78b8c7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OCSDV-453 — PR-E, the demand-execution half (Release R3).
Problem
A filter set on a demand execution param died before reaching the sync layer. Both parameter-rewrite functions rebuilt their resolvable from scratch, keeping only
localandremote— so anything else was dropped at the rewrite hop.Changes
All in
handlers/demand/context_manager.py:update_demand_execution_parameter_inputs— preserveinclude/excludeinstead of rebuildingResolvable(local=, remote=)from scratch. Only the local path is being rewritten.pre_execution_data_sync_requests— passfilter_config, and setdelete=False. The input destination is a cross-execution cache anddeleteis the caller's gate on the destructive filters + mirroring interaction OCSDV-452 deliberately left open; this is the caller opting out.post_execution_data_sync_requests— passfilter_configthrough.retain_source_data=False,sync_local_to_s3ends withremove_path(source_path), andcleanup_working_dirdefaults toTrue, so excluded outputs leave EFS regardless.Plus one change outside the stated scope, called out for review:
update_demand_execution_parameter_outputshad the identical drop-on-rebuild bug. Item 3 is a no-op without fixing it — the output params reachingpost_execution_data_sync_requestsstill carried no filters, sofilter_configwasNoneend to end. Caught by a failing test, not by inspection.Known gap, documented not fixed
The shared-input cache key is
sha256(remote_value)and does not vary with filters, so two executions filtering the same remote differently share one EFS directory — and withdelete=Falsethe contents accumulate as a union. This is the deferred gap OCSDV-452 records; exposure is limited becauseisolate_inputsdefaults toTrue. I corrected an inaccurate docstring line that would have told readers filters guarantee only the filtered subset is present at the input path, and noted the real behavior instead.Testing
make format && make lint && make test— 168 passed, ruff and mypy clean.New coverage in
demand/test_context_manager.py: propagation into both pre- and post-execution requests, preservation across bothisolate_inputsbranches, unfiltered control cases on both paths,delete=Falseasserted on filtered and unfiltered input requests, anddelete=Trueasserted on filtered output requests.Mutation-checked: removing
delete=Falsefrom inputs fails 3 tests; flipping outputs todelete=Falsefails 1; reverting either filter-preservation fix fails the propagation tests.Not done here
The ticket's DoD also lists a
DemandExecutionround-trip test and theget_execution_hashstability regression. Those guard PR-B's serializer and hash changes and belong in the core repo — they are not exercisable from this repo, wheresanitize_serialized_paramsis an upstream dependency. Worth confirming they landed in PR-B before R1 ships.🤖 Generated with Claude Code
End-to-end verification (dev)
This PR contributes the propagation of per-parameter filters into the pre- and post-execution data syncs, plus the EFS volume-name fix that unblocked these runs, to the behavior verified below.
Four DemandExecutions were run against the deployed
dev-deenvironment (account051791135335,us-west-2) using a shared 12-object / 541,610-byte S3 fixture. All four Step Functions executions reachedSUCCEEDED, and in every case the set of objects that landed in S3 matches the declared include/exclude filters exactly — no missing matches and no leaked non-matches.Because a single demand execution exercises the whole chain — the data-sync request/filter model in
aibs-informatics-core, the S3 transfer and filtering implementation inaibs-informatics-aws-utils, and the Batch/EFS execution wiring inaibs-informatics-aws-lambda— this same evidence block is attached to every PR in the set rather than split across them.Related PRs
These five PRs are one change set and are intended to merge in release order (R1 first, then R2, then R3), since each release depends on the one before it.
Fixture
s3://dev-de-core-us-west-2-051791135335/test_data/ocsdv-452-filters/input/— 12 objects, 541,610 bytes total (aws s3 ls --recursive --summarizereportsTotal Objects: 12 / Total Size: 541610). The fixture now lives in the environment's own bucket, having been moved off an orphaned bucket that no CloudFormation stack owns, so inputs and outputs now share a single bucket:README.mdmetadata.jsonsampleA/aligned.bamsampleA/aligned.bam.baisampleA/qc_summary.txtsampleA/reads_R1.fastqsampleA/reads_R2.fastqsampleB/aligned.bamsampleB/aligned.bam.baisampleB/qc_summary.txtsampleB/reads_R1.fastqsampleB/reads_R2.fastqEach case runs the same container command: list
${INPUT_DATA},cp -R ${INPUT_DATA}/. ${OUTPUT_DATA}/, list${OUTPUT_DATA}. The container itself does no filtering, so any difference between the fixture and the uploaded set is attributable to the data-sync filters.Results
Counts are objects. "In" is the fixture, "localized" is what survived the input filter and was present under
${INPUT_DATA}in the container, "uploaded" is what was observed in S3 under the case prefix.tc1-input-exclude-bamexclude: [".*\\.bam", ".*\\.bam\\.bai"]tc2-input-include-fastqinclude: [".*\\.fastq"]tc3-output-exclude-fastqexclude: [".*\\.fastq"]tc4-both-ends-filteredinclude: ["sampleA/.*"]include: [".*\\.txt"]Across the four cases: 21 objects, 551,275 bytes written under
s3://dev-de-core-us-west-2-051791135335/test_results/ocsdv-452-filters/.Two independent confirmations worth noting:
.fastqfiles and nothing else. Uploaded file sizes are byte-identical to their sources, so nothing was truncated in transit.jobId 7f83fd65-6e9e-4817-a48d-e6e796b1cedf) prints all 12 files under--- localized input (post input-filter) ---and all 12 again under--- staged for upload (pre output-filter) ---, yet only 8 objects reached S3. That rules out the output filter being misapplied at localization time, which an object count alone could not.Note on attribution for tc1 and tc2: because the container copies
${INPUT_DATA}wholesale into${OUTPUT_DATA}, the final S3 state for an input-side filter is indistinguishable from an equivalent output-side filter. For those two cases the input-side attribution rests on the payload (filters declared oninput_data,output_databare) plus the container'sfindoutput, not on the S3 listing alone. tc3 and tc4 are unambiguous.Directory structure is preserved end to end in all cases — the
sampleA/andsampleB/prefixes survive the round trip, so patterns are matched against the same relative path that is used to construct the destination key.tc1-input-exclude-bam — input-side exclude, 8 objects uploaded (9,202 bytes)
Uploaded (relative to
s3://dev-de-core-us-west-2-051791135335/test_results/ocsdv-452-filters/tc1-input-exclude-bam/;aws s3 ls --recursive --summarizereportsTotal Objects: 8 / Total Size: 9202):Correctly not uploaded — these four match one of the two input-side exclude patterns (
.*\.bam,.*\.bam\.bai), so they were never localized and therefore never staged:sampleA/aligned.bam.*\.bamsampleA/aligned.bam.bai.*\.bam\.baisampleB/aligned.bam.*\.bamsampleB/aligned.bam.bai.*\.bam\.baiEvery excluded object matches a declared pattern, and every object matching a declared pattern is excluded. Step Functions execution
d247218f-34e6-47cc-ad4b-a175180b2e45onarn:aws:states:us-west-2:051791135335:stateMachine:dev-de-demand-execution,SUCCEEDED, 2026-08-12 14:46:07 → 14:51:02 (-07:00); object timestamps (14:50:37) fall inside that window.{ "execution_type": "ocsdv452", "execution_id": "ocsdv452-tc1-input-exclude-bam", "execution_image": "public.ecr.aws/ubuntu/ubuntu:22.04", "execution_parameters": { "command": [ "set", "-eu", "&&", "echo", "'=== tc1-input-exclude-bam ==='", "&&", "echo", "'--- localized input (post input-filter) ---'", "&&", "find", "${INPUT_DATA}", "-type", "f", "|", "sort", "&&", "mkdir", "-p", "${OUTPUT_DATA}", "&&", "cp", "-R", "${INPUT_DATA}/.", "${OUTPUT_DATA}/", "&&", "echo", "'--- staged for upload (pre output-filter) ---'", "&&", "find", "${OUTPUT_DATA}", "-type", "f", "|", "sort" ], "params": { "input_data": { "remote": "s3://dev-de-core-us-west-2-051791135335/test_data/ocsdv-452-filters/input/", "local": "input_data", "exclude": [ ".*\\.bam", ".*\\.bam\\.bai" ] }, "output_data": { "local": "output_data", "remote": "s3://dev-de-core-us-west-2-051791135335/test_results/ocsdv-452-filters/tc1-input-exclude-bam/" } }, "inputs": [ "input_data" ], "outputs": [ "output_data" ], "output_s3_prefix": "s3://dev-de-core-us-west-2-051791135335/test_results/ocsdv-452-filters/tc1-input-exclude-bam/" }, "execution_platform": { "aws_batch": { "job_queue_name": "dev-de-demand-on-demand-job-queue" } }, "resource_requirements": { "vcpus": 1, "memory": 2048 } }tc2-input-include-fastq — input-side include, 4 objects uploaded (7,932 bytes)
Uploaded (
Total Objects: 4 / Total Size: 7932; 4 × 1,983 = 7,932, matching source sizes exactly):Correctly not uploaded — the input side declares a single include pattern
.*\.fastqand no exclude, so any key that does not match is not localized. None of these eight keys end in.fastq:All four fixture keys that do match
.*\.fastqare present, so membership is correct in both directions. Step Functions executionarn:aws:states:us-west-2:051791135335:execution:dev-de-demand-execution:ocsdv452-tc2-input-include-fastq,SUCCEEDED, 2026-08-12 15:37:27 → 15:42:09 (-07:00).{ "execution_type": "ocsdv452", "execution_id": "ocsdv452-tc2-input-include-fastq", "execution_image": "public.ecr.aws/ubuntu/ubuntu:22.04", "execution_parameters": { "command": [ "set", "-eu", "&&", "echo", "'=== tc2-input-include-fastq ==='", "&&", "echo", "'--- localized input (post input-filter) ---'", "&&", "find", "${INPUT_DATA}", "-type", "f", "|", "sort", "&&", "mkdir", "-p", "${OUTPUT_DATA}", "&&", "cp", "-R", "${INPUT_DATA}/.", "${OUTPUT_DATA}/", "&&", "echo", "'--- staged for upload (pre output-filter) ---'", "&&", "find", "${OUTPUT_DATA}", "-type", "f", "|", "sort" ], "params": { "input_data": { "remote": "s3://dev-de-core-us-west-2-051791135335/test_data/ocsdv-452-filters/input/", "local": "input_data", "include": [ ".*\\.fastq" ] }, "output_data": { "local": "output_data", "remote": "s3://dev-de-core-us-west-2-051791135335/test_results/ocsdv-452-filters/tc2-input-include-fastq/" } }, "inputs": [ "input_data" ], "outputs": [ "output_data" ], "output_s3_prefix": "s3://dev-de-core-us-west-2-051791135335/test_results/ocsdv-452-filters/tc2-input-include-fastq/" }, "execution_platform": { "aws_batch": { "job_queue_name": "dev-de-demand-on-demand-job-queue" } }, "resource_requirements": { "vcpus": 1, "memory": 2048 } }tc3-output-exclude-fastq — output-side exclude, 8 objects uploaded (533,678 bytes)
Uploaded (
Total Objects: 8 / Total Size: 533678):Correctly not uploaded — the input side is unfiltered, so all 12 objects were localized and all 12 were staged under
${OUTPUT_DATA}. These four were dropped at upload because they match the output-side exclude.*\.fastq:This is the case that distinguishes output-side filtering from input-side filtering by observation rather than by declaration: the container log lists all 12 files at both checkpoints, yet only 8 objects reached S3. Step Functions execution
ocsdv452-tc3-output-exclude-fastq,SUCCEEDED, 2026-08-12 15:38:21 → 15:41:58 (-07:00); Batch job7f83fd65-6e9e-4817-a48d-e6e796b1cedfondev-de-demand-on-demand-job-queue,SUCCEEDED.{ "execution_type": "ocsdv452", "execution_id": "ocsdv452-tc3-output-exclude-fastq", "execution_image": "public.ecr.aws/ubuntu/ubuntu:22.04", "execution_parameters": { "command": [ "set", "-eu", "&&", "echo", "'=== tc3-output-exclude-fastq ==='", "&&", "echo", "'--- localized input (post input-filter) ---'", "&&", "find", "${INPUT_DATA}", "-type", "f", "|", "sort", "&&", "mkdir", "-p", "${OUTPUT_DATA}", "&&", "cp", "-R", "${INPUT_DATA}/.", "${OUTPUT_DATA}/", "&&", "echo", "'--- staged for upload (pre output-filter) ---'", "&&", "find", "${OUTPUT_DATA}", "-type", "f", "|", "sort" ], "params": { "input_data": { "remote": "s3://dev-de-core-us-west-2-051791135335/test_data/ocsdv-452-filters/input/", "local": "input_data" }, "output_data": { "local": "output_data", "remote": "s3://dev-de-core-us-west-2-051791135335/test_results/ocsdv-452-filters/tc3-output-exclude-fastq/", "exclude": [ ".*\\.fastq" ] } }, "inputs": [ "input_data" ], "outputs": [ "output_data" ], "output_s3_prefix": "s3://dev-de-core-us-west-2-051791135335/test_results/ocsdv-452-filters/tc3-output-exclude-fastq/" }, "execution_platform": { "aws_batch": { "job_queue_name": "dev-de-demand-on-demand-job-queue" } }, "resource_requirements": { "vcpus": 1, "memory": 2048 } }tc4-both-ends-filtered — include on both ends, 1 object uploaded (463 bytes)
Uploaded (verbatim from
aws s3 ls ... --recursive):Correctly not uploaded — two stages, in order:
sampleA/.*admits only the 5 objects undersampleA/. Dropped at localization (7 objects):README.mdandmetadata.json(top level, nosampleA/prefix), and all 5sampleB/*objects..*\.txtadmits onlyqc_summary.txtof the 5 that were staged. Dropped at upload (4 objects):sampleA/aligned.bam,sampleA/aligned.bam.bai,sampleA/reads_R1.fastq,sampleA/reads_R2.fastq.Each of the 11 absent files is absent for a reason one of the two filters calls for, and the single present file is the only one both filters admit. The
sampleA/path component survives into the destination key, confirming the include pattern is matched against the same relative path that is preserved on upload. Step Functions executionarn:aws:states:us-west-2:051791135335:execution:dev-de-demand-execution:ocsdv452-tc4-both-ends-filtered,SUCCEEDED, 2026-08-12 15:38:39 → 15:42:23 (-07:00); the object'sLastModifiedfalls inside that window.{ "execution_type": "ocsdv452", "execution_id": "ocsdv452-tc4-both-ends-filtered", "execution_image": "public.ecr.aws/ubuntu/ubuntu:22.04", "execution_parameters": { "command": [ "set", "-eu", "&&", "echo", "'=== tc4-both-ends-filtered ==='", "&&", "echo", "'--- localized input (post input-filter) ---'", "&&", "find", "${INPUT_DATA}", "-type", "f", "|", "sort", "&&", "mkdir", "-p", "${OUTPUT_DATA}", "&&", "cp", "-R", "${INPUT_DATA}/.", "${OUTPUT_DATA}/", "&&", "echo", "'--- staged for upload (pre output-filter) ---'", "&&", "find", "${OUTPUT_DATA}", "-type", "f", "|", "sort" ], "params": { "input_data": { "remote": "s3://dev-de-core-us-west-2-051791135335/test_data/ocsdv-452-filters/input/", "local": "input_data", "include": [ "sampleA/.*" ] }, "output_data": { "local": "output_data", "remote": "s3://dev-de-core-us-west-2-051791135335/test_results/ocsdv-452-filters/tc4-both-ends-filtered/", "include": [ ".*\\.txt" ] } }, "inputs": [ "input_data" ], "outputs": [ "output_data" ], "output_s3_prefix": "s3://dev-de-core-us-west-2-051791135335/test_results/ocsdv-452-filters/tc4-both-ends-filtered/" }, "execution_platform": { "aws_batch": { "job_queue_name": "dev-de-demand-on-demand-job-queue" } }, "resource_requirements": { "vcpus": 1, "memory": 2048 } }Note on the first attempt
The first attempt at these four runs failed before the container started, with an
efs-utilserror reportingFile name too long. The cause was the generated ECS volume name: it made the mount helper's CA database path exceed OpenSSL's 256-byteBSIZEguard by 4 characters. It was worked around for the run above by shorteningexecution_typetoocsdv452. PR-E addresses it properly by condensing the generated volume name from 50 characters to 16, which puts the path back under the limit without relying on short caller-supplied names.Scope
This is a dev-environment functional check of four small cases against a 12-object fixture. It demonstrates that the filter semantics behave as declared on both the input and output sides of a real demand execution; it is not a performance or scale test, and it does not cover large transfers, multipart uploads, or concurrent executions. Verification was read-only (
s3 ls,s3api list-objects-v2,sts get-caller-identity,stepfunctions list/describe-execution,batch list/describe-jobs,logs get-log-events).Addendum: container logs confirm the input filter runs on the download, not just the upload
The S3 result alone cannot fully separate input-side from output-side filtering for
tc1andtc2, because the container copies its input directory wholesale into the output directory — an unfiltered download plus an equivalent output-side exclude would produce the same uploaded set.tc3andtc4disambiguate by construction (tc3uploads the.bamfiles thattc1drops, from the same fixture), but it is worth closing directly.The container prints its localized input before copying. From
/aws/batch/job, streamdev-de-ocsdv452-43c8f938.../bc357a293a09432681668be032cb1cb3(tc1-input-exclude-bam):Eight files, not twelve. The four
.bam/.bam.baiobjects never reached the container's EFSworking directory — the filter was applied during the download, so the 524 KB of alignment data
was never transferred. That is the behavior the feature exists to provide, and it is what
distinguishes filtering from post-hoc pruning.
Only this one stream was still retained in CloudWatch when the logs were pulled, so this direct
confirmation covers
tc1; the other three cases rest on the S3 results and their Step Functionsexecutions above.