Skip to content

Optimize SWE-Bench image builds: add local Docker check before remote registry #481

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
juanmichelini wants to merge 2 commits into from
Closed

Optimize SWE-Bench image builds: add local Docker check before remote registry #481

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7e43c84
Fix SWE-Bench batch builds hanging by restoring local image check
openhands-agent Mar 4, 2026
3273fa0
Update: reframe as optimization, remove incorrect root cause claims
openhands-agent Mar 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions benchmarks/utils/build_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -303,9 +303,12 @@ def build_image(
sdk_version=sdk_version,
)
for t in opts.all_tags:
# Check if image exists or not
# Check local Docker first (fast ~100ms), then remote registry (slow HTTP)
if local_image_exists(t):
logger.info("Image %s already exists locally. Skipping build.", t)
return BuildOutput(base_image=base_image, tags=[t], error=None)
Copy link
Collaborator

@all-hands-bot all-hands-bot Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Testing Gap: While this is a solid performance fix, add a unit test that verifies the call order: mock both local_image_exists() and remote_image_exists(), make the local check return True, and assert the remote check was never called. This would have caught the regression in commit 92efb47 and prevents future breakage.

Example test structure:

def test_build_image_checks_local_before_remote(mocker):
    mock_local = mocker.patch("...local_image_exists", return_value=True)
    mock_remote = mocker.patch("...remote_image_exists", return_value=True)
    
    build_image(...)
    
    mock_local.assert_called_once()
    mock_remote.assert_not_called()  # Should never hit remote if local exists

if remote_image_exists(t):
logger.info("Image %s already exists. Skipping build.", t)
logger.info("Image %s already exists in registry. Skipping build.", t)
return BuildOutput(base_image=base_image, tags=[t], error=None)
tags = build(opts)
return BuildOutput(base_image=base_image, tags=tags, error=None)
Expand Down
Loading