Skip to content

Remove pin for s3fs < 2026.2.0 #237

Description

@martyngigg

The extract and load tests fail when upgrading s3fs past 2026.1.0, e.g

> cd infra/local
> docker compose up -d
> cd warehouses/accelerator_ops/extract_load/opralogweb/
> pytest --catalog-type=rest tests/
File "/home/ubuntu/code/analytics-data-platform/.venv/lib/python3.13/site-packages/s3fs/core.py", line 2587, in commit
    write_result = self._call_s3("put_object", **kw, **match)
  File "/home/ubuntu/code/analytics-data-platform/.venv/lib/python3.13/site-packages/s3fs/core.py", line 2421, in _call_s3
    return self.fs.call_s3(method, self.s3_additional_kwargs, *kwarglist, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/code/analytics-data-platform/.venv/lib/python3.13/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/ubuntu/code/analytics-data-platform/.venv/lib/python3.13/site-packages/fsspec/asyn.py", line 103, in sync
    raise return_result
  File "/home/ubuntu/code/analytics-data-platform/.venv/lib/python3.13/site-packages/fsspec/asyn.py", line 56, in _runner
    result[0] = await coro
                ^^^^^^^^^^
  File "/home/ubuntu/code/analytics-data-platform/.venv/lib/python3.13/site-packages/s3fs/core.py", line 451, in _call_s3
    return await _error_wrapper(
           ^^^^^^^^^^^^^^^^^^^^^
        method, kwargs=additional_kwargs, retries=self.retries
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/ubuntu/code/analytics-data-platform/.venv/lib/python3.13/site-packages/s3fs/core.py", line 214, in _error_wrapper
    raise err
PermissionError: Access Denied.

The issues seems to have arisen when fsspec/s3fs#1002 was merged. Reversing the changes locally recovers the original behaviour.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    technical debtTasks to deliver a more robust solution to a quick fix required for speed of implementation

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions