Skip to content

fix: enhance error handling in raise_if_error function to manage non-JSON responses #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Nov 19, 2025
Merged

fix: enhance error handling in raise_if_error function to manage non-JSON responses #16

merged 1 commit into from
Nov 19, 2025

Conversation

@joscha
Copy link
Member

@joscha joscha commented Nov 19, 2025 •
edited
Loading

This handles errors like:

pydantic_core._pydantic_core.ValidationError: 1 validation error for Errors
  Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value='<html>\r\n<head><title>5...n</body>\r\n</html>\r\n', input_type=str]
    For further information visit https://errors.pydantic.dev/2.12/v/json_invalid
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venvs/c570d3a20db0/lib/python3.12/site-packages/dlt/extract/utils.py", line 255, in _parallel_gen
    return next(gen)  # type: ignore[call-overload]
           ^^^^^^^^^
  File "/venvs/c570d3a20db0/lib/python3.12/site-packages/dlt_source_affinity/__init__.py", line 446, in __entities
    response = rest_client.get(
               ^^^^^^^^^^^^^^^^
  File "/venvs/c570d3a20db0/lib/python3.12/site-packages/dlt/sources/helpers/rest_client/client.py", line 182, in get
    return self.request(path, method="GET", params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venvs/c570d3a20db0/lib/python3.12/site-packages/dlt/sources/helpers/rest_client/client.py", line 179, in request
    return self._send_request(prepared_request, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venvs/c570d3a20db0/lib/python3.12/site-packages/dlt/sources/helpers/rest_client/client.py", line 155, in _send_request
    return self.session.send(prepared_request, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venvs/c570d3a20db0/lib/python3.12/site-packages/tenacity/__init__.py", line 338, in wrapped_f
    return copy(f, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^
  File "/venvs/c570d3a20db0/lib/python3.12/site-packages/tenacity/__init__.py", line 477, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venvs/c570d3a20db0/lib/python3.12/site-packages/tenacity/__init__.py", line 378, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/venvs/c570d3a20db0/lib/python3.12/site-packages/tenacity/__init__.py", line 400, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/venvs/c570d3a20db0/lib/python3.12/site-packages/tenacity/__init__.py", line 480, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/venvs/c570d3a20db0/lib/python3.12/site-packages/dlt/sources/helpers/requests/session.py", line 61, in send
    return super().send(request, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venvs/c570d3a20db0/lib/python3.12/site-packages/requests/sessions.py", line 710, in send
    r = dispatch_hook("response", hooks, r, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venvs/c570d3a20db0/lib/python3.12/site-packages/requests/hooks.py", line 30, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venvs/c570d3a20db0/lib/python3.12/site-packages/dlt_source_affinity/rest_client.py", line 60, in raise_if_error
    error = error_adapter.validate_json(response.text)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venvs/c570d3a20db0/lib/python3.12/site-packages/pydantic/type_adapter.py", line 492, in validate_json
    return self.validator.validate_json(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

where processing stops because the 5XX error returns a non-JSON response.

(Crucial part here: input_value='<html>\r\n<head><title>5...n</body>\r\n</html>\r\n' and here: in raise_if_error error = error_adapter.validate_json(response.text)

With the changes in this pull request we don't exit due to a validation error, but return back into raise_for_status which then also enables retries.

@joscha joscha merged commit c94ae78 into main Nov 19, 2025
8 checks passed
@joscha joscha deleted the joscha/non-json-errors branch November 19, 2025 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@joscha