chore(internal): simplify http snapshots #1092

karpetrosyan · 2025-12-03T18:09:31Z

Before:

    @pytest.mark.respx(base_url=base_url)
    def test_basic_call_sync(self, client: Anthropic, respx_mock: MockRouter, monkeypatch: pytest.MonkeyPatch) -> None:
        @beta_tool
        def get_weather(location: str, units: Literal["c", "f"]) -> BetaFunctionToolResultType:
            """Lookup the weather for a given city in either celsius or fahrenheit

            Args:
                location: The city and state, e.g. San Francisco, CA
                units: Unit for the output, either 'c' for celsius or 'f' for fahrenheit
            Returns:
                A dictionary containing the location, temperature, and weather condition.
            """
            return json.dumps(_get_weather(location, units))

        message = make_snapshot_request(
            lambda c: c.beta.messages.tool_runner(
                max_tokens=1024,
                model="claude-haiku-4-5",
                tools=[get_weather],
                messages=[{"role": "user", "content": "What is the weather in SF?"}],
            ).until_done(),
            content_snapshot=snapshots["basic"]["responses"],
            path="/v1/messages",
            mock_client=client,
            respx_mock=respx_mock,
        )

After:

    @pytest.mark.parametrize(
        "http_snapshot",
        [cast(Any, external("uuid:ad005e89-f72b-4f0e-941a-9ec3473c1bc8.json"))],
    )
    def test_basic_call_sync(self, snapshot_client: Anthropic) -> None:
        @beta_tool
        def get_weather(location: str, units: Literal["c", "f"]) -> BetaFunctionToolResultType:
            """Lookup the weather for a given city in either celsius or fahrenheit

            Args:
                location: The city and state, e.g. San Francisco, CA
                units: Unit for the output, either 'c' for celsius or 'f' for fahrenheit
            Returns:
                A dictionary containing the location, temperature, and weather condition.
            """
            return json.dumps(_get_weather(location, units))

        message = snapshot_client.beta.messages.tool_runner(
            max_tokens=1024,
            model="claude-haiku-4-5",
            tools=[get_weather],
            messages=[{"role": "user", "content": "What is the weather in SF?"}],
        ).until_done()

karpetrosyan · 2025-12-05T15:15:24Z

.../__inline_snapshot__/test_runners/TestSyncRunTools/ff848716-2309-477f-8e90-39877809aaad.json

+        "x-robots-tag": "none",
+        "server": "cloudflare"
+      },
+      "body": "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"model\":\"claude-haiku-4-5-20251001\",\"id\":\"msg_01SAGfXDtnWyD3j4h5Wz3kky\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"stop_reason\":null,\"stop_sequence\":null,\"usage\":{\"input_tokens\":656,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":26,\"service_tier\":\"standard\"}}       }\n\nevent: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"tool_use\",\"id\":\"toolu_01LZ7WbFLVzFd16Ur9RHXeGz\",\"name\":\"get_weather\",\"input\":{}}   }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"\"}}\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"{\\\"loca\"}             }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"tion\\\": \\\"Sa\"}         }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"n Fran\"}          }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"cisco, CA\\\"\"}        }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\", \\\"units\\\"\"}            }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\": \\\"f\\\"}\"}              }\n\nevent: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0 }\n\nevent: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"tool_use\",\"stop_sequence\":null},\"usage\":{\"input_tokens\":656,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"output_tokens\":74}              }\n\nevent: message_stop\ndata: {\"type\":\"message_stop\"        }\n\n"


I'll add a serializer for text/event-stream content-type, so it'll be pretty printed too

You planning on doing that in a follow-on? or in this PR too

I'll do it in a different one

RobertCraigie · 2025-12-09T12:45:21Z

tests/lib/_parse/test_beta_messages.py

+        response = await async_stream_parse(async_snapshot_client)

-        assert response.content[0].text == snapshot("[12345,67890]")
+        assert response.content[0].text == snapshot("[12345,67890]")  # type: ignore


q: why do we need the type ignore now?

Ah, that's for accessing the text property, since it might be missing. Idk what's better an extra line to check for it, or just type: ignore. pytest would anyway fail with the good message if there would not be .text 🤷‍♂️

RobertCraigie · 2025-12-09T12:46:47Z

.../tools/__inline_snapshot__/test_runners/__module__/c4061e82-ea59-4756-9549-71a35da24499.json

+  {
+    "response": {
+      "status_code": 200,
+      "headers": {


q: can / should we filter out a bunch of these headers? some of them will frequently update and aren't helpful, e.g. date

That's a good point. I can add a whitelist/blacklist for headers, or an option to exclude them entirely. I think in the SDKs, we're more interested in the request headers, not the response headers

Yeah can you update the default header blacklist to include the following?

date

request-id

anthropic-organization-id

x-envoy-upstream-service-time

cf-ray

RobertCraigie · 2025-12-09T12:47:32Z

.../tools/__inline_snapshot__/test_runners/__module__/c4061e82-ea59-4756-9549-71a35da24499.json

@@ -0,0 +1,93 @@
+[
+  {


q: can / should we include the request here as well?

Hmmm. I initially disabled it by default for more compact snapshots. Since we won't need request snapshots in every test, maybe it makes more sense to enable it per-test when needed

ooc how do you enable it for a test? I think we should probably include the requests for the tool runner tests?

I do...

@pytest.mark.parametrize( "http_snapshot, http_snapshot_serializer_options", [ ( inline_snapshot.external("uuid:my-test-snapshot.json"), SnapshotSerializerOptions( exclude_request_headers=["X-API-Key"], include_request=True, # Include request details in snapshot ), ), ], )

@RobertCraigie Okay, I’ve updated it so that all snapshots now include the request, and it doesn’t seem too messy.

RobertCraigie · 2025-12-09T12:48:00Z

.../__inline_snapshot__/test_runners/TestSyncRunTools/ff848716-2309-477f-8e90-39877809aaad.json

+        "x-robots-tag": "none",
+        "server": "cloudflare"
+      },
+      "body": "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"model\":\"claude-haiku-4-5-20251001\",\"id\":\"msg_01SAGfXDtnWyD3j4h5Wz3kky\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"stop_reason\":null,\"stop_sequence\":null,\"usage\":{\"input_tokens\":656,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":26,\"service_tier\":\"standard\"}}       }\n\nevent: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"tool_use\",\"id\":\"toolu_01LZ7WbFLVzFd16Ur9RHXeGz\",\"name\":\"get_weather\",\"input\":{}}   }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"\"}}\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"{\\\"loca\"}             }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"tion\\\": \\\"Sa\"}         }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"n Fran\"}          }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"cisco, CA\\\"\"}        }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\", \\\"units\\\"\"}            }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\": \\\"f\\\"}\"}              }\n\nevent: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0 }\n\nevent: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"tool_use\",\"stop_sequence\":null},\"usage\":{\"input_tokens\":656,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"output_tokens\":74}              }\n\nevent: message_stop\ndata: {\"type\":\"message_stop\"        }\n\n"


You planning on doing that in a follow-on? or in this PR too

RobertCraigie · 2025-12-09T12:49:04Z

tests/lib/utils.py

        return self.__class__.__name__.split("[", maxsplit=1)[0]

-    with monkeypatch.context() as m:
+    with pytest.MonkeyPatch.context() as m:


q: does this still work as a test-scoped patch? I did the forward monkeypatch setup before to make sure any patches are cleaned up properly; If this works too that would be cool.

I did the forward monkeypatch setup before to make sure any patches are cleaned up properly

It cleans up immediately after exiting the context manager, in both cases

oh very nice, I missed the context manager 🤦

karpetrosyan requested a review from a team as a code owner December 3, 2025 18:09

karpetrosyan marked this pull request as draft December 3, 2025 18:09

karpetrosyan force-pushed the improve-requests-snapshots branch 3 times, most recently from afe02bc to 9914730 Compare December 5, 2025 14:57

chore(internal): simplify http snapshots

87759a3

karpetrosyan force-pushed the improve-requests-snapshots branch from 73d3f9a to 87759a3 Compare December 5, 2025 15:12

minor fixes

b987a0b

karpetrosyan commented Dec 5, 2025

View reviewed changes

karpetrosyan added 4 commits December 5, 2025 19:16

minor fix

813cf79

fix lockfile

2e8c049

fix griffe

5011e12

remove new typevars from the snapshots

68202af

karpetrosyan marked this pull request as ready for review December 5, 2025 15:41

karpetrosyan requested a review from RobertCraigie December 5, 2025 15:41

karpetrosyan added the ready-to-merge label Dec 9, 2025

RobertCraigie reviewed Dec 9, 2025

View reviewed changes

karpetrosyan added 2 commits December 16, 2025 14:04

include request data in a snapshot, exclude some headers

9f97d6c

bump http-snapshot version

4d1813d

chore(internal): simplify http snapshots #1092

Are you sure you want to change the base?

chore(internal): simplify http snapshots #1092

Conversation

karpetrosyan commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karpetrosyan Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

karpetrosyan commented Dec 3, 2025 •

edited

Loading

karpetrosyan Dec 9, 2025 •

edited

Loading