linkerd-proxy rejects large headers (>510KB) despite increased MAX_HEADER_LIST_SIZE

[sratt]

Use Case: We need to support request headers of approximately 1MB in size due to large authentication headers in our service requests.

Setup:

1. A simple GET request is made to Service B, routed through Service A.
2. Service A pods have the linkerd-proxy sidecar injected.
3. We’ve applied the Helm values suggested in Make max_header_list_size configurable (Currently 16KB only) #14046 to the linkerd-proxy. These values are correctly reflected in the proxy container environment.

Problem:
When the GET request includes a large header (greater than ~510 KB), it is rejected by the linkerd-proxy. The following error appears in the logs of Service A's linkerd-proxy container:
INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=message head is too large client.addr=xx.xx.xxx.xx:xxxxx server.addr=xx.xx.xxx.xx:xx

Questions:

1. Is there anything missing in our setup to support large headers?
2. Is there a documented or hard-coded limit on header size in linkerd-proxy that we should be aware of?

Helm Chart Versions:

Chart.yaml:

  - condition: linkerd-crds.enabled
    name: linkerd-crds
    repository: https://helm.linkerd.io/edge
    version: 1.9.4-edge

   condition: linkerd-control-plane.enabled
   name: linkerd-control-plane
   repository: https://helm.linkerd.io/edge
   version: 1.18.10-edge

Values.yaml:

  proxy: # linkerd-proxy-injector
    additionalEnv:
      - name: LINKERD2_PROXY_OUTBOUND_SERVER_HTTP2_MAX_HEADER_LIST_SIZE
        value: "1048576"
      - name: LINKERD2_PROXY_INBOUND_SERVER_HTTP2_MAX_HEADER_LIST_SIZE
        value: "1048576"

[cratelyn]

thank you for filing this @sratt. do you know if your traffic is HTTP/1?

the LINKERD2_PROXY_OUTBOUND_SERVER_HTTP2_MAX_HEADER_LIST_SIZE and LINKERD2_PROXY_INBOUND_SERVER_HTTP2_MAX_HEADER_LIST_SIZE environment variables only increase the header list size for HTTP/2 traffic.

[sratt]

Thanks . We are now explicitly setting HTTP/2 on the request to make sure the environment variables are used and investigating why we are receiving a NGHTTP2_REFUSED_STREAM error when sending a large header. Will share more findings soon.