Istio Part 3-4: Detecting 507 Status Codes and istiod Disconnections

The original version of this post is available on the Channel.io Tech Blog.

Hi, this is Dylan and Jetty (Jaehong Jung) from the Channel.io DevOps team.

This is Part 3-4 of our Istio Ambient mode adoption series. Parts 3-1, 3-2, and 3-3 each explored one topic in depth.

• Part 3-1: 503 and Half-open Connections
• Part 3-2: Partially Enrolled Pods and the Untaint Controller
• Part 3-3: Safely Upgrading Ambient Mode

Part 3-4 is a little different. Rather than following one major root cause to the end, it collects two memorable Istio and Envoy issues we encountered in production as an appendix.

Neither issue is exclusive to Ambient mode. We observed both at waypoints, but they are fundamentally about how Envoy buffers and retries requests, and how to reflect the xDS connection between Envoy and istiod in readiness. The same perspective can be useful in environments running sidecar mode or ingress gateways.

This post covers two topics.

1. An unfamiliar 507 Insufficient Storage: large request payloads and retry buffer limits
2. How to detect gateways and waypoints that appear disconnected from istiod

---

1. An Unfamiliar 507 Status Code

1.1 The Problem

During production operation, an application reported an unfamiliar error.

MailException: MessengerServer Error: failed to send email
MessengerServer Error: failed to send email
status: 507
msg: exceeded request buffer limit while retrying upstream

HTTP 503 and 504 responses are common enough when operating Istio and Envoy, but 507 was unfamiliar. 507 Insufficient Storage generally means the server could not secure enough storage to process the request, so we initially suspected insufficient storage resources on the application server.

The waypoint access log showed, however, that the application did not create this response. It was a local reply generated directly by Envoy.

{
  "method": "POST",
  "path": "/xxx/xxx/email/send",
  "protocol": "HTTP/1.1",
  "bytes_received": 1114112,
  "bytes_sent": 53,
  "duration": 4,
  "response_code": 507,
  "response_code_details": "request_payload_exceeded_retry_buffer_limit",
  "upstream_cluster": "inbound-vip|4001|http|cht-app-xxx-xxx.channel.svc.cluster.local;",
  "upstream_host": "envoy://connect_originate/10.190.xxx.xxx:xxx",
  "user_agent": "okhttp/5.3.0"
}

The key field was response_code_details.

request_payload_exceeded_retry_buffer_limit

Envoy’s documentation describes it this way.

Envoy is doing streaming proxying but too much data arrived while waiting to attempt a retry.

Envoy was streaming the request body to the upstream while buffering some of the data for a possible retry. The request payload it needed to retain for that retry exceeded the buffer limit.

1.2 A 507 Is Not Quite a Request Size Limit

The confusing part is that this 507 differs from 413 Content Too Large.

A 413 is a 4xx response returned when a request body exceeds a limit defined by the server. This 507 is less “the payload itself is too large, so reject it” and more “retrying this request requires replaying the request body, but Envoy’s retained buffer is not large enough to replay it.”

For normal streaming proxying, a large payload is not always a problem. Envoy does not load the entire request body into memory before sending it upstream. It accepts and forwards the body in chunks.

sequenceDiagram
    participant C as Client
    participant E as Envoy<br/>(waypoint/gateway/sidecar)
    participant U as Upstream

    C->>E: chunk 1
    E->>U: chunk 1
    C->>E: chunk 2
    E->>U: chunk 2
    C->>E: chunk 3
    E->>U: chunk 3
    U-->>E: 200 OK
    E-->>C: 200 OK

Even when the full payload is tens of megabytes, Envoy does not need to hold all of it in the buffer at once.

The problem appears when a retry is needed. To retry after an upstream failure, Envoy must be able to send the same request upstream again. For a POST body that has already been streamed onward, Envoy needs to retain at least enough of that body to replay the request.

Once the buffer exceeds its limit, Envoy can no longer retain the request body for retry. If a retry condition such as an upstream reset or 5xx response then occurs, Envoy cannot replay the request and returns a local 507 reply.

sequenceDiagram
    participant C as Client
    participant E as Envoy<br/>(retry enabled)
    participant U as Upstream

    C->>E: chunk 1 (200KB)
    Note over E: Save in retry buffer
    E->>U: chunk 1

    C->>E: chunk 2 (500KB)
    Note over E: 700KB buffered
    E->>U: chunk 2

    C->>E: chunk 3 (500KB)
    Note over E: Buffer limit exceeded<br/>Cannot buffer more data for retry
    E->>U: chunk 3

    U--xE: reset / retryable failure
    Note over E: Retry requires the complete request body<br/>but the buffer lacks the full payload
    E-->>C: 507 Insufficient Storage<br/>request_payload_exceeded_retry_buffer_limit

From this perspective, “the request failed because its payload was large” is less accurate than “the request failed when a large payload met a condition that required a retry.”

1.3 Why 507?

In the request_payload_exceeded_retry_buffer_limit case, Envoy’s implementation sends a local reply with Http::Code::InsufficientStorage, or 507.

Conceptually, the flow is as follows.

1. Request body data arrives.
2. Envoy accumulates data in the decoding buffer for retries or shadowing.
3. It checks whether adding the new data would exceed the effective buffer limit.
4. If the limit is exceeded while retrying or shadowing may be needed, Envoy stops buffering.
5. If a retry is later required, Envoy cannot replay the request.
6. Envoy returns a 507 local reply.

In Envoy’s code, exceeding the buffer limit resets the retry state. When Envoy later creates the local reply, it sets the response code detail to RequestPayloadExceededRetryBufferLimit.

response_code: 507
response_code_details: request_payload_exceeded_retry_buffer_limit
body: exceeded request buffer limit while retrying upstream

This 507 does not mean that the application is short on storage or disk space. It signals that Envoy can no longer replay the request payload for a retry.

1.4 per_connection_buffer_limit_bytes and 1 MB

Our investigation naturally led to Envoy’s per_connection_buffer_limit_bytes. Envoy’s default buffer limit is 1 MB, and this default can apply when Istio does not provide a separate setting.

This value is not exactly the same as a maximum request payload size. A large request can pass when it is simply streamed. But when Envoy needs to retain the request body for a retry, the buffer limit can act as an effective boundary.

This is awkward for operators. Even without an explicit request body limit, requests over a certain size can fail with 507 under particular conditions. It is especially hard to identify when the condition only appears as “large payload plus retryable failure,” surfacing when a separate outage or reset occurs.

1.5 Evaluating Mitigations

We considered three broad options.

1) Increase the buffer limit

The most direct option is to increase per_connection_buffer_limit_bytes. A larger value lets Envoy preserve the retry buffer for larger payloads.

The operational choice is not simple. We need to decide how many megabytes to allow, and media or file uploads make it impractical to increase the value without a bound. Envoy’s memory use must also be considered. Although the limit is only consumed as data accumulates, many concurrent connections with large requests can create memory pressure.

We also need to decide whether to apply the change across every gateway and waypoint, or only to selected routes and services. A global setting increases the blast radius; targeted settings increase management complexity.

2) Disable retries

Disabling retries can eliminate this problem because Envoy no longer needs to replay the request body.

That was difficult to choose in our environment. As covered in Part 3-1, operating Ambient mode had already shown us that retries are needed for resets between waypoints and ztunnel. Disabling retries might reduce 507 responses while exposing many other 503 responses directly to users.

3) Retry large payloads in the client

The most practical direction was to handle retries for large payload requests in the client or calling application.

Safely buffering and retrying every large POST body in Envoy is expensive. Large-payload requests often represent heavy, long-running operations such as uploads, email delivery, or document processing. For these APIs, application-level idempotency keys or duplicate-processing protection combined with 5xx retries can be more explicit and safer.

The main lesson was this.

Enabling retries in Envoy or Istio can create an “invisible buffer limit” for large request bodies. Large payloads do not always fail, but when a retry is required they may fail with 507 request_payload_exceeded_retry_buffer_limit.

2. Gateways and Waypoints That Appear Disconnected from istiod

We have not yet identified a definitive root cause for the second issue, so we are cautious about attributing it to any one cause. Instead, this section focuses on what we observed and which metrics we evaluated for detection and mitigation.

2.1 The Problem

In production, a gateway or waypoint Envoy sometimes appeared unable to maintain its xDS connection to istiod. The following message repeated in the logs.

DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1614s ago:
14, connection error: desc = "transport: Error while dialing: dial tcp:
lookup istiod.istio-system.svc: i/o timeout"{via_upstream}

The important fields were these.

• DeltaAggregatedResources: Envoy was using a Delta xDS stream.
• xds-grpc: Envoy’s xDS cluster, which actually points to the pilot-agent UDS proxy inside the Pod.
• 14: gRPC Unavailable.
• lookup istiod.istio-system.svc: i/o timeout: a DNS timeout while pilot-agent was connecting to istiod.
• closed since 1614s ago: stream closure and retry had continued for a long time after the first failure.

The message looked like an istiod DNS lookup failure. lookup ... i/o timeout itself strongly suggested DNS resolution failure.

We initially suspected CoreDNS or a cluster-wide DNS issue. But we found no widespread DNS lookup errors across the cluster at the same time, and CoreDNS itself showed no obvious anomaly. The error looked like a DNS failure, but we lacked evidence to claim that CoreDNS was the root cause or that cluster DNS was broadly broken.

At that point, we changed the direction of the task. Rather than continuing to chase a single root cause in DNS lookup, we focused on how to avoid a gateway or waypoint continuing to receive traffic while holding only stale xDS configuration if the situation recurred—or at least how to detect and mitigate that condition quickly.

This section therefore focuses on how to detect and mitigate the state rather than on its underlying cause.

2.2 Connection Structure: Envoy, pilot-agent, and istiod

Istio gateway and waypoint Pods contain both Envoy and pilot-agent. Envoy does not connect directly to istiod; it communicates through the local pilot-agent.

flowchart LR
    subgraph POD["gateway/waypoint Pod"]
        direction LR
        E["Envoy<br/>(istio-proxy)"]
        A["pilot-agent"]
        E <-->|"UDS<br/>xds-grpc"| A
    end

    A <-->|"TCP/TLS<br/>DNS: istiod.istio-system.svc"| I["istiod"]

    style POD fill:#f8f9fa,stroke:#adb5bd,stroke-dasharray: 5 5
    style E fill:#e3f2fd
    style A fill:#e8f5e9
    style I fill:#fff3cd

From Envoy’s perspective, the xDS upstream is the xds-grpc cluster. But that cluster does not point to a remote istiod. It points to the pilot-agent UDS in the same Pod. pilot-agent then establishes a TCP/TLS connection to istiod and relays the xDS stream between Envoy and istiod.

There are therefore two separate places where a failure can occur.

1. Envoy ↔ pilot-agent: a UDS connection inside the Pod
2. pilot-agent ↔ istiod: a TCP/TLS connection that traverses Kubernetes DNS and the network

In the logs we observed, pilot-agent failed while connecting to istiod, and that error propagated to Envoy as a closed gRPC stream.

2.3 pilot-agent Errors Propagate to Envoy

The Istio agent’s xDS proxy implementation does not hide an upstream istiod connection failure from the downstream Envoy stream. Conceptually, it behaves as follows.

func (p *XdsProxy) handleStream(downstream adsStream) error {
    upstreamConn, err := p.buildUpstreamConn(ctx)
    if err != nil {
        return err
    }

    upstream, err := xds.StreamAggregatedResources(ctx, ...)
    if err != nil {
        return err
    }

    for {
        select {
        case err := <-con.upstreamError:
            return err
        case err := <-con.downstreamError:
            return err
        }
    }
}

When Recv() returns an error on the istiod-side stream, the error is sent to the upstream error channel and the handler returns it. That also closes the gRPC configuration stream visible to Envoy.

When the gRPC stream closes, Envoy sets control_plane.connected_state to 0, waits for backoff, and then attempts to reconnect.

onRemoteClose()
  -> control_plane.connected_state = 0
  -> setRetryTimer()
  -> create a new xDS stream after backoff

If the failure continues, it can produce the following loop.

sequenceDiagram
    participant E as Envoy
    participant A as pilot-agent
    participant D as DNS
    participant I as istiod

    E->>A: xDS stream open (UDS)
    Note over E: connected_state = 1
    A->>D: lookup istiod.istio-system.svc
    D--xA: i/o timeout
    A--xE: Propagate gRPC error
    Note over E: connected_state = 0<br/>backoff timer
    E->>A: retry xDS stream
    Note over E,A: Repeats while the failure persists

One important detail is that connected_state does not necessarily remain cleanly at 0. Each reconnection attempt can briefly open the Envoy ↔ pilot-agent UDS stream, at which point connected_state rises to 1. If pilot-agent then fails to connect to istiod, it falls back to 0.

This means connected_state=1 can appear at certain observation points even while the Pod cannot actually communicate with istiod.

2.4 Limitations of the Default readinessProbe

Istio’s default readiness probe is closer to asking “has the first xDS configuration arrived?” The pilot-agent status probe maintains states such as receivedFirstUpdate and atleastOnceReady. Once it has become ready, a later xDS disconnection may not immediately fail readiness.

The implementation is shaped roughly like this.

type Probe struct {
    receivedFirstUpdate bool
    atleastOnceReady    bool
}

func (p *Probe) isReady() error {
    if !p.receivedFirstUpdate {
        return errors.New("config not received from XDS proxy")
    }
    return nil
}

This works for startup readiness because it prevents Envoy from receiving traffic before its first configuration arrives.

But it is not enough to detect a gateway or waypoint that became ready once and then remained disconnected from istiod for a long time. The fact that it received the first configuration remains true.

The distinction matters operationally. A gateway or waypoint can continue processing traffic for a while with configuration it has already received. But after a long disconnection from istiod, it may miss control plane updates such as new configuration, endpoints, and certificate rotation. We therefore want to observe not only whether the process is live, but also whether it is currently connected to the xDS control plane.

2.5 Which Metrics Can We Use?

We evaluated two major metrics.

1) envoy_control_plane_connected_state

The internal Envoy stat is named:

control_plane.connected_state

When exported to Prometheus, it usually appears as:

envoy_control_plane_connected_state

This gauge is 1 while the xDS gRPC stream is open and 0 while it is closed. As discussed earlier, it reflects the Envoy ↔ pilot-agent stream. Because pilot-agent propagates istiod connection failures as errors, it can also indirectly reflect istiod connectivity problems.

Collecting the stat requires a correct proxyStatsMatcher configuration. There is one easy mistake to make here.

proxyStatsMatcher.inclusionRegexps matches the internal Envoy stat name, not the Prometheus metric name. The following configuration is therefore incorrect.

proxyStatsMatcher:
  inclusionRegexps:
    - envoy_control_plane_connected_state  # Prometheus name; does not match

The correct setting uses the internal Envoy stat name, control_plane.connected_state.

proxyStatsMatcher:
  inclusionRegexps:
    - control_plane\.connected_state

With an incorrect setting, the stat itself may not be generated even when /stats is queried. This setting was incorrect during our incident, which prevented us from using the metric immediately in the post-incident analysis.

2) envoy_cluster_upstream_cx_active{cluster_name="xds-grpc"}

Another useful value is the number of active connections in the xds-grpc cluster.

envoy_cluster_upstream_cx_active{cluster_name="xds-grpc"}

This represents the number of upstream connections Envoy has established to the pilot-agent UDS. It is normally 1. If the gRPC stream repeatedly closes because pilot-agent cannot connect to istiod—for example during a DNS failure—it may fall to 0. If the Pod itself cannot be scraped, the time series may disappear entirely.

This is not a perfect signal of istiod connectivity either, because it observes the Envoy ↔ pilot-agent UDS connection. Still, it is practical in cases like ours where pilot-agent errors propagate into stream closure in Envoy.

2.6 Detecting It with a readinessProbe

Metrics can drive alerts, but they do not remove the affected gateway or waypoint from Kubernetes Service endpoints. We therefore considered incorporating the xDS connection state into the readiness probe.

For example, the probe can check both the standard /healthz/ready endpoint and control_plane.connected_state.

readinessProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - |
        curl -sf http://localhost:15021/healthz/ready || exit 1
        STATE=$(pilot-agent request GET 'stats?filter=control_plane.connected_state' \
          | awk -F': ' '/control_plane\.connected_state/ {print $2; exit}')
        [ "$STATE" = "1" ] || exit 1
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 3

If the xDS stream remains at 0 for long enough, Pod readiness becomes false and Kubernetes can remove the Pod from Service endpoints.

There is another trap. As described above, a reconnection loop can move connected_state between 0 and 1 during an ongoing failure. The default successThreshold is 1, so a single successful probe immediately returns the Pod to Ready. The endpoint can then flap in and out while the failure continues.

sequenceDiagram
    autonumber
    participant X as xDS stream<br/>connected_state
    participant P as readinessProbe<br/>(period 5s)
    participant K as Kubernetes<br/>EndpointSlice
    participant T as Traffic

    Note over X,T: The istiod connection failure persists<br/>Envoy alternates between 0 and 1 due to the reconnect loop

    rect rgb(255, 235, 238)
    Note over X: connected_state = 0<br/>backoff period
    P->>X: probe #1 → fail
    P->>X: probe #2 → fail
    P->>X: probe #3 → fail
    P->>K: failureThreshold=3<br/>Ready=false
    K->>T: Remove endpoint
    end

    rect rgb(232, 245, 233)
    Note over X: connected_state = 1<br/>Temporarily appears successful during reconnection
    P->>X: probe #4 → pass
    P->>K: successThreshold=1<br/>Ready=true
    K->>T: Re-add endpoint
    T-->>K: Traffic may return to the failing Pod
    end

    rect rgb(255, 235, 238)
    Note over X: pilot-agent → istiod connection fails<br/>Stream closes and state returns to 0
    P->>X: probe #5 → fail
    P->>X: probe #6 → fail
    P->>X: probe #7 → fail
    P->>K: Ready=false
    K->>T: Remove endpoint
    end

    Note over K,T: Ready/NotReady keeps flapping<br/>even though the failure has not ended

It is safer to require consecutive successes before returning to Ready, rather than configuring only a failure threshold.

readinessProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - |
        curl -sf http://localhost:15021/healthz/ready || exit 1
        STATE=$(pilot-agent request GET 'stats?filter=control_plane.connected_state' \
          | awk -F': ' '/control_plane\.connected_state/ {print $2; exit}')
        [ "$STATE" = "1" ] || exit 1
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 3
  successThreshold: 3

failureThreshold: 3 marks the Pod NotReady only after three consecutive failures, while successThreshold: 3 requires three consecutive successes before returning it to Ready. A brief connected_state=1 during the reconnection loop is no longer enough to re-enter Ready.

This approach is not perfect. connected_state observes Envoy’s xDS stream, not a direct connection from Envoy to istiod. Parsing /stats on every probe is also somewhat crude, and an incorrect configuration can introduce startup delays or flapping.

Even so, the default readiness check is close to “has the first configuration arrived?” For critical entry points such as gateways and waypoints, we found it operationally meaningful to observe the current control plane connection separately.

---

Conclusion

This appendix covered two Istio and Envoy issues we encountered in production.

The first issue showed how Envoy’s buffer limit can surface as an unexpected response when large request payloads meet retries. Even without an explicit request body size limit, replaying a payload for a retry can produce a 507 with request_payload_exceeded_retry_buffer_limit. When applying retry policies to requests with large bodies such as POSTs, we need to decide where Envoy’s responsibility ends and where retry responsibility moves to the client or application.

We have not conclusively identified the root cause of the second issue, but it showed how to detect a gateway or waypoint that appears to have lost its xDS connection to istiod. The default readiness probe may not adequately reflect a control plane disconnection after the first configuration has arrived. Consider control_plane.connected_state, the xds-grpc connection metric, and, when appropriate, a custom readiness probe together.

Neither issue is unique to Ambient mode. Both arise from operating Istio with Envoy as the actual data plane. Even without Ambient mode, they are useful cases to keep in mind when operating Istio gateways or sidecar Envoys.

Thank you for reading.

• Part 1: Why Istio Ambient Mode?
• Part 2: Ambient Mode Under the Hood via Envoy Configs
• Part 3: Surprising Issues and Troubleshooting in Production
  • Part 3-1: 503 and Half-open Connections
  • Part 3-2: Partially Enrolled Pods and the Untaint Controller
  • Part 3-3: Safely Upgrading Ambient Mode
  • Part 3-4: Detecting 507 Status Codes and istiod Disconnections (this post)