Identifying Client IP in Istio (Proxy Protocol v2)

This post covers the configuration process for obtaining the real client IP in an AWS NLB + Istio Gateway environment, along with the problems encountered and their solutions.

Previous Setup: ALB + Direct Pod Connection

The existing infrastructure used ALB with IP-based Target Groups.

• AWS Application Load Balancer (ALB)
• Target Group: ip mode (protocol: HTTP)
• Traffic flow: Client → ALB → Pod

Client IP was obtained via ALB’s X-Forwarded-For (Append) option. The backend extracted the IP like this:

// Pseudo-code
func getClientIP(req) string {
    if xff := req.Header["X-Forwarded-For"]; xff not empty {
        return xff[0] // first IP
    }
    return req.RemoteAddr // fallback
}

New Setup: NLB + Istio Gateway

The Problem

After switching to the new architecture, it became impossible to obtain the client IP.

• AWS NLB (Network Load Balancer)
• Target Group: ip mode (protocol: TCP)
• Istio Gateway (Envoy-based)
• Traffic flow: Client → NLB → Istio Gateway (Envoy Proxy) → Pod

When checking from pods like httpbin, X-Envoy-External-Address contained the NLB’s IP address. Changing numTrustedProxies to 1 or 2 on the Istio Gateway had no effect. The XFF header didn’t exist, and RemoteAddr also resolved to the NLB address.

Resolution Process

Step 1: Enable NLB’s Preserve Client IP Setting

Enabled the Preserve Client IP address option on the NLB Target Group.

After this change, X-Envoy-External-Address in httpbin showed the actual client IP. The assumption is that Envoy reads the TCP Client IP and sets it as RemoteAddr.

Note: postmanlabs/httpbin intentionally drops the X-Forwarded-For header. To inspect XFF, consider using mccutchen/go-httpbin instead.

However, a new problem emerged:

• Some client requests received no response (pending, empty response)
• Reproducible via curl and browser testing
• Disabling Preserve Client IP made the issue disappear

Root Cause Analysis

The Preserve Client IP feature maintains the actual packet’s source IP as the client’s public IP.

The inbound flow Client → NLB → Envoy works fine, but the problem occurs when Envoy sends the response.

With Preserve Client IP enabled, the NLB preserves the source IP as-is. The EC2 (Pod) receives packets with the client’s public IP (50.1.1.1) as the source.

flowchart LR
    Client["Client<br/>50.1.1.1"] -->|"Src: 50.1.1.1:51234<br/>Dst: NLB IP:Port"| NLB["NLB ENI<br/>10.1.1.10"]
    NLB -->|"Src: 50.1.1.1:51234<br/>Dst: 10.1.1.100:80<br/><b>source IP preserved!</b>"| EC2["EC2 (Pod)<br/>10.1.1.100"]

The problem occurs on the response path. When EC2 sends a response, the destination IP is the client’s public IP (50.1.1.1), so return traffic goes through the NAT Gateway instead of the NLB.

flowchart TB
    subgraph VPC
        IGW["IGW"]
        subgraph Public Subnet
            NLB["NLB"]
            NATGW["NAT GW"]
        end
        subgraph Private Subnet
            EC2["EC2 (Pod)<br/>10.1.1.100"]
        end
    end
    Client["Client"] -->|"request (normal inbound)"| IGW
    IGW --> NLB
    NLB --> EC2
    EC2 -->|"response: dst=50.1.1.1 (client)<br/>bypasses NLB!"| NATGW
    NATGW -->|"connection broken ✗"| Client

    style NATGW fill:#f66,stroke:#c00,color:#fff

The response packet from EC2 has the client’s public IP as its destination, so VPC routing sends it through the NAT Gateway. The client sent the request via the NLB but gets a response from a different path, breaking the TCP connection.

Final Solution: Enable Proxy Protocol v2

Configure the NLB Target Group as follows:

• Preserve Client IP: OFF
• Proxy Protocol v2: ON

Proxy Protocol support must also be enabled on the Istio Gateway.

How It Works

Proxy Protocol transmits the origin (client) IP information during the initial TCP handshake. Envoy extracts the Client IP using this Layer-4 information.

Since actual packets are delivered with the NLB IP, it doesn’t affect NAT flows. The original source/destination IP structure is preserved while IP information is provided through a separate layer.

Results by Configuration

Preserve Client IP	Proxy Protocol v2	Result
OFF	OFF	Cannot obtain client IP
ON	OFF	Unstable responses, Empty Response errors
OFF	ON	Client IP accurately received, stable communication (healthcheck still fails)
ON	ON	Still produces Empty Response errors

Recommended Configuration

Recommended settings when using AWS NLB + Istio:

NLB Target Group:

• Preserve Client IP: OFF
• Proxy Protocol v2: ON
• HealthCheck: Replace HTTP port:15021 /healthz/ready with TCP healthcheck on trafficPort

Istio Gateway (Envoy):

• proxyProtocol: ENABLE
• numTrustedProxies: Set if needed (when using XFF alongside)

Istio Configuration

Proxy Protocol must be enabled not only on the NLB Target Group but also on the traffic receiver (Envoy proxy).

Enabling proxyProtocol at the Cluster level via configMap only applies to Gateway TCP Listeners. For HTTP listeners, you need to add the following annotation to the Gateway Deployment:

proxy.istio.io/config: '{"gatewayTopology": {"proxyProtocol": {}}}'

Both sides must be configured for it to work properly.

targetGroup Proxy Protocol	Istio Proxy Protocol	Result
ON	OFF	Empty reply from server
OFF	ON	400 Bad Request from envoy

Note: Envoy config has an option to accept non-proxy-protocol traffic even when proxy protocol is enabled.

Proxy Protocol v2 Considerations

There are two things to consider when enabling Proxy Protocol v2 on the NLB:

1. The upstream gateway/proxy must support Proxy Protocol — The Envoy (Istio Gateway) behind the NLB needs to be able to parse the Proxy Protocol header. For Istio, this is handled by the configuration described above.
2. Performance overhead — A Proxy Protocol header is added during the TCP handshake, which could introduce some overhead.

To measure the performance impact, we ran benchmarks with 100 RPS GET requests.

Proxy Protocol v2 OFF (baseline):

CPU (req/limit)	Mem (req/limit)	RPS	p(90)	p(95)	p(99)
500m / 3000m	1024Mi / 1024Mi	1500	~3.1ms	~3.5ms	~8.4ms
500m / 3000m	1024Mi / 1024Mi	2000	~3.7ms	~4.4ms	~13ms

Proxy Protocol v2 ON:

CPU (req/limit)	Mem (req/limit)	RPS	p(90)	p(95)	p(99)
500m / 3000m	1024Mi / 1024Mi	1500	~3.7ms	~4.4ms	~9.8ms
500m / 3000m	1024Mi / 1024Mi	2000	~3.5ms	~4ms	~7ms
500m / 3000m	1024Mi / 1024Mi	4000	~4.4ms	~6.3ms	~13.9ms

At the same RPS, the latency difference was negligible. The performance overhead from Proxy Protocol v2 was insignificant in practice.

Conclusion

In the previous ALB environment, Client IP was obtained via the XFF header Append mechanism. In an NLB + Istio environment, the client IP can be reliably extracted using the TCP Target Group’s Proxy Protocol v2 feature.

The Preserve Client IP feature caused connection stability and NAT routing issues in our environment, and enabling Proxy Protocol v2 resolved the problem by reliably conveying client information at L4.