An Ambient mode troubleshooting appendix covering two Istio and Envoy issues encountered in production.
July 13, 2026#kubernetes
14 posts
- Istio Part 3-4: Detecting 507 Status Codes and istiod Disconnections
- Istio Part 3-3: Safely Upgrading Ambient Mode
A runbook for upgrading Istio Ambient mode.
July 8, 2026 - Istio Part 3-2: Partially Enrolled Pods and the Untaint Controller
Ambient mode troubleshooting: when Pods are scheduled before istio-cni is ready on a node.
July 2, 2026 - Istio Part 3-1: 503 and Half-open Connections
Ambient mode troubleshooting: 503 errors caused by Envoy reusing stale connections.
June 26, 2026 - Istio Part 2: Ambient Mode Under the Hood via Envoy Configs
A deep dive into how Istio Ambient mode's HBONE tunneling and traffic redirection are implemented in Envoy configs.
April 14, 2026 - Istio Part 1: Why Istio Ambient Mode?
Introduction to why Channel.io's DevOps team chose Istio Ambient mode and how it works.
March 20, 2026 - Investigating Istio AuthorizationPolicy ext_authz in Envoy Config
Analyzing how Istio AuthorizationPolicy's action: CUSTOM works in Envoy config based on actual envoy config dump.
March 10, 2026 - A common Python async trap: one sync function that blocked the Liveness Probe
How calling a CPU-bound sync function inside an async endpoint blocked /v1/health
February 17, 2026 - RegularExpression Path Matching in HTTPRoute
Deep dive into Istio's HTTPRoute path matching priority and why PathPrefix always wins over RegularExpression
July 15, 2025 - Identifying Client IP in Istio (Proxy Protocol v2)
How to obtain the real client IP in an AWS NLB + Istio Gateway environment where even numTrustedProxies doesn't help, solved with Proxy Protocol v2.
April 2, 2025 - How Channel.io Adopted GitHub Actions Part 2: Operating ARC in Production and Troubleshooting
Case studies and lessons learned from operating and troubleshooting the Kubernetes Actions Runner Controller (ARC) at Channel.io.
March 19, 2025 - How Channel.io Adopted GitHub Actions Part 1: Building ARC and Supporting Container Jobs
A story about how Channel.io moved from CircleCI to GitHub Actions with self-hosted runners.
March 18, 2025 - Ubuntu; How to install on-premise Kubernetes using kubeadm
Complete guide to install vanilla Kubernetes on Ubuntu machines using kubeadm. Also providing ways to operate Nvidia GPU nodes on kubernetes.
May 11, 2024 - Arch Linux; How to install on-premise Kubernetes using kubeadm
Complete guide to install vanilla Kubernetes on Arch Linux machines using kubeadm.
March 14, 2024