Debugging Silent Drops with VPC Flow Logs: Ingress vs. Egress Rule Pitfalls

When a microservice deployed on Google Kubernetes Engine (GKE) or Compute Engine fails to connect to an external API, a database, or another internal subnet, application logs usually report something generic:

Connection timed out after 30000ms
# or
HTTPConnectionPool: Max retries exceeded (Caused by ConnectTimeoutError)

Application teams often assume the remote host is down or misconfigured. Infrastructure engineers frequently discover that the packet was silently dropped before it ever left the VPC.

In Google Cloud Platform (GCP), VPC Flow Logs provide the definitive ground truth for network traffic. However, reading them effectively requires understanding how firewall evaluation, ingress vs. egress rules, and stateful tracking interact.

Here is a practical troubleshooting workflow for isolating silent connection drops.

1. Enabling VPC Flow Logs on the Subnet

VPC Flow Logs must be enabled on the specific subnet where your workload resides. If it is disabled, turn it on with appropriate sampling:

• In Cloud Console: VPC network > VPCs > Click Subnet > Edit > Toggle Flow Logs ON.
• Aggregation Interval: Set to 5 seconds or 30 seconds during active troubleshooting.
• Sample Rate: Set to 1.0 (100%) temporarily to capture every single connection attempt.
• Include Metadata: Ensure metadata is enabled so logs display GKE pod names, service names, and VM instance identifiers.

2. The Filter Query in Cloud Logging

Open Cloud Logging (Log Explorer) and run a targeted query focusing on network traffic from your source IP or destination IP:

resource.type="gce_subnetwork"
log_name="projects/YOUR_PROJECT_ID/logs/compute.googleapis.com%2Fvpc_flows"
jsonPayload.connection.src_ip="10.128.0.45"
jsonPayload.connection.dest_ip="198.51.100.20"

Look closely at two fields in the output payload:

• jsonPayload.reporter: Indicates whether the log was captured at the SRC (source VM/node) or DEST (destination).
• jsonPayload.action: Either ACCEPT or REJECT.

3. The Classic Pitfalls

Pitfall A: Ingress vs. Egress Asymmetry

GCP VPC firewalls are stateful. If an egress connection is allowed out, the return traffic (the TCP response) is automatically permitted back in, regardless of ingress rules.

However:

1. Outbound blocked by Egress Rule: If you see action: "REJECT" with reporter: "SRC", an outbound firewall rule or the default implied egress deny has blocked the initial TCP SYN packet.
2. Inbound blocked at Destination: If your source VM sends the packet (action: "ACCEPT" at SRC), but the destination host reports action: "REJECT" (or no log appears), an ingress rule on the target VPC or instance tag rejected the incoming packet.

Pitfall B: Priority Shadowing (Rule Overlap)

GCP firewall rules evaluate in order of Priority (an integer from 0 to 65535, where lower numbers have higher priority).

A common trap:

• You add a rule with priority 1000 to allow traffic to 10.20.0.0/16.
• A broader organization or network policy rule exists at priority 500 denying all egress to internal RFC 1918 subnets.
• The priority 500 rule matches first and silently drops the traffic.

In Cloud Logging, inspect the rule attribution:

jsonPayload.rule_details.rule_name
jsonPayload.rule_details.priority

This confirms exactly which firewall rule triggered the decision.

Pitfall C: Missing Cloud NAT or Public IP Mappings

If your GKE cluster uses Private Nodes (nodes without public external IPs) and your container attempts to reach an external third-party API:

• Outbound traffic must traverse a Cloud NAT Gateway.
• If the Cloud NAT gateway runs out of allocated IP ports or is not configured for that specific subnet, connections will hang indefinitely.
• The VPC flow log on the subnet may show action: "ACCEPT" because the packet left the node toward the NAT gateway, but the connection never completes external handshake.
• Check Cloud NAT metrics (cloudnat.googleapis.com/nat_allocation_failed) in Cloud Monitoring.

4. Quick Runbook for Triage

When facing a connection timeout:

1. Verify Source Egress:

   text

   jsonPayload.connection.src_ip="<SRC_IP>" 
   jsonPayload.connection.dest_port=<PORT>

   Action: Is action == "ACCEPT"? If REJECT, update the egress firewall rule or target tag.

2. Verify Destination Ingress (if internal GCP host):

   text

   jsonPayload.connection.dest_ip="<DEST_IP>"
   jsonPayload.connection.dest_port=<PORT>

   Action: If missing entirely, check routing tables and VPC Peering / Shared VPC connectivity. If REJECT, fix destination ingress firewall tags.

3. Check MTU Mismatches: If small packets (TCP SYN) establish a connection, but larger payload transfers stall or disconnect immediately, check the network interface MTU (e.g., 1460 bytes vs standard 1500 bytes).

Summary

Never guess whether network connectivity is an application bug or a firewall issue. Turning on VPC Flow Logs for a 15-minute diagnostic window and filtering on jsonPayload.action provides immediate, undeniable proof of where and why packets are dropped.