Packets Arrive but the App Times Out: The rp_filter Trap in Kubernetes

Asymmetric routing made me doubt my sanity; rp_filter was the culprit. “The packets are arriving. I can see them in tcpdump. But the application times out.” This was the most confusing debugging session I’d had in months. We were setting up a hybrid network where Kubernetes pods needed to communicate with services on an external network connected via a secondary interface. Health checks worked in one direction but failed in the other.

The smoking gun was that tcpdump on the destination interface showed incoming SYN packets, but the application socket never received them. Connection tracking showed no entries being created. It was as if the packets were arriving at the kernel and then vanishing into thin air before reaching the application.

After hours of debugging, I discovered the problem was rp_filter—Linux’s reverse path filtering feature. When packets arrive on an interface, the kernel checks whether the source address would be routable back through that same interface. If not, the packet is silently dropped as a “martian” packet. Our asymmetric routing (packets arriving on eth1, but return route through eth0) was being interpreted as spoofed traffic.

What made this particularly frustrating was that standard network debugging tools didn’t reveal the issue. tcpdump works before rp_filter, so it shows packets that are about to be dropped. conntrack shows nothing because the packet never makes it to connection tracking. You have to know to look at kernel martian packet counters or logs to see what’s happening.

Environment: Kubernetes 1.28+, multi-homed nodes (multiple NICs), hybrid/on-prem networks, VPN connections

Understanding rp_filter

What Reverse Path Filtering Does

Normal symmetric routing (rp_filter passes):

Client (10.0.0.5)
    |
    | SYN packet arrives on eth0
    v
+-------------------+
|    eth0           |
| (10.0.0.0/24)     |
+-------------------+
    |
    | Kernel checks: "If I were to send a packet TO 10.0.0.5,
    |                 which interface would I use?"
    | Route table says: 10.0.0.0/24 → eth0 ✓
    |
    v
Packet accepted → Application receives it

Asymmetric routing (rp_filter DROPS packet):

Client (10.0.0.5) → Router → Secondary Network
                              |
                              | SYN arrives on eth1
                              v
+-------------------+    +-------------------+
|    eth0           |    |    eth1           |
| (192.168.1.0/24)  |    | (10.100.0.0/24)   |
|  default route    |    |                   |
+-------------------+    +-------------------+
                              |
    Kernel checks: "If I were to send a packet TO 10.0.0.5,
                    which interface would I use?"
    Route table says: default route → eth0 ← DIFFERENT!
                              |
                              v
                    PACKET DROPPED AS MARTIAN!
                    (tcpdump saw it, app never will)

The Three rp_filter Modes

# Check current setting
cat /proc/sys/net/ipv4/conf/all/rp_filter
cat /proc/sys/net/ipv4/conf/eth0/rp_filter

# Modes:
# 0 = No validation (disabled) - packets always accepted
# 1 = Strict mode (default on many distros)
#     Source must be reachable via the SAME interface
# 2 = Loose mode
#     Source must be reachable via ANY interface

Why Kubernetes/CNI Creates Asymmetric Routes

Common scenarios where rp_filter bites:

1. Multi-homed nodes (VPN + primary network):
   Pod → VPN gateway → Remote service
   Return traffic: Remote → Primary interface (different path)

2. External load balancers with DSR (Direct Server Return):
   Client → LB → Node (eth0)
   Response: Node → Client directly (bypassing LB)
   If LB changed source IP, return path differs

3. CNI with multiple networks (Multus):
   Pod has: eth0 (cluster network), net1 (external network)
   Traffic flows asymmetrically between networks

4. Calico/Cilium BGP peering:
   Ingress on one interface, egress on another
   Based on BGP route selection

The Problem in Practice

Scenario: VPN Connection to On-Prem

Kubernetes node setup:

eth0: 192.168.1.100 (primary, default route to internet)
eth1: 10.100.0.100 (VPN tunnel to on-prem 10.0.0.0/8)

Route table:
default via 192.168.1.1 dev eth0
10.0.0.0/8 via 10.100.0.1 dev eth1
10.100.0.0/24 dev eth1

Problem: On-prem server 10.0.50.5 sends traffic that arrives via eth0
         (because on-prem router uses different path)
         But OUR route to 10.0.50.5 goes via eth1
         → rp_filter drops it!

The Debugging Journey

# 1. Application reports timeout
curl: (7) Failed to connect to 10.0.50.5 port 8080: Connection timed out

# 2. Check if packets are arriving (they are!)
tcpdump -i eth0 host 10.0.50.5
# 10:00:01 IP 10.0.50.5.8080 > 192.168.1.100.45678: Flags [S.], ...
# 10:00:04 IP 10.0.50.5.8080 > 192.168.1.100.45678: Flags [S.], ...
# ^ Server is responding! But we never see it...

# 3. Check conntrack (nothing!)
conntrack -L | grep 10.0.50.5
# No entries - packet never made it to connection tracking

# 4. Check iptables counters (nothing!)
iptables -L -v -n | grep 10.0.50.5
# No matches - packet never made it to iptables

# 5. Where are the packets going?!

Diagnosing rp_filter Drops

Check Martian Packet Logs

# Enable martian logging
echo 1 > /proc/sys/net/ipv4/conf/all/log_martians
echo 1 > /proc/sys/net/ipv4/conf/eth0/log_martians

# Watch kernel logs
dmesg -w | grep -i martian
# IPv4: martian source 10.0.50.5 from 192.168.1.100, on dev eth0

# Or check after the fact
dmesg | grep -i "martian\|ll header"

Check rp_filter Statistics

# IP statistics include rp_filter drops
cat /proc/net/snmp | grep -i ip
# Look for InAddrErrors - includes rp_filter drops

# Better: Use nstat for deltas
nstat -z | grep -i InAddrErrors
# IpInAddrErrors     543     0.0

# Watch in real-time
watch -n 1 'nstat -z | grep InAddrErrors'
# Send test traffic, see counter increment → rp_filter dropping

Verify the Route Asymmetry

# Check which interface WE would use to reach the source
ip route get 10.0.50.5
# 10.0.50.5 via 10.100.0.1 dev eth1 src 10.100.0.100

# But packet arrived on eth0!
# Asymmetric routing confirmed → rp_filter victim

Using eBPF for Deep Diagnosis

# If you have bcc-tools installed
# Trace packets being dropped by rp_filter
trace-cmd record -e 'skb:*' -f 'dev == "eth0"'

# Or use bpftrace
bpftrace -e 'kprobe:fib_validate_source {
  printf("rp_filter check: src=%s iif=%d\n",
         ntop(arg1), arg3);
}'

The Fix

Option 1: Loose Mode (Recommended)

# Set rp_filter to loose mode (2)
# Packet is accepted if source is reachable via ANY interface

# Temporary (until reboot)
echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 2 > /proc/sys/net/ipv4/conf/eth0/rp_filter

# Permanent via sysctl.conf
cat >> /etc/sysctl.d/99-rp-filter.conf << 'EOF'
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.eth0.rp_filter = 2
EOF

sysctl -p /etc/sysctl.d/99-rp-filter.conf

Option 2: Disable for Specific Interface

# If only one interface has asymmetric routing
echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter

# Note: The effective value is MAX(all, interface)
# So you may need to set 'all' to 0 or 2 as well

Option 3: Fix the Routing (Symmetric)

# If possible, make routing symmetric
# Add policy-based routing to ensure responses go back
# the same way requests came in

# Mark packets arriving on eth1
iptables -t mangle -A PREROUTING -i eth1 -j MARK --set-mark 100

# Create separate routing table
echo "100 vpn" >> /etc/iproute2/rt_tables

# Add routes to that table
ip route add default via 10.100.0.1 dev eth1 table vpn

# Use marked packets with vpn table
ip rule add fwmark 100 table vpn

Kubernetes-Specific: DaemonSet for Node Config

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: rp-filter-config
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: rp-filter-config
  template:
    metadata:
      labels:
        app: rp-filter-config
    spec:
      hostNetwork: true
      hostPID: true
      initContainers:
      - name: set-rp-filter
        image: busybox:1.36
        command:
        - sh
        - -c
        - |
          echo "Setting rp_filter to loose mode..."
          echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter
          echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
          for iface in /proc/sys/net/ipv4/conf/*/rp_filter; do
            echo 2 > "$iface" 2>/dev/null || true
          done
          echo "Done. Current settings:"
          cat /proc/sys/net/ipv4/conf/*/rp_filter
        securityContext:
          privileged: true
      containers:
      - name: pause
        image: gcr.io/google_containers/pause:3.9
      tolerations:
      - operator: Exists

Reproduction Lab

Using kind with Asymmetric Network

# Create kind cluster
cat > kind-config.yaml << 'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
EOF

kind create cluster --config kind-config.yaml

# Get worker node container
WORKER=$(docker ps --filter "name=kind-worker" -q)

# Create external network simulating VPN/on-prem
docker network create --subnet=10.200.0.0/24 external-net

# Connect worker to external network
docker network connect external-net $WORKER

# Create "on-prem" server
docker run -d --name onprem-server \
  --network external-net \
  --ip 10.200.0.50 \
  nginx:alpine

# Now the fun part: create asymmetric routing
# On-prem server will respond via a different path

Simpler: Docker-Compose Lab

# docker-compose.yml
version: '3.8'

services:
  server:
    image: nginx:alpine
    container_name: server
    networks:
      primary:
        ipv4_address: 192.168.100.10
      secondary:
        ipv4_address: 10.200.0.10
    sysctls:
      - net.ipv4.conf.all.rp_filter=1  # Strict mode

  client:
    image: alpine:3.18
    container_name: client
    command: sleep infinity
    networks:
      primary:
        ipv4_address: 192.168.100.20
    depends_on:
      - server
      - router

  router:
    image: alpine:3.18
    container_name: router
    command: sh -c "apk add iptables && sysctl -w net.ipv4.ip_forward=1 && sleep infinity"
    cap_add:
      - NET_ADMIN
    networks:
      primary:
        ipv4_address: 192.168.100.1
      secondary:
        ipv4_address: 10.200.0.1

networks:
  primary:
    driver: bridge
    ipam:
      config:
        - subnet: 192.168.100.0/24
  secondary:
    driver: bridge
    ipam:
      config:
        - subnet: 10.200.0.0/24

Reproduce the Issue

docker compose up -d

# Configure asymmetric routing
# Client sends to server via router, but server responds directly

# On server: route to client via router
docker exec server ip route add 192.168.100.20 via 10.200.0.1

# On client: add route to server's secondary IP via router
docker exec client ip route add 10.200.0.10 via 192.168.100.1

# Test from client to server's secondary IP
docker exec client sh -c "apk add curl && curl -v --connect-timeout 5 http://10.200.0.10"
# Timeout! But...

# tcpdump on server shows packets arriving
docker exec server sh -c "apk add tcpdump && tcpdump -i eth1 -n" &
# SYN packets visible!

# Check martian logs
docker exec server sh -c "echo 1 > /proc/sys/net/ipv4/conf/all/log_martians"
docker exec server dmesg | tail
# "martian source" messages

# Fix it
docker exec server sh -c "echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter"

# Retry
docker exec client curl -v http://10.200.0.10
# Works!

Monitoring

Prometheus Alerts

groups:
- name: rp-filter-drops
  rules:
  - alert: MartianPacketsDetected
    expr: |
      rate(node_netstat_Ip_InAddrErrors[5m]) > 0
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Possible rp_filter drops on {{ $labels.instance }}"
      description: "InAddrErrors increasing - check for asymmetric routing or rp_filter issues"

  - alert: HighMartianPacketRate
    expr: |
      rate(node_netstat_Ip_InAddrErrors[5m]) > 100
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "High rate of dropped packets on {{ $labels.instance }}"
      description: "{{ $value }} packets/sec being dropped - likely rp_filter with asymmetric routing"

Dashboard Query

# Martian/rp_filter drops over time
rate(node_netstat_Ip_InAddrErrors[5m])

# By node
sum by (instance) (rate(node_netstat_Ip_InAddrErrors[5m]))

Node Health Check Script

#!/bin/bash
# rp-filter-check.sh

echo "=== rp_filter Configuration ==="
for iface in /proc/sys/net/ipv4/conf/*/rp_filter; do
  echo "$(dirname $iface | xargs basename): $(cat $iface)"
done

echo ""
echo "=== InAddrErrors (martian packets) ==="
cat /proc/net/snmp | grep -E "^Ip:" | head -2

echo ""
echo "=== Recent Martian Logs ==="
dmesg | grep -i martian | tail -10

echo ""
echo "=== Route Asymmetry Check ==="
echo "Interfaces with routes:"
ip route show | awk '{print $NF}' | sort -u

Checklist

## rp_filter Debugging Checklist

### Detection
- [ ] tcpdump shows packets arriving that app doesn't see
- [ ] conntrack/iptables show no entries for the flow
- [ ] Enable log_martians and check dmesg
- [ ] Check InAddrErrors counter with nstat

### Root Cause Analysis
- [ ] Verify route asymmetry: `ip route get <source_ip>`
- [ ] Compare ingress interface vs route output interface
- [ ] Check current rp_filter settings on all interfaces

### Fix Options
- [ ] Set rp_filter=2 (loose mode) - recommended
- [ ] Fix routing to be symmetric (policy routing)
- [ ] Disable rp_filter=0 if security allows

### Prevention
- [ ] Document network topology and routing paths
- [ ] Add rp_filter config to node provisioning
- [ ] Monitor InAddrErrors metric
- [ ] Test connectivity during network changes

Conclusion

The rp_filter trap is one of those problems that makes you question your debugging skills. You can see packets arriving in tcpdump. The firewall isn’t blocking them. The application is listening on the right port. But the connection times out anyway. The packets are being dropped in a place most network debugging tools don’t show—between the network interface and iptables/conntrack processing.

The key insight is understanding that Linux’s reverse path filtering is a security feature that happens very early in packet processing, before most debugging hooks. When packets arrive on one interface but the kernel’s route to the source address points to a different interface, strict mode (rp_filter=1) drops the packet as potentially spoofed. This makes sense for simple networks but breaks multi-homed setups where asymmetric routing is normal.

The fix is usually straightforward: switch to loose mode (rp_filter=2), which only requires that the source address be routable via some interface, not the same one the packet arrived on. This maintains basic anti-spoofing protection while allowing legitimate asymmetric routing.

Key principles:

1. tcpdump captures happen before rp_filter—seeing packets in tcpdump doesn’t mean they’ll reach the application
2. conntrack/iptables miss the drops—these happen before connection tracking, so no entries are created
3. Check martian logs—enable log_martians and watch dmesg for the smoking gun
4. rp_filter=2 (loose mode) is usually safe—still provides anti-spoofing while allowing asymmetric routes
5. Multi-homed = asymmetric routing risk—VPNs, dual NICs, BGP peering all create asymmetric paths

The next time your packets arrive but your application sees nothing, check cat /proc/sys/net/ipv4/conf/*/rp_filter. The silent drop might already be happening.

Related Articles

• PMTU Blackholes in Overlay Networks - Another invisible packet drop scenario
• Kubernetes conntrack Table Exhaustion - Connection tracking edge cases