Network Observability with eBPF

Network observability with eBPF enables real-time, low-overhead visibility into network traffic, packet processing, and application behavior across the full stack—from the kernel to the user space—without requiring code instrumentation or agent deployments. It matters most when diagnosing latency, packet loss, connection failures, and service-to-service communication in distributed systems, especially at scale, where traditional tools like tcpdump, netstat, and Prometheus metrics fall short.

Instrumenting Network Traffic with eBPF

To observe network traffic end-to-end, attach eBPF programs to network events: packet arrival, socket creation, and TCP state transitions. The primary tool is bpftool, which manages eBPF programs, maps, and links.

bpftool prog load my_trace_kern.o /sys/fs/bpf/trace_pkt

# Attach the program to a network interface
bpftool map create /sys/fs/bpf/conn_map 16 64 4 0 0
bpftool prog attach \
  --map-id $(bpftool map id /sys/fs/bpf/conn_map) \
  --target-type dev \
  --target-name eth0 \
  --prog-id $(bpftool prog id /sys/fs/bpf/trace_pkt)

The eBPF program, written in C, runs in the kernel and captures packet headers and metadata:

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <linux/if_packet.h>

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 1024);
    __type(key, __u32);
    __type(value, struct {
        __u64 rx_bytes;
        __u64 rx_packets;
        __u64 last_seen;
    });
} conn_map SEC("maps");

SEC("xdp")
int xdp_trace_pkt(struct xdp_md *ctx) {
    void *data = (void *)(long)ctx->data;
    void *data_end = (void *)(long)ctx->data_end;
    struct ethhdr *eth = data;

    if (data + sizeof(*eth) > data_end)
        return XDP_DROP;

    if (eth->h_proto == htons(ETH_P_IP)) {
        struct iphdr *ip = data + sizeof(*eth);
        if (data + sizeof(*eth) + sizeof(*ip) > data_end)
            return XDP_DROP;

        __u32 src = ip->saddr;
        __u32 dst = ip->daddr;
        __u32 port = 0;

        if (ip->protocol == IPPROTO_TCP) {
            struct tcphdr *tcp = data + sizeof(*eth) + sizeof(*ip);
            if (data + sizeof(*eth) + sizeof(*ip) + sizeof(*tcp) > data_end)
                return XDP_DROP;

            port = ntohs(tcp->dest);
        }

        struct conn_key key = { .src = src, .dst = dst, .port = port };
        struct conn_value *val = bpf_map_lookup_elem(&conn_map, &key);
        if (val) {
            val->rx_bytes += ctx->data_end - ctx->data;
            val->rx_packets++;
            val->last_seen = bpf_ktime_get_ns();
        } else {
            struct conn_value new_val = {
                .rx_bytes = ctx->data_end - ctx->data,
                .rx_packets = 1,
                .last_seen = bpf_ktime_get_ns()
            };
            bpf_map_update_elem(&conn_map, &key, &new_val, BPF_ANY);
        }
    }

    return XDP_PASS;
}

This program runs on every incoming packet and updates a map with per-connection statistics. It captures the full path from layer 2 to layer 4 with zero-copy data access.

Tracing TCP State Transitions

To trace TCP connection lifecycle events—SYN, SYN-ACK, ACK, FIN—use bpftrace, a high-level eBPF tracer.

# Trace TCP connection events from a specific pod
sudo bpftrace -e '
tracepoint:tcp:tcp_connect
{
    printf("TCP connect: %s -> %s\n", comm, args->peername);
}

tracepoint:tcp:tcp_state_change
{
    printf("TCP state change: %s: %s -> %s\n",
           comm,
           tcp_state_str(args->oldstate),
           tcp_state_str(args->newstate));
}
' -c 'podman run -it --network=host alpine sh'

The tcp_state_change tracepoint fires on every state transition. The tcp_state_str function is a built-in helper in bpftrace that maps numeric states (e.g., 1 = TCP_ESTABLISHED) to human-readable strings.

This setup reveals subtle issues: a client sending SYN but never receiving SYN-ACK, or a server closing connections without FIN, indicating a misconfigured keepalive.

Capturing Latency Across Microservices

To measure end-to-end latency between services, use perf and bpftrace together, leveraging bpftrace’s ability to create dynamic tracepoints from user-space functions.

# Trace function calls in a gRPC service and measure duration
sudo bpftrace -e '
tracepoint:syscalls:sys_enter_write
{
    @start[pid] = nsecs;
}

tracepoint:syscalls:sys_exit_write
{
    $start = @start[pid];
    if ($start) {
        @latency[comm] = hist(nsecs - $start);
        delete(@start[pid]);
    }
}

tracepoint:syscalls:sys_enter_read
{
    @start[pid] = nsecs;
}

tracepoint:syscalls:sys_exit_read
{
    $start = @start[pid];
    if ($start) {
        @latency[comm] = hist(nsecs - $start);
        delete(@start[pid]);
    }
}

// Trace gRPC call entry and exit
kprobe:grpc_call_start
{
    printf("gRPC call: %s\n", str(args->method));
    @call_start[pid] = nsecs;
}

kprobe:grpc_call_done
{
    $start = @call_start[pid];
    if ($start) {
        @call_latency[comm] = hist(nsecs - $start);
        delete(@call_start[pid]);
    }
}
' -c 'grpc_server --port=50051'

The @latency map accumulates per-process and per-function latency distributions. The hist() function produces a histogram of latency values, which can be exported to JSON or visualized in real time.

This approach captures the full stack: kernel I/O, system calls, gRPC framing, and service processing, all with sub-microsecond resolution.

Observing Application Behavior at the Network Layer

eBPF can observe how applications use network APIs. The bpftrace tool excels here with uprobe and kprobe support.

Monitoring Socket Operations

To trace all sendto and recvfrom calls across a cluster, deploy a shared eBPF program via bpftool.

# Attach a user-space probe to sendto in all processes
bpftool prog load sendto_trace.o /sys/fs/bpf/sendto

# Create a map to store per-process statistics
bpftool map create /sys/fs/bpf/sendto_map 16 1024 8 0 0

# Attach the program to all sendto calls
bpftool uprobe attach \
  --map-id $(bpftool map id /sys/fs/bpf/sendto_map) \
  --prog-id $(bpftool prog id /sys/fs/bpf/sendto) \
  --binary-path /lib/x86_64-linux-gnu/libc.so.6 \
  --symbol-name sendto \
  --name sendto_trace

The program in sendto_trace.o runs in user space and captures the size, destination, and timestamp of each sendto call.

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 1024);
    __type(key, __u32);
    __type(value, struct {
        __u64 total_bytes;
        __u64 calls;
        __u64 last_time;
    });
} sendto_map SEC("maps");

SEC("uprobe/sendto")
int trace_sendto(struct pt_regs *ctx) {
    __u32 pid = bpf_get_current_pid_tgid() >> 32;
    __u32 len = (long)PT_REGS_PARM2(ctx);
    __u32 dest = (long)PT_REGS_PARM3(ctx);

    struct sendto_value *val = bpf_map_lookup_elem(&sendto_map, &pid);
    if (val) {
        val->total_bytes += len;
        val->calls++;
        val->last_time = bpf_ktime_get_ns();
    } else {
        struct sendto_value new_val = {
            .total_bytes = len,
            .calls = 1,
            .last_time = bpf_ktime_get_ns()
        };
        bpf_map_update_elem(&sendto_map, &pid, &new_val, BPF_ANY);
    }

    return 0;
}

This program runs on every sendto call, even in dynamically loaded libraries, and aggregates statistics per process.

Diagnosing Network Performance with eBPF

When a service experiences high latency or dropped packets, use bpftrace to correlate events across time.

# Trace packet arrival and TCP processing
sudo bpftrace -e '
tracepoint:net:netif_receive_skb
{
    printf("Packet received: %d bytes from %s\n",
           args->len,
           args->dev->name);
}

tracepoint:tcp:tcp_rcv_established
{
    printf("TCP packet received: %s (%d bytes)\n",
           args->sk->sk_state == TCP_ESTABLISHED ? "ESTABLISHED" : "CLOSED",
           args->len);
}

tracepoint:tcp:tcp_ack_sent
{
    printf("TCP ACK sent: %d bytes, window=%d\n",
           args->len, args->window);
}

// Trace system call latency for network-related calls
kprobe:sys_connect
{
    @connect_start[pid] = nsecs;
}

kprobe:sys_sendto
{
    @send_start[pid] = nsecs;
}

kprobe:sys_recvfrom
{
    @recv_start[pid] = nsecs;
}

// Output per-process latency histograms
profile:ms:1000
{
    printf("Connect latency: %s\n", comm);
    @connect_latency = hist(nsecs - @connect_start[pid]);
    delete(@connect_start[pid]);
}

profile:ms:1000
{
    printf("Send latency: %s\n", comm);
    @send_latency = hist(nsecs - @send_start[pid]);
    delete(@send_start[pid]);
}

profile:ms:1000
{
    printf("Recv latency: %s\n", comm);
    @recv_latency = hist(nsecs - @recv_start[pid]);
    delete(@recv_start[pid]);
}
' -c 'python3 app.py'

This script captures the full flow: packet arrival, TCP state changes, and system call latencies, all in a single command.

Failure Modes and Sharp Edges

eBPF programs silently fail in production when the kernel version changes or when the program is miscompiled. The most common failure is the BPF_MAP_TYPE_PERCPU_HASH map not being supported in older kernels.

# Check if a BPF map type is supported
bpftool map show --json /sys/fs/bpf/conn_map

Output:

{
  "id": 42,
  "type": "hash",
  "key_size": 4,
  "value_size": 24,
  "map_flags": 0,
  "numa_node": 0,
  "name": "conn_map",
  "map_flags": 0
}

If the map_flags field contains BPF_F_NUMA_NODE, the map is not supported on the current kernel.

Another edge case: bpftrace relies on perf events, and when perf is not enabled, tracepoints are missed. The error perf: Failed to open perf event appears in dmesg when the perf subsystem is not ready.

eBPF programs can also leak memory. The bpf_map_update_elem function uses BPF_ANY, which updates or inserts. However, BPF_REPLACE only updates existing entries, and BPF_ADD only adds new ones. Confusing BPF_ANY and BPF_REPLACE leads to incorrect state aggregation.

When attaching eBPF programs to multiple targets (e.g., multiple network interfaces), the program may run on the wrong interface due to a mismatch in device name or index. Use bpftool link show to verify attachment:

bpftool link show

Output:

2: xdp  ifindex 2  prog 42  map 43  name trace_pkt
3: tc   ifindex 3  prog 42  map 43  name trace_pkt

If a program is attached to eth0 but appears as ifindex 2, the interface might have been renamed or reconfigured.

Finally, eBPF programs can be compiled with different ABI requirements. When a program expects __u64 but the kernel uses __u32 for struct sockaddr, the bpf_trace_printk output shows mismatched data.

bpf_trace_printk("src=%u, dst=%u, port=%u\n", src, dst, port);

This prints src=192.168.1.1, dst=10.0.0.1, port=80 but with port truncated to 32 bits, even when it was passed as 16 bits.

Summary

Network observability with eBPF is not a single tool but a coordinated set of practices using bpftool, bpftrace, perf, and standard C eBPF programs. It enables full-stack visibility without requiring application instrumentation. The key to success is understanding the interaction between kernel and user-space, the semantics of map operations, and the subtle failure modes that arise from kernel versioning, memory layout, and event timing.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.