eBPF engineers build and maintain Linux kernel extension programs using the extended Berkeley Packet Filter runtime — writing programs in C or Rust that the kernel verifier safely executes in privileged contexts (network interfaces, system call hooks, scheduler probes) without requiring kernel module development or kernel recompilation, attaching them to kprobes, tracepoints, XDP hooks, and socket operations to implement custom packet processing, performance profiling, security enforcement, and observability with nanosecond-precision that user-space tools cannot achieve. At remote-first technology companies, they serve as the systems and infrastructure engineers who build the low-level instrumentation layer that powers the observability platforms, cloud native network plugins, and runtime security tools that the entire infrastructure stack depends on — creating the tracing programs behind Datadog's eBPF agent, the packet processing behind Cilium, and the system call monitoring behind Falco and Tetragon.
What eBPF engineers do
eBPF engineers write eBPF programs — authoring kernel-space C programs with the __attribute__((section("xdp"))) or SEC("kprobe/sys_connect") annotations that tell the BPF loader which hook point to attach to, reading packet data with void *data = (void *)(long)ctx->data; void *data_end = (void *)(long)ctx->data_end with mandatory bounds checking, accessing process context with bpf_get_current_pid_tgid(), bpf_get_current_comm(buf, sizeof(buf)), and bpf_get_current_task() for identity attribution; use BPF helper functions — calling bpf_map_lookup_elem, bpf_map_update_elem, bpf_map_delete_elem for map data access, bpf_perf_event_output and bpf_ringbuf_submit for streaming events to user space, bpf_probe_read_kernel and bpf_probe_read_user for safe memory reads across address space boundaries, and bpf_skb_load_bytes for packet payload reading; work with BPF map types — using BPF_MAP_TYPE_HASH for key-value lookup tables, BPF_MAP_TYPE_ARRAY for indexed arrays and per-CPU statistics, BPF_MAP_TYPE_PERF_EVENT_ARRAY and BPF_MAP_TYPE_RINGBUF for kernel-to-user event streaming, BPF_MAP_TYPE_LRU_HASH for connection tracking with automatic eviction, and BPF_MAP_TYPE_PROG_ARRAY for tail calls that chain multiple eBPF programs; write XDP programs — implementing XDP_DROP, XDP_PASS, XDP_TX, and XDP_REDIRECT actions for line-rate packet filtering and forwarding before the kernel networking stack processes packets, achieving DDoS mitigation and load balancing at 100Gbps on commodity hardware; write TC (Traffic Control) programs — attaching at tc ingress and tc egress hooks for both packet inspection and modification, enabling NAT, traffic shaping, and L7 protocol parsing on packets entering or leaving a network interface; use libbpf — linking eBPF programs with libbpf for bpf_object__open_file, bpf_object__load, bpf_program__attach_kprobe, and bpf_program__attach_xdp in the user-space loader, using CO-RE (Compile Once, Run Everywhere) with BTF (BPF Type Format) to write portable programs that adapt to different kernel struct layouts without per-kernel compilation; use BPF skeletons — generating prog.skel.h with bpftool gen skeleton prog.o for type-safe user-space control code that eliminates manual map and program name string lookups; implement tracing programs — attaching kprobe and kretprobe programs to kernel function entry and return points, tracepoint programs to stable kernel ABI trace events (sys_enter_connect, sched_process_exec), and uprobe/uretprobe for user-space function tracing without instrumentation code; use bcc and bpftrace — writing bpftrace one-liners like bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }' for interactive kernel tracing, and BCC Python tools for more complex analysis programs; and use higher-level frameworks — building on Tetragon for security enforcement, Falco for runtime threat detection, Pixie for auto-instrumentation observability, or Beyla for zero-code application metrics from eBPF probes.
Key skills for eBPF engineers
- eBPF programs: C (BPF subset); sections/SEC(); verifier constraints; stack size limits; loop limits
- Hook points: kprobe/kretprobe; tracepoints; uprobe/uretprobe; XDP; TC; socket filter; cgroup
- BPF maps: HASH/ARRAY/LRU_HASH/PERF_EVENT_ARRAY/RINGBUF/PROG_ARRAY; map operations
- Helper functions: bpf_map_*; bpf_probe_read_*; bpf_perf_event_output; bpf_ringbuf_*; bpf_skb_*
- libbpf: bpf_object__open/load/attach; CO-RE; BTF; BPF skeletons; bpftool
- XDP: XDP_DROP/PASS/TX/REDIRECT; packet parsing; bounds checking; line-rate processing
- bpftrace: one-liners; probes syntax; built-in variables; aggregations; maps
- BCC: BPF Python bindings; kprobes; tracepoints; perf buffers; BCC tools
- Observability frameworks: Tetragon; Falco; Pixie; Beyla; Cilium; Katran
- Kernel internals: kernel struct layout; BTF; kernel version feature matrix; kallsyms
Salary expectations for remote eBPF engineers
Remote eBPF engineers earn $125,000–$195,000 total compensation. Base salaries range from $105,000–$162,000, with equity at technology companies where kernel-level observability, sub-microsecond packet processing, and runtime security enforcement at the Linux system call layer directly determine infrastructure capability and competitive differentiation. eBPF engineers with production XDP programs processing terabit-scale DDoS mitigation traffic, comprehensive tracing systems that attribute every kernel event to application code with nanosecond timestamps, and bespoke eBPF security enforcement programs enforcing process allow-lists and network policy at the kernel level command the strongest premiums. This is one of the highest-ceiling engineering disciplines in systems software — engineers who combine eBPF expertise with kernel internals knowledge and distributed systems design earn at the top of engineering compensation bands.
Career progression for eBPF developers
The path from eBPF engineer leads to senior systems engineer (broader scope across the Linux kernel, network stack, and performance engineering), kernel engineer (contributing directly to eBPF subsystem development and new BPF program type proposals), or infrastructure security engineer (applying eBPF to runtime threat detection, system call auditing, and network policy enforcement at kernel speed). Some eBPF engineers specialize into network performance engineering, writing XDP and TC eBPF programs for high-frequency trading infrastructure, CDN edge packet processing, or 5G user-plane function (UPF) implementations where the difference between 1μs and 10μs latency is commercially significant. Others transition into cloud native platform engineering, building the eBPF-based CNI plugins, service meshes, and observability agents that the broader Kubernetes ecosystem consumes. eBPF engineers who contribute to the Linux kernel's BPF subsystem, bpftrace, or libbpf participate in one of the most technically demanding open-source communities where work has direct impact on every Linux system globally.
Remote work considerations for eBPF engineers
Building eBPF-based infrastructure for distributed platform and application teams requires program safety documentation, kernel version compatibility matrices, and production deployment guardrails that prevent distributed engineers from deploying eBPF programs that pass the verifier on development kernels but fail on production kernels with different BPF feature support, using bpf_probe_read without bounds checking in ways the verifier accepts on older kernels but rejects on newer stricter verifiers, or attaching kprobes to internal kernel functions that change signatures across kernel minor versions. eBPF engineers at remote companies establish the kernel feature matrix — maintaining a documented table of every eBPF feature used in production (ringbuf, CO-RE, BPF timer, TCX) against the minimum kernel version each requires — because distributed engineers who develop on Ubuntu 24.04 with kernel 6.8 and deploy to production RHEL 8 with kernel 4.18 hit verifier rejections that do not reproduce in development; enforce the CO-RE requirement — requiring that all production eBPF programs use CO-RE with BTF-based struct access via BPF_CORE_READ() macros rather than hardcoded struct offsets — because hardcoded offsets produce programs that silently read wrong memory on kernel versions where struct layouts differ, producing incorrect metrics or security bypasses; define the map sizing policy — requiring that all production BPF maps have documented maximum entry counts with overflow behavior (LRU eviction strategy for hash maps, bounded arrays) and resource limit calculations — because unbounded BPF_MAP_TYPE_HASH maps in high-throughput packet processing paths consume kernel memory that cannot be reclaimed without detaching the program; and establish the production attach discipline — requiring that all eBPF program attachments in production are managed as code (deployed via Kubernetes DaemonSet, systemd service, or Helm chart) rather than manual bpftool or tc CLI commands — because manually attached eBPF programs survive reboots unpredictably and are invisible to infrastructure change management.
Top industries hiring remote eBPF engineers
- Cloud infrastructure companies and CDN providers using XDP eBPF programs for DDoS mitigation at line rate (100Gbps+) on commodity hardware, implementing packet filtering, IP reputation lookup, and rate limiting in the kernel driver layer before any user-space processing
- Observability and monitoring companies building eBPF-based agents that automatically instrument processes (Datadog, New Relic, Elastic) — attaching uprobe programs to user-space function boundaries and tracepoint programs to kernel system calls to capture traces, metrics, and logs without requiring application code changes or language-specific agents
- Kubernetes CNI and networking companies (Cilium, Calico) where eBPF is the fundamental technology for kube-proxy replacement, transparent encryption, and L7 network policy enforcement at the kernel level
- Runtime security companies (Falco, Tetragon, Aqua, Sysdig) building eBPF-based threat detection systems that hook into system call tracepoints to detect malicious process behavior, file access patterns, and network connections with millisecond detection latency
- High-frequency trading and financial services technology companies using eBPF for kernel bypass networking, CPU scheduling optimization, and latency measurement at nanosecond precision for trading systems where microsecond advantages determine profitability
Interview preparation for eBPF engineer roles
Expect program type questions: explain the difference between a kprobe program and an XDP program — what kernel hook point each attaches to, what context struct they receive, and what actions they can take. BPF map questions ask how you'd implement a connection tracking table that associates client IP:port pairs with connection state and automatically evicts stale entries — what BPF_MAP_TYPE_LRU_HASH with a struct key and struct value looks like. Verifier questions ask why a perfectly valid-looking memory access fails the BPF verifier — what bounds checking with if (ptr + sizeof(struct) > data_end) return XDP_DROP looks like and why the verifier requires it. CO-RE questions ask how you'd read a field from a kernel struct that has different field offsets in different kernel versions — what BPF_CORE_READ(task, mm, total_vm) provides. Perf event questions ask how you'd stream events from an eBPF program to user space with minimal overhead — what BPF_MAP_TYPE_RINGBUF provides over BPF_MAP_TYPE_PERF_EVENT_ARRAY. Security questions ask how you'd detect when a process opens a sensitive file like /etc/shadow — what a tracepoint:syscalls:sys_enter_openat program that checks args->filename and writes to a perf buffer looks like.
Tools and technologies for eBPF engineers
Core: Linux kernel eBPF subsystem (4.x+); BPF instruction set; BPF verifier; BTF (BPF Type Format); CO-RE. Low-level: clang/LLVM (BPF target); libbpf (bpf_object__open/load/attach); bpftool (prog/map/gen skeleton); linux/bpf.h; vmlinux.h (BTF headers). BPF helpers: bpf_map_lookup/update/delete_elem; bpf_probe_read_kernel/user; bpf_perf_event_output; bpf_ringbuf_reserve/submit; bpf_get_current_pid_tgid; bpf_get_current_comm; bpf_ktime_get_ns; bpf_skb_load_bytes. Map types: BPF_MAP_TYPE_HASH; ARRAY; LRU_HASH; PERCPU_HASH; PERCPU_ARRAY; PERF_EVENT_ARRAY; RINGBUF; PROG_ARRAY; SOCKHASH; SOCKMAP; DEVMAP; CPUMAP. Hook points: kprobe/kretprobe; tracepoint (sys_enter_*); raw_tracepoint; uprobe/uretprobe; XDP; TC (cls_bpf); cgroup/skb; socket_filter; sk_msg; lsm (BPF LSM). XDP: XDP_DROP/PASS/TX/REDIRECT; xdp_md; bpf_xdp_redirect; bpf_fib_lookup; XDP offload. TC: tc ingress/egress; __sk_buff; cls_bpf; bpf_skb_change_head; bpf_redirect; bpf_clone_redirect. High-level tools: bpftrace (scripting language, probes, maps); BCC (Python bindings, bcc-tools); ply. Frameworks: Cilium (CNI, network policy); Falco (security); Tetragon (security enforcement); Pixie (auto-instrumentation); Beyla (application metrics); Katran (load balancer). Language bindings: libbpf-rs (Rust); ebpf-go (Go, Cilium); aya (Rust, eBPF in Rust). Kernel internals: kallsyms; /sys/kernel/debug/tracing; BTF /sys/kernel/btf/vmlinux; bpf() syscall; BPF_PROG_LOAD. Alternatives: kernel modules (more powerful, less safe, requires kernel dev env); SystemTap (older, kernel module based); DTrace (Solaris/macOS origin, limited Linux support); perf (profiling, no programmability).
Global remote opportunities for eBPF engineers
eBPF engineer expertise is in strong and sustained global demand, with eBPF's position as the most transformative Linux kernel innovation in a decade — now powering the default networking in Google Kubernetes Engine and Amazon EKS, used by Cloudflare for global DDoS mitigation at terabit scale, and forming the foundation of the cloud native observability and security tool ecosystem (Cilium, Falco, Tetragon, Pixie, Beyla) — creating strong demand for the relatively small global population of engineers who combine eBPF program development with kernel internals knowledge. US-based eBPF engineers are in extremely high demand at cloud infrastructure companies, observability platform vendors, and financial services technology firms where eBPF expertise is rare and commands premium compensation. EMEA-based eBPF engineers are well-positioned given eBPF's strong European engineering community — the eBPF Summit and BPFConf draw significant European participation, Isovalent (the primary Cilium company, now part of Cisco) was founded in Europe, and European cloud infrastructure companies increasingly seek eBPF expertise for networking and security tooling. eBPF's continued expansion — BPF LSM for security policy, BPF timer for scheduled kernel programs, TCX for ordered TC attachment, and BPF token for unprivileged deployment — ensures sustained and growing demand as every Linux infrastructure layer acquires an eBPF interface.
Frequently asked questions
What makes eBPF safe to run in the kernel and how does the verifier work? The BPF verifier is a static analyzer that runs on every eBPF program before it executes in the kernel. It performs two passes: first, it builds a control flow graph and verifies the program has no unreachable instructions, no loops that might run forever (or bounds-checks bounded loops in kernels 5.3+), and that it terminates. Second, it performs abstract interpretation — tracking the type and range of every register at every program point — to verify that every memory access is within bounds, every pointer dereference comes from a valid kernel object (not an arbitrary integer), every BPF helper call receives arguments of the correct type, and the program returns an acceptable return code. Programs that fail verification are rejected with a human-readable error message. This makes eBPF fundamentally safer than kernel modules: a buggy eBPF program fails to load rather than panicking the kernel. The trade-off is constraints: eBPF programs have a maximum instruction count (1 million in modern kernels), stack size limit (512 bytes), and restricted access to kernel data structures — which is why CO-RE and BPF_CORE_READ exist to safely access kernel structs across versions.
What is the difference between kprobe, tracepoint, and XDP attachment points? kprobes attach to arbitrary kernel function entry/exit points by function name — powerful because you can probe any kernel function, but fragile because kernel function names, signatures, and existence change between kernel versions. Use kprobes for instrumenting specific kernel subsystems during development or when no tracepoint exists. Tracepoints are stable, kernel-maintained probe points with documented, stable argument structures — tracepoint:syscalls:sys_enter_openat is guaranteed to exist with the same arguments across kernel versions. Prefer tracepoints over kprobes for production programs whenever a tracepoint covers the needed event. XDP (eXpress Data Path) attaches to the network driver's receive path, before the kernel's networking stack processes the packet — the program runs in the network driver interrupt context, which makes it the fastest possible packet processing point (nanoseconds per packet at line rate) but restricts what kernel helpers are available (no socket access, no process context). Use XDP for packet filtering, DDoS mitigation, and load balancing where performance at the driver level matters. TC (Traffic Control) hooks attach after the kernel networking stack but before/after the network interface queue, giving access to more context (socket state, netns) at slightly higher overhead than XDP.
How does CO-RE (Compile Once, Run Everywhere) work and why is it essential for production eBPF programs? Without CO-RE, an eBPF program accessing task_struct->mm->total_vm compiles to a hardcoded byte offset into the struct — say, offset 0x3f8. If the kernel version on the target machine has a different task_struct layout (due to CONFIG_xxx options or kernel version changes), the program reads garbage from the wrong offset, silently producing incorrect data. With CO-RE: the program uses BPF_CORE_READ(task, mm, total_vm) which compiles to a BPF relocation instruction that references the BTF type name and field name. When the eBPF loader (libbpf) loads the program, it reads the target kernel's BTF from /sys/kernel/btf/vmlinux to find the actual runtime offset of task_struct->mm and mm_struct->total_vm and patches the relocation before loading. This means one compiled eBPF program works across all kernel versions that have those struct fields, without per-kernel compilation. CO-RE requires BTF support in both the eBPF program (compiled with -g debug info preserved in BPF ELF) and the target kernel (kernel 5.4+ with CONFIG_DEBUG_INFO_BTF=y, or paravirtualized via BTF overlay for older kernels).