Enabling eBPF CO-RE on Older Kernels: A Guide to BTF Configuration

eBPF’s Compile Once - Run Everywhere (CO-RE) paradigm is a game-changer for portability, allowing eBPF programs to adapt to different kernel versions without on-host recompilation. This magic heavily relies on BPF Type Format (BTF), a metadata format describing kernel and eBPF program types. Modern Linux kernels (typically 5.2+) often provide embedded BTF, but deploying eBPF applications on older kernels, or even newer ones where BTF support wasn’t enabled by the distribution, presents a significant challenge.

This article provides a definitive guide to understanding and overcoming these BTF limitations, enabling CO-RE capabilities for your eBPF applications on a wider range of Linux systems. We’ll cover how to leverage external BTF sources like BTFHub, minimize BTF file sizes, and integrate this into your eBPF loading logic using libbpf and cilium/ebpf.

The Crux of the Problem: CO-RE Needs Kernel BTF

eBPF CO-RE works by embedding BTF information within the compiled eBPF object file. This “program BTF” describes the types the eBPF program expects. At load time, libbpf (or equivalent libraries like cilium/ebpf) compares this program BTF with the “kernel BTF” of the target host. If discrepancies are found (e.g., a struct field offset has changed between kernel versions), libbpf performs runtime relocations, adjusting the eBPF bytecode to correctly access kernel data structures.

The problem arises when the target kernel doesn’t provide its own BTF. This typically happens in:

• Kernels older than Linux 5.2: These versions predate the CONFIG_DEBUG_INFO_BTF Kconfig option necessary for generating embedded BTF.
• Kernels (even 5.2+) where CONFIG_DEBUG_INFO_BTF=y was not set: Some distributions didn’t enable this by default, often due to older build toolchains (e.g., pahole version < v1.16).

Without kernel BTF, libbpf has no reference to understand the target kernel’s actual data structure layouts, and CO-RE relocations cannot be performed, leading to load failures or runtime errors. The standard location for kernel-provided BTF is /sys/kernel/btf/vmlinux.

You can quickly check for native BTF support:

1

ls -l /sys/kernel/btf/vmlinux

If this file is missing, you’ll likely need an alternative BTF solution for CO-RE.

Solution 1: Leveraging External BTF with BTFHub

The most robust solution for kernels lacking native BTF is to provide it externally. BTFHub is a community project that collects and archives BTF files for a vast number of Linux distributions and kernel versions.

• BTFHub: https://github.com/aquasecurity/btfhub
• BTFHub-Archive (where BTF files reside): https://github.com/aquasecurity/btfhub-archive

The workflow involves:

1. Identifying the target kernel: Determine the distribution, version, architecture, and kernel release string (e.g., uname -r).
2. Fetching the corresponding BTF file: Download the appropriate .btf file from the BTFHub-Archive. These are often compressed (e.g., .btf.tar.xz).
3. Providing it to the eBPF loader: Instruct libbpf or cilium/ebpf to use this external BTF file.

Using External BTF with libbpf (C/C++)

libbpf allows specifying a custom path for kernel BTF via bpf_object_open_opts:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

#include <bpf/libbpf.h>
#include <stdio.h>
#include <string.h> // For strerror
#include <errno.h>  // For errno

// Function to dynamically determine the path to the custom BTF file
// based on kernel version, etc. This is a placeholder for your logic.
const char* get_custom_btf_path_for_kernel() {
    // In a real application, you would:
    // 1. Get kernel release (e.g., from uname()).
    // 2. Map this to a BTF file path you've bundled or downloaded from BTFHub.
    // Example: return "/opt/my-app/btf-cache/5.4.0-100-generic.btf";
    return "/path/to/your/downloaded/kernel.btf"; // Replace with actual logic
}

int main(int argc, char **argv) {
    struct bpf_object_open_opts opts = {};
    struct bpf_object *obj;
    const char *bpf_program_path = "my_bpf_program.o"; // Your eBPF object file
    const char *custom_btf_path;

    // Check for native BTF first (conceptual)
    // if (access("/sys/kernel/btf/vmlinux", F_OK) == -1) {
    custom_btf_path = get_custom_btf_path_for_kernel();
    if (custom_btf_path) {
        opts.btf_custom_path = custom_btf_path;
        fprintf(stdout, "Using custom BTF path: %s\n", custom_btf_path);
    } else {
        fprintf(stderr, "Custom BTF path not found, proceeding without it.\n");
    }
    // }

    opts.sz = sizeof(opts); // Important to set the size of the opts struct

    obj = bpf_object__open_file(bpf_program_path, &opts);
    if (!obj) {
        fprintf(stderr, "ERROR: failed to open BPF object '%s': %s\n",
                bpf_program_path, strerror(-errno));
        return 1;
    }

    // ... further BPF program loading (bpf_object__load) and attachment ...
    fprintf(stdout, "BPF object '%s' opened successfully.\n", bpf_program_path);

    bpf_object__close(obj);
    return 0;
}

Note: Ensure opts.sz is correctly initialized. The btf_custom_path option effectively overrides libbpf’s default search for /sys/kernel/btf/vmlinux.

Using External BTF with cilium/ebpf (Go)

The cilium/ebpf library provides similar functionality through CollectionOptions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

package main

import (
	"fmt"
	"log"
	"os" // For basic file existence check example

	"github.com/cilium/ebpf"
	"github.com/cilium/ebpf/btf"
)

// Placeholder for your logic to find the appropriate BTF file path
func getCustomBTFPathForKernel() (string, error) {
	// Real logic: determine kernel version, map to BTFHub file.
	// Example: return "/opt/my-app/btf-cache/5.4.0-100-generic.btf", nil
	// This example assumes the BTF file is already downloaded.
	btfPath := "/path/to/your/downloaded/kernel.btf" // Replace
	if _, err := os.Stat(btfPath); os.IsNotExist(err) {
		return "", fmt.Errorf("custom BTF file not found at %s", btfPath)
	}
	return btfPath, nil
}

func main() {
	bpfProgramPath := "my_bpf_program.o" // Your eBPF object file

	collSpec, err := ebpf.LoadCollectionSpec(bpfProgramPath)
	if err != nil {
		log.Fatalf("Failed to load BPF collection spec from '%s': %v",
			bpfProgramPath, err)
	}

	opts := ebpf.CollectionOptions{}

	// Attempt to load external BTF if native is likely missing
	// A more robust check for native BTF would be to try loading without
	// KernelTypes first and see if it fails with a BTF-related error.
	if _, err := os.Stat("/sys/kernel/btf/vmlinux"); os.IsNotExist(err) {
		log.Println("Native kernel BTF not found, attempting to use custom BTF.")
		customBTFPath, err := getCustomBTFPathForKernel()
		if err != nil {
			log.Printf("Could not get custom BTF path: %v. "+
				"Proceeding without external BTF.", err)
		} else {
			kernelBTFSpec, err := btf.LoadSpec(customBTFPath)
			if err != nil {
				log.Printf("Failed to load custom BTF spec from '%s': %v. "+
					"Proceeding without external BTF.", customBTFPath, err)
			} else {
				opts.KernelTypes = kernelBTFSpec
				log.Printf("Successfully loaded custom BTF from '%s'", customBTFPath)
			}
		}
	} else {
		log.Println("Native kernel BTF found, will attempt to use it.")
	}


	coll, err := ebpf.NewCollectionWithOptions(collSpec, opts)
	if err != nil {
		log.Fatalf("Failed to create BPF collection: %v", err)
	}
	defer coll.Close()

	// ... attach programs, interact with maps ...
	log.Println("BPF collection created and programs loaded successfully.")
}

This approach allows your Go eBPF application to bundle or fetch BTF files and supply them at runtime.

Solution 2: Minimizing BTF with bpftool gen min_core_btf

Full kernel BTF files from BTFHub can be several megabytes. Shipping many such files with an application is often impractical. The solution is to generate a minimal BTF file containing only the types relevant to your specific eBPF program.

bpftool (a standard eBPF utility) provides the gen min_core_btf command for this purpose (this functionality was historically part of btfgen):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# You need the full BTF for the target kernel (e.g., from BTFHub)
FULL_KERNEL_BTF_PATH="/path/to/btfhub-archive/ubuntu/20.04/x86_64/5.4.0-100-generic.btf"
# Your compiled eBPF object file
EBPF_OBJECT_PATH="my_bpf_program.o"
# Output path for the minimal BTF
MINIMAL_BTF_PATH="my_bpf_program.min.btf"

bpftool gen min_core_btf "$FULL_KERNEL_BTF_PATH" "$MINIMAL_BTF_PATH" "$EBPF_OBJECT_PATH"

# Now, MINIMAL_BTF_PATH can be used as the btf_custom_path or KernelTypes source
# Its size will be significantly smaller.

This command analyzes my_bpf_program.o, identifies all kernel types it references via CO-RE, and extracts only those type definitions from the FULL_KERNEL_BTF_PATH, writing them to MINIMAL_BTF_PATH.

This process is typically integrated into the eBPF application’s build system:

1. For each target kernel (or range of kernels) you want to support:
   • Obtain its full BTF from BTFHub.
   • Run bpftool gen min_core_btf against your eBPF object(s).
2. Bundle these minimal BTF files with your application.
3. At runtime, select the appropriate minimal BTF based on the detected kernel and provide it to the loader library.

This dramatically reduces the storage overhead for supporting multiple kernel versions.

Diagnostic Steps and Considerations

1. Kernel Version and Configuration:

   • Always start by checking the kernel version: uname -r.
   • Check for native BTF: ls -l /sys/kernel/btf/vmlinux.
   • If possible, check the kernel’s build config: grep CONFIG_DEBUG_INFO_BTF /boot/config-$(uname -r) (This file might not always be available or accurate for custom kernels).

2. Loader Library Verbosity:

   • libbpf: Use libbpf_set_print(LIBBPF_PRINT_KERNEL) or LIBBPF_PRINT_DEBUG to get detailed output from libbpf during the object opening and loading process. This will show which BTF files it’s trying to use and any errors encountered.

     1 2	// Example: Set libbpf print level libbpf_set_print(LIBBPF_PRINT_DEBUG); // Or LIBBPF_PRINT_KERNEL

   • cilium/ebpf: Program loading options can include ebpf.ProgramOptions{ LogLevel: ebpf.LogLevelInstruction } to get verifier logs.

3. bpftool feature probe: This command can provide information about BPF features supported by the currently running kernel, although it doesn’t directly confirm BTF availability for arbitrary types.

4. BTF File Accuracy: Ensure the BTF file (full or minimal) precisely matches the target kernel’s architecture and exact version. Even minor patch differences could alter structures, though BTFHub aims for compatibility where possible.

5. Build System Complexity: Integrating BTFHub downloads and bpftool gen min_core_btf into a build pipeline adds complexity but is essential for robust CO-RE on older systems. Projects like Eunomia-bpf’s bpf-compatible tool showcase packaging eBPF programs with necessary BTF.

6. Limitations: Even with external BTF, some eBPF program types or features that intrinsically require newer kernel APIs (beyond just type information) might not function correctly on very old kernels.

Conclusion: Expanding eBPF’s Reach

While native BTF support is becoming more common, the ability to provide kernel BTF externally via BTFHub and tools like bpftool gen min_core_btf is crucial for maximizing the portability of CO-RE eBPF applications. By understanding these mechanisms and integrating them into your development and deployment workflows, you can confidently target a broader spectrum of Linux kernels, including older ones that don’t ship with BTF.

This approach, while adding some build-time overhead, empowers developers to deliver powerful eBPF solutions that truly “Compile Once - Run Everywhere,” bridging the gap left by varying kernel configurations and ensuring wider applicability of modern eBPF observability and security tools.