adllm Insights logo adllm Insights logo

Resolving EPERM: cap_net_bind_service, Systemd, and Go on Privileged Ports

Published on by The adllm Team. Last modified: . Tags: golang systemd linux capabilities eperm networking devops

Software developers frequently encounter scenarios where Go applications need to bind to privileged network ports (those below 1024), such as port 80 for HTTP or 443 for HTTPS. The standard Linux mechanism for granting this permission without full root privileges is the CAP_NET_BIND_SERVICE capability. However, integrating this with systemd-managed services can lead to the dreaded EPERM (Operation not permitted) error, often due to subtle misconfigurations in how capabilities, especially ambient capabilities, are handled.

This article provides a comprehensive guide to understanding and resolving these EPERM issues. We’ll explore Linux capabilities, systemd’s role in privilege management, and how to correctly configure your Go application’s systemd service unit to use CAP_NET_BIND_SERVICE effectively, focusing on the modern ambient capabilities model.

Understanding the Core Components

Successfully binding to privileged ports involves the interplay of Linux capabilities, your systemd service definition, and your Go application’s behavior.

Linux Capabilities: The “Why” and “What”

Linux capabilities capabilities(7) break down the all-powerful root privileges into smaller, distinct units. This allows processes to be granted only the specific permissions they need, adhering to the principle of least privilege.

  • CAP_NET_BIND_SERVICE: This is the star of our show. It specifically grants a process the permission to bind to TCP/UDP sockets on privileged ports (0-1023).

Each process possesses several capability sets:

  • Permitted (P): The full set of capabilities the process may use.
  • Effective (E): The capabilities currently in effect for permission checks by the kernel.
  • Inheritable (I): Capabilities that can be passed to a new program during an execve(2) call.
  • Bounding (B): A limiting superset. Once a capability is dropped from this set, it cannot be regained by the process or its children.
  • Ambient (A): Introduced in Linux kernel 4.3+, this set allows capabilities to be inherited by child processes across execve(2) calls, even for non-privileged executables (those without file capabilities set). This is crucial for systemd services.

Systemd: Managing Services and Their Privileges

Systemd systemd.exec(5) is the de facto init system and service manager on most modern Linux distributions. It provides robust mechanisms for controlling the execution environment of services, including their capabilities.

Key systemd service unit directives for capability management include:

  • User= and Group=: Define the user and group under which the service process runs. Running as a non-root user is a security best practice.
  • CapabilityBoundingSet=: Specifies the limiting set of capabilities for the service. It’s often used to drop unneeded capabilities.
  • AmbientCapabilities=: Defines the capabilities that are added to the service process’s ambient set. This is the preferred modern way to grant specific capabilities like CAP_NET_BIND_SERVICE to services that don’t run as root.
  • NoNewPrivileges=true: A security feature that prevents the service process or its children from gaining more privileges than they started with (e.g., via setuid binaries). Ambient capabilities are designed to work correctly with NoNewPrivileges=true.

Go Applications: The Receiving End

Go’s standard net package, when attempting to Listen on a TCP or UDP port, ultimately makes a bind(2) syscall. If the port is privileged and the Go process lacks CAP_NET_BIND_SERVICE in its effective capability set, the kernel will deny the request, resulting in an error like “bind: permission denied” (which is an EPERM error).

Go applications themselves don’t typically manipulate capability sets directly. They inherit them from the environment in which they are launched, which, in our case, is systemd.

The EPERM Culprit: Why Permissions Fail

An EPERM error when trying to use CAP_NET_BIND_SERVICE often boils down to one or more of these common issues:

  1. Systemd Unit Misconfiguration (Most Common):

    • AmbientCapabilities=CAP_NET_BIND_SERVICE is missing or misspelled.
    • CAP_NET_BIND_SERVICE is inadvertently dropped from the CapabilityBoundingSet=.
    • The service is configured to run as a non-root user (e.g., User=myuser), but capabilities aren’t correctly passed through this user transition. Ambient capabilities solve this.
    • Misunderstanding the interaction with NoNewPrivileges=true (though ambient capabilities generally handle this well).

  2. Incorrect Use of File Capabilities (setcap):

    • If using file capabilities (e.g., sudo setcap 'cap_net_bind_service=+ep' /path/to/app), this might not play well with systemd’s user switching or NoNewPrivileges=true, especially if the user is non-root. Ambient capabilities are generally superior for systemd services.
    • The binary was updated, and setcap was not reapplied.

  3. Overly Restrictive Global Settings: System-wide security policies or parent systemd scope unit settings might be unexpectedly limiting capabilities.

The Solution: Correctly Configuring Systemd with Ambient Capabilities

For Go applications managed by systemd, using the AmbientCapabilities= directive is the most robust and recommended way to grant CAP_NET_BIND_SERVICE.

The Preferred Method: AmbientCapabilities in Systemd

Let’s construct a systemd service unit that correctly grants our Go application the ability to bind to privileged ports.

First, create or ensure you have a dedicated non-root user for your application. This example assumes a user gouser and group gogroup.

1
2

sudo groupadd gogroup
sudo useradd -r -s /bin/false -g gogroup gouser

Below is an example systemd service unit file, typically located at /etc/systemd/system/mygoapp.service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

[Unit]
Description=My Go Application Service
After=network.target network-online.target
Requires=network-online.target

[Service]
# Run as a dedicated non-root user for security
User=gouser
Group=gogroup

# Path to your Go executable
ExecStart=/usr/local/bin/mygoapp -config /etc/mygoapp/config.json

# Standard output and error logging
StandardOutput=journal
StandardError=journal

# Restart policy
Restart=on-failure
RestartSec=5s

# ---- Capability Management ----
# Grant CAP_NET_BIND_SERVICE via the ambient set
AmbientCapabilities=CAP_NET_BIND_SERVICE

# Ensure CAP_NET_BIND_SERVICE is in the bounding set.
# Systemd's default for system services usually includes all capabilities
# in the bounding set unless explicitly restricted. For clarity, or if
# you have a very minimal global bounding set, you might add it:
# CapabilityBoundingSet=CAP_NET_BIND_SERVICE
# Or, if starting from a minimal set for other reasons:
CapabilityBoundingSet=CAP_CHOWN CAP_DAC_OVERRIDE CAP_NET_BIND_SERVICE
# (Adjust CAP_CHOWN, CAP_DAC_OVERRIDE as per your app's actual needs)
# If your app *only* needs port binding, then:
# CapabilityBoundingSet=CAP_NET_BIND_SERVICE

# Recommended for security; works well with AmbientCapabilities
NoNewPrivileges=true

# Other security hardening options (optional, adapt to your needs)
# ProtectSystem=full
# PrivateTmp=true
# PrivateDevices=true
# ProtectHome=true

[Install]
WantedBy=multi-user.target

Explanation of Key Directives:

  • User=gouser, Group=gogroup: Runs the Go application as the gouser user. This is crucial for security.
  • ExecStart=/usr/local/bin/mygoapp ...: Specifies the command to run your application.
  • AmbientCapabilities=CAP_NET_BIND_SERVICE: This is the core directive. It adds CAP_NET_BIND_SERVICE to the ambient capability set of the process, allowing it to be inherited and made effective even when running as a non-root user and with NoNewPrivileges=true.
  • CapabilityBoundingSet=...: This defines the “ceiling” of capabilities your service can ever possess. CAP_NET_BIND_SERVICE must be present in this set for it to be made ambient and effective. If you don’t specify this, systemd’s defaults usually allow it for system services. However, if you are crafting a minimal set, ensure CAP_NET_BIND_SERVICE is included.
  • NoNewPrivileges=true: This security measure prevents the process from acquiring any new privileges after it starts. Ambient capabilities are granted before the main executable runs, so this directive is compatible and recommended.

A Practical Go Application Example

Here’s a simple Go HTTP server that attempts to bind to port 80. This code, when run directly without proper permissions, would fail.

File: main.go

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

package main

import (
	"fmt"
	"log"
	"net/http"
	"os"
)

func helloHandler(w http.ResponseWriter, r *http.Request) {
	hostname, _ := os.Hostname()
	fmt.Fprintf(w, "Hello from Go on port 80! Served by: %s\n", hostname)
}

func main() {
	http.HandleFunc("/", helloHandler)

	port := ":80"
	log.Printf("Attempting to listen on port %s\n", port)

	// This is the call that requires CAP_NET_BIND_SERVICE
	err := http.ListenAndServe(port, nil)
	if err != nil {
		// This will print the "bind: permission denied" (EPERM)
		// if capabilities are not correctly set.
		log.Fatalf("Error starting server: %v\n", err)
	}
}

Compile this application:

1
2
3
4
5

go build -o mygoapp main.go
sudo mv mygoapp /usr/local/bin/
# Ensure the gouser can execute it (though systemd handles permissions)
sudo chown root:root /usr/local/bin/mygoapp # Or gouser:gogroup
sudo chmod 755 /usr/local/bin/mygoapp

Applying and Verifying the Setup

After creating or modifying your systemd service file (/etc/systemd/system/mygoapp.service):

  1. Reload Systemd Configuration:

    1

    sudo systemctl daemon-reload

  2. Enable and Start Your Service:

    1
2

    sudo systemctl enable mygoapp.service
sudo systemctl start mygoapp.service

  3. Check Service Status:

    1

    sudo systemctl status mygoapp.service

    Look for “active (running)”. If it failed, the status output will often show recent log lines indicating the error.

  4. Inspect Logs:

    1

    sudo journalctl -u mygoapp.service -f

    If the Go application failed with EPERM, you’d see the log.Fatalf output here. If successful, you’ll see your “Attempting to listen…” message. You can then test by accessing http://your_server_ip/ in a browser.

Diagnosing Persistent EPERM Issues

If you still face EPERM errors, systematic troubleshooting is key.

Step-by-Step Troubleshooting

  1. Double-Check Systemd Unit:

    • Typos in capability names (CAP_NET_BIND_SERVICE is correct).
    • Correct paths in ExecStart=.
    • Ensure User= and Group= exist and have permissions to execute the binary (though systemd usually handles this if the binary is executable by others).

  2. Inspect Process Capabilities: Once the service attempts to start (even if it fails quickly), find its Process ID (PID). If it’s running, pgrep mygoapp might work. If it fails fast, you might need to temporarily add a sleep in your Go app or systemd unit’s ExecStartPre to catch it.

    With the PID, check its capability sets:

    1
2

    # Replace <PID> with the actual process ID
cat /proc/<PID>/status | grep Cap

    This outputs hexadecimal masks for CapInh (Inheritable), CapPrm (Permitted), CapEff (Effective), CapBnd (Bounding), and CapAmb (Ambient). CAP_NET_BIND_SERVICE has the numerical value 10. Its bitmask is 2^10 = 1024, which is 0x400 in hexadecimal. You want to see this bit set in CapEff and CapAmb. For example: CapAmb: 0000000000000400 CapEff: 0000000000000400

    A helper script can get the PID and display capabilities:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

    #!/bin/bash
APP_NAME="mygoapp" # Change to your app's executable name
PID=$(pgrep -u gouser "${APP_NAME}") # Search by user and name

if [ -z "$PID" ]; then
    echo "Process ${APP_NAME} not found or not running as gouser."
    # Fallback if it runs very briefly or under a different name context
    # This is less reliable:
    # PID=$(ps aux | grep "[/]usr/local/bin/${APP_NAME}" | awk '{print $2}' | head -n1)
fi

if [ -n "$PID" ]; then
    echo "PID for ${APP_NAME}: ${PID}"
    echo "Capabilities from /proc/${PID}/status:"
    cat "/proc/${PID}/status" | grep Cap

    # If libcap-ng-utils is installed, pscap is more readable
    if command -v pscap &> /dev/null; then
        echo -e "\nCapabilities from pscap:"
        sudo pscap | grep "${PID}"
    fi
    # Or getpcaps if installed (usually part of libcap-progs or similar)
    if command -v getpcaps &> /dev/null; then
         echo -e "\nCapabilities from getpcaps ${PID}:"
         sudo getpcaps "${PID}"
    fi
else
    echo "Could not find PID for ${APP_NAME}."
fi

    Save this, chmod +x script.sh, and run it.

  3. Test with capsh (Capability Shell): capsh allows you to simulate launching processes with specific capabilities. This can help verify if the user and capability combination can work.

    1
2
3
4
5
6
7
8

    # As root, simulate how systemd might try to grant ambient caps
sudo capsh --user='gouser' \
           --caps='cap_net_bind_service+eip' \
           --ambient='cap_net_bind_service' \
           --inh='cap_net_bind_service' \
           --addamb='cap_net_bind_service' \
           --keep=1 \
           -- -c "/usr/local/bin/mygoapp"

    This is a complex command; refer to man capsh. The goal is to see if mygoapp can run under gouser if the capabilities are correctly set up by capsh. If this works but systemd fails, the issue is almost certainly in the systemd unit file or broader systemd environment.

  4. Simplify the Systemd Unit: Temporarily comment out other security hardening options (ProtectSystem, PrivateTmp, etc.) to rule out interference. Re-introduce them one by one. Do not disable NoNewPrivileges=true lightly; strive to make ambient capabilities work with it enabled.

  5. Use strace for Deep Dives: strace can show the exact syscalls your application makes.

    1
2
3
4
5
6

    # Modify your systemd unit temporarily:
# ExecStart=strace -o /tmp/mygoapp.strace -f -e trace=bind /usr/local/bin/mygoapp
# Then, after attempting to start:
# sudo systemctl daemon-reload
# sudo systemctl restart mygoapp.service
# cat /tmp/mygoapp.strace

    Look for the bind(...) call. If it returns -1 EPERM (Operation not permitted), this confirms the permission issue at the syscall level.

Beyond Ambient Capabilities: Alternatives and Advanced Scenarios

While ambient capabilities are preferred for systemd services, other methods exist.

File Capabilities (setcap): When and Why (with Caveats)

You can attach capabilities directly to an executable file:

1

sudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/mygoapp

And verify with:

1
2

getcap /usr/local/bin/mygoapp
# Expected output: /usr/local/bin/mygoapp = cap_net_bind_service+ep
  • Pros: Simple for standalone binaries run directly, works outside systemd.
  • Cons:
    • Must be reapplied every time the binary is updated.
    • Can interact poorly with systemd’s User= directive (file capabilities might be dropped during UID transition if the user is non-root) and NoNewPrivileges=true.
    • Generally less robust for services managed by systemd compared to AmbientCapabilities.

Socket Activation with Systemd

Systemd can pre-bind to sockets (as root) and then pass the ready-to-use file descriptors to your service, which then runs as a non-root user without needing any capabilities itself.

  1. Create a .socket unit (e.g., /etc/systemd/system/mygoapp.socket):

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

    [Unit]
Description=My Go Application Socket

[Socket]
ListenStream=0.0.0.0:80
# For IPv6, add: ListenStream=[::]:80
# SocketUser=root # systemd binds the socket, often as root
# SocketMode=0660
# SocketGroup=gogroup # Allow the service group to use it

[Install]
WantedBy=sockets.target

  2. Modify your .service unit (e.g., /etc/systemd/system/mygoapp.service):

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

    [Unit]
Description=My Go Application Service (Socket Activated)
# No "After=network.target" needed if only socket activated
Requires=mygoapp.socket # Important dependency
After=mygoapp.socket

[Service]
User=gouser
Group=gogroup
ExecStart=/usr/local/bin/mygoapp -socket-activation # App needs to handle this

# No capabilities needed here!
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

  3. Go Application Changes: Your Go app needs to be aware of socket activation. It will receive the listener file descriptor (usually FD 3). Libraries like go.einride.tech/linters/std/ υπηρεσιακός/go/systemd or github.com/coreos/go-systemd/v22/activation can help, or you can use net.FileListener(). A conceptual Go snippet:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

    // import "net"
// import "os"
// import "strconv"

// Check for systemd socket activation environment variables
// Typically, LISTEN_FDS will be set.
// listenFdsStr := os.Getenv("LISTEN_FDS")
// if listenFdsStr != "" {
//     fds, _ := strconv.Atoi(listenFdsStr)
//     if fds > 0 {
//         // FD 3 is the first passed listener by convention
//         file := os.NewFile(uintptr(3), "socket")
//         listener, err := net.FileListener(file)
//         // ... use this listener with http.Serve(listener, nil)
//     }
// }

Enable and start mygoapp.socket, then mygoapp.service.

Using a Reverse Proxy (e.g., Nginx, Caddy, HAProxy)

A very common and robust pattern is to run your Go application on an unprivileged port (e.g., 8080) as a non-root user (no special capabilities needed). Then, place a dedicated web server/reverse proxy like Nginx in front. Nginx listens on port 80/443 (it handles its own CAP_NET_BIND_SERVICE needs or runs initial processes as root) and forwards traffic to your Go app.

  • Pros: Enhanced security (Go app has minimal privileges), load balancing, TLS termination, caching, serving static files, etc., handled by the proxy.
  • Cons: Adds another component to configure and manage. Might be overkill for very simple services.

Important Security Considerations

  • Principle of Least Privilege: Always run your Go service as a non-root user. Use AmbientCapabilities to grant only CAP_NET_BIND_SERVICE if that’s all it needs. Keep CapabilityBoundingSet as minimal as possible.
  • SELinux/AppArmor: Mandatory Access Control systems like SELinux or AppArmor can also prevent port binding, even if Linux capabilities are correctly configured. If EPERM persists, check your system’s audit logs (e.g., /var/log/audit/audit.log or journalctl | grep 'avc: denied') for denials from these systems and adjust policies accordingly (e.g., setsebool -P httpd_can_network_connect 1 for SELinux, or updating AppArmor profiles).

Conclusion

Resolving EPERM errors when your Go application needs to bind to privileged ports under systemd typically involves a correct understanding and application of Linux capabilities, particularly AmbientCapabilities. By configuring your systemd service unit to run as a non-root user and explicitly granting CAP_NET_BIND_SERVICE through the ambient set, you can achieve secure and functional deployments.

While alternatives like file capabilities or socket activation exist, ambient capabilities provide a clean, declarative, and robust solution within the systemd ecosystem. Remember to always apply the principle of least privilege and consider other security layers like reverse proxies or MAC systems for a comprehensive security posture. With careful configuration and systematic troubleshooting, you can conquer EPERM and get your Go services listening on the ports they need.