Using strace to Diagnose File Descriptor Leaks in Python Celery Workers

Long-running background task processors, like Python Celery workers, are critical components of many modern applications. However, their longevity can also make them susceptible to insidious issues like file descriptor (FD) leaks. An FD leak, where a process opens files, sockets, or other resources but fails to close them, can gradually cripple a worker, leading to “Too many open files” errors, task failures, and eventual process starvation. Diagnosing these leaks can be challenging, especially in complex codebases or when third-party libraries are involved.

This is where strace, a powerful Linux diagnostic tool, becomes invaluable. By tracing the system calls made by a Celery worker, strace allows us to observe its interactions with the operating system at the most fundamental level, including every file or socket opening and closing. This article provides a definitive guide for experienced engineers on leveraging strace to meticulously identify and debug file descriptor leaks in Python Celery workers.

Understanding the Core Concepts

Before diving into strace techniques, let’s solidify our understanding of file descriptors and how leaks manifest.

What are File Descriptors (FDs)?

In Unix-like operating systems, a file descriptor is a non-negative integer that serves as an abstract handle or identifier for an I/O resource. When a process opens a file, creates a socket, or establishes a pipe, the kernel allocates an FD for that resource within the process’s FD table. The first three FDs are typically standard:

0: Standard input (stdin)
1: Standard output (stdout)
2: Standard error (stderr)

Subsequent resources opened by the process will receive the lowest available FD number. For more details on file descriptors, see the open(2) man page.

How Do File Descriptor Leaks Occur?

A file descriptor leak happens when a program requests an FD by opening a resource but does not release it back to the system by closing it when it’s no longer needed. Over time, the process accumulates these unclosed FDs. Each process has a limit on the number of FDs it can simultaneously hold (configurable via ulimit or system-wide settings).

Impact of FD Leaks on Celery Workers

For long-running Celery workers, which may process thousands or millions of tasks without restarting, even a small FD leak within a task can have a cumulative effect. Eventually, the worker process may hit its FD limit. When this occurs:

The worker can no longer open new files, including log files.
The worker cannot establish new network connections (e.g., to the message broker, databases, or external APIs).
Tasks start failing with errors like IOError: [Errno 24] Too many open files or similar socket errors.
The worker process itself might become unstable or unresponsive.

These issues can be intermittent and difficult to reproduce, making debugging a frustrating endeavor without the right tools.

`strace`: Your System Call Detective

strace is a command-line utility for Linux that intercepts and records the system calls made by a process and any signals it receives. It provides a direct window into how a program interacts with the kernel.

What `strace` Does

When you attach strace to a process (or start a process under strace), it uses the ptrace(2) system call to gain control of the target process. It then logs every system call, its arguments, and its return value. For FD leak detection, we are primarily interested in system calls that create FDs (like open(2), openat(2), socket(2), pipe(2), dup(2)) and the call that releases them (close(2)).

Key `strace` Options for FD Leak Detection

Mastering a few strace options is crucial for effective FD leak hunting:

-p PID: Attaches strace to an already running process specified by PID.
-f: Traces child processes created by the currently traced process. This is essential for Celery workers using the default prefork pool, as tasks are executed in child processes.
-e trace=syscall_set: Filters the trace output to specific system calls. This is vital to reduce noise. Useful sets include:
- open,openat,close,socket,pipe,dup,dup2,connect,accept: A good starting point for FD-related issues.
- desc: Traces all system calls related to file descriptors (as defined by strace).
- file: Traces all system calls that take a filename as an argument.
- network: Traces all network-related system calls.
-o output_file: Writes the trace output to output_file instead of stderr, which is highly recommended for long-running traces.
-t, -tt, -ttt: Prepends timestamps to each line of output (-ttt provides microsecond precision and is often the most useful).
-s strsize: Specifies the maximum string size to print for arguments (default is 32). Increase this (e.g., -s 128) to see longer filenames or paths.
-c: Counts the time, calls, and errors for each system call and prints a summary upon exiting strace. This can quickly highlight imbalances (e.g., many more openat calls than close calls).

Step-by-Step: Diagnosing FD Leaks with `strace`

Here’s a systematic approach to using strace to find FD leaks in your Celery workers.

Step 1: Confirming an FD Leak (Monitoring FD Counts)

Before deploying strace, confirm that an FD leak is indeed occurring. Monitor the FD count of your Celery worker processes over time.

You can check the number of open FDs for a process using the /proc filesystem (see proc(5) for details on /proc/[pid]/fd):

1
2
# Replace <PID> with the actual process ID
sudo ls /proc/<PID>/fd | wc -l

Alternatively, lsof (List Open Files) is a powerful utility:

1
2
# Replace <PID> with the actual process ID
sudo lsof -p <PID> | wc -l

If these counts steadily increase over time without a corresponding increase in legitimate workload or active connections, you likely have an FD leak.

A simple monitoring script could be:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
WORKER_PID=$(pgrep -f "celery worker -A your_app.celery") # Adjust as needed
if [ -z "$WORKER_PID" ]; then
  echo "Celery worker process not found."
  exit 1
fi

echo "Monitoring FD count for Celery worker PID: $WORKER_PID"
echo "Timestamp                FD_Count"
while true; do
  FD_COUNT=$(sudo ls /proc/$WORKER_PID/fd | wc -l)
  echo "$(date '+%Y-%m-%d %H:%M:%S')   $FD_COUNT"
  sleep 60 # Check every minute
done

Run this and observe if FD_COUNT consistently climbs.

Step 2: Identifying Celery Worker PIDs

To attach strace, you need the Process ID (PID) of the Celery worker(s).

Use ps aux | grep celery or pgrep -f "celery worker":

1
2
3
4
5
pgrep -af "celery worker"
# Example output:
# 12345 /path/to/venv/bin/python -m celery -A myapp worker -l INFO
# 12350 /path/to/venv/bin/python -m celery -A myapp worker -l INFO
# 12351 /path/to/venv/bin/python -m celery -A myapp worker -l INFO

If using the prefork pool (default), there will be a main parent worker process and several child worker processes. You can visualize this with pstree -p <MAIN_PARENT_PID>.
- Leaks often occur in child processes as they execute tasks.
- The main parent PID is often the one you’ll want to trace with strace -f.

Step 3: Attaching `strace` to Celery Workers

Once you have the PID(s):

To trace a specific child worker process (if you’ve isolated a leaky one):

1
2
3
4
5
# Replace <CHILD_PID> with the child worker's PID
sudo strace -p <CHILD_PID> \
  -ttt -s 128 \
  -e trace=open,openat,close,socket,pipe,dup,connect,accept \
  -o /tmp/celery_child_trace_<CHILD_PID>.txt

To trace the main worker process and all its forked children (most common for prefork pool):

1
2
3
4
5
# Replace <MAIN_PARENT_PID> with the main worker's PID
sudo strace -f -p <MAIN_PARENT_PID> \
  -ttt -s 128 \
  -e trace=open,openat,close,socket,pipe,dup,connect,accept \
  -o /tmp/celery_main_forked_trace.txt

The -f option ensures that system calls from any child processes forked by the main worker are also captured, each prefixed with its PID.

To start a worker under strace (useful for leaks occurring early or if attaching is problematic):

1
2
3
4
5
sudo strace -f \
  -ttt -s 128 \
  -e trace=open,openat,close,socket,pipe \
  -o /tmp/celery_startup_trace.txt \
  /path/to/your/venv/bin/celery -A your_app_name worker -l INFO -P prefork

Let strace run while the suspected leaky tasks are being processed or while the FD count is observed to rise.

Step 4: Analyzing `strace` Output

This is the most critical step. The strace output file can be large. Look for these patterns:

FD Allocation: System calls like openat(), socket(), pipe(), dup() return a new file descriptor (a small integer) on success.

1
2
3
4
# Example of openat returning a new FD (15 in this case)
[pid 21453] 10:30:01.123456 openat(AT_FDCWD, "/tmp/some_file.txt",
    O_RDONLY|O_CLOEXEC) = 15
[pid 21453] 10:30:01.123500 socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 16

FD Deallocation: The close() system call releases an FD.

1
2
3
# Example of closing FD 15
[pid 21453] 10:30:02.678910 close(15)          = 0
[pid 21453] 10:30:02.679000 close(16)          = 0

The Leak Pattern: A leak is indicated when you see many FDs being allocated but significantly fewer (or no) corresponding close() calls for those specific FD numbers within a reasonable timeframe or logical unit of work (e.g., a task).
- The OS reuses the lowest available FD numbers. If you see allocated FD numbers consistently increasing over time (e.g., ... = 15, then later ... = 16, ... = 17, etc., without close(15), close(16) in between for those PIDs), this is a strong sign of a leak.

Filtering and Searching: Use tools like grep, awk, or scripts to analyze the trace file.

grep 'openat(' /tmp/celery_trace.txt
grep 'close(' /tmp/celery_trace.txt

To find FDs opened but potentially not closed within a PID’s activity:

1
2
3
4
5
6
7
# For a specific PID, e.g., 21453 from the -f output
# This is a basic conceptual check; robust parsing may be needed.
grep '^[pid  *21453]' /tmp/celery_main_forked_trace.txt | \
grep -E 'openat\(|socket\(|pipe\('

grep '^[pid  *21453]' /tmp/celery_main_forked_trace.txt | \
grep 'close('

Pay attention to the PIDs in the output if using -f. Leaks are often specific to certain child processes executing particular tasks.

Correlate with Timestamps: Use the -ttt timestamps to correlate strace activity with Celery task logs or known event times. If a specific task is suspected, trigger it and observe the strace output around that time.

Illustrative strace Snippet (Conceptual Leak):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# Conceptual output for leaky_file_task
[pid 12350] 11:05:10.000100 openat(AT_FDCWD, "/tmp/task_A.tmp",
    O_WRONLY|O_CREAT) = 12
[pid 12350] 11:05:10.000200 write(12, "data", 4)             = 4
# ... task processing ...
# Missing: close(12) for this task completion

[pid 12350] 11:05:15.000300 openat(AT_FDCWD, "/tmp/task_B.tmp",
    O_WRONLY|O_CREAT) = 13
[pid 12350] 11:05:15.000400 write(13, "data", 4)             = 4
# Missing: close(13)

# Conceptual output for good_file_task
[pid 12350] 11:05:20.000100 openat(AT_FDCWD, "/tmp/task_C.tmp",
    O_WRONLY|O_CREAT) = 14
# ... (write syscall) ...
[pid 12350] 11:05:20.000200 close(14)                        = 0

If you see /path/to/some/library/file.py or a specific network connection consistently being opened without a corresponding close, that’s your prime suspect.

Step 5: Simulating and Catching a Leak (A Practical Example)

Let’s create a Celery task that intentionally leaks a file descriptor and then use strace to observe it.

Python Celery Task with an FD Leak:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
# tasks.py
from celery import Celery
import os
import time

# Adjust broker URL as per your Celery setup
app = Celery('tasks', broker='redis://localhost:6379/0')

# A list to hold open file objects to ensure they are not garbage
# collected prematurely, making the leak more apparent. This simulates
# a complex scenario where a reference might be unintentionally kept.
leaked_fds_refs = []

@app.task
def leaky_file_task(filename_suffix):
    filepath = f"/tmp/leaky_file_{os.getpid()}_{filename_suffix}.txt"
    try:
        # Problem: File is opened but not closed
        f = open(filepath, "w")
        f.write(f"This is a test from PID {os.getpid()}.\n")
        # To make the leak more "sticky" for demonstration,
        # append to a global list. In real scenarios, leaks are often due to
        # forgotten close() calls or resources held by objects that aren't
        # properly cleaned up.
        leaked_fds_refs.append(f)
        return f"File '{filepath}' written, FD potentially leaked."
    except Exception as e:
        return f"Error in leaky_file_task: {e}"

@app.task
def good_file_task(filename_suffix):
    filepath = f"/tmp/good_file_{os.getpid()}_{filename_suffix}.txt"
    try:
        with open(filepath, "w") as f:
            f.write(f"This test from PID {os.getpid()} handled correctly.\n")
        return f"File '{filepath}' written and closed."
    except Exception as e:
        return f"Error in good_file_task: {e}"

Now, run your Celery worker:

1
2
3
4
celery -A tasks worker -l INFO -P prefork -c 1 --max-tasks-per-child=10
# Using -c 1 for easier PID tracking in this example
# --max-tasks-per-child will eventually recycle the worker, hiding the
# leak's long-term impact but for tracing a few tasks it's fine.

Find the main Celery worker PID. Let’s say it’s MAIN_PID. Attach strace:

1
2
3
4
sudo strace -f -p MAIN_PID \
  -ttt -s 128 \
  -e trace=openat,close \
  -o /tmp/celery_leaky_task_trace.txt

Now, trigger the leaky task multiple times from a Python shell or another script:

1
2
3
4
5
6
# trigger_tasks.py
from tasks import leaky_file_task, good_file_task

for i in range(5):
    leaky_file_task.delay(i)
    good_file_task.delay(i) # Also run good tasks for comparison

After the tasks have run, stop strace (Ctrl+C). Examine /tmp/celery_leaky_task_trace.txt. You should be able to find sections related to the child process that executed leaky_file_task:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
# Conceptual output for leaky_file_task
[pid CHILD_PID] HH:MM:SS.micros openat(AT_FDCWD,
    "/tmp/leaky_file_CHILD_PID_0.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 12
... (other syscalls, maybe write) ...
# NO corresponding close(12) for this FD before the task ends or another openat

[pid CHILD_PID] HH:MM:SS.micros openat(AT_FDCWD,
    "/tmp/leaky_file_CHILD_PID_1.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 13
...
# NO corresponding close(13)

# Conceptual output for good_file_task
[pid CHILD_PID] HH:MM:SS.micros openat(AT_FDCWD,
    "/tmp/good_file_CHILD_PID_0.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 14
... (write syscall) ...
[pid CHILD_PID] HH:MM:SS.micros close(14)          = 0

This contrast clearly shows the openat calls without matching close calls for the leaky task’s files.

Common Causes of FD Leaks in Python Celery Applications

Unclosed Files: Forgetting file.close() after file = open(...), especially in complex logic or error handling paths. Refer to Python’s documentation on File Objects and the with statement for File I/O.
Network Socket Issues: Clients for databases, HTTP requests (e.g., requests library), or other network services might not release sockets if not used correctly (e.g., connections not closed, pooling issues).
Leaks in Third-Party Libraries: A library used by your task might have its own FD leaks. strace can help identify which library calls are problematic.
Improper Subprocess Management: Using subprocess.Popen without carefully managing the close_fds argument (though default is True for Python 3.7+) or not closing pipes (stdin, stdout, stderr) to the child process.
Resources Held by Long-Lived Objects: Objects that are not garbage-collected and hold references to open FDs can cause leaks.

Best Practices for Preventing FD Leaks

Use Context Managers (with statement): This is the most Pythonic and robust way to ensure resources are closed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# For files
with open("myfile.txt", "r") as f:
    # do stuff with f
# f is automatically closed here, even if exceptions occur

# Similarly for libraries that support context managers
# import requests
# with requests.Session() as session:
#     with session.get("http://example.com") as response:
#         # process response
#     # response is closed
# # session is closed

Explicit close() in finally Blocks: If context managers aren’t an option, use try...finally:

1
2
3
4
5
6
7
f = None
try:
    f = open("myfile.txt", "w")
    # ... operate on f ...
finally:
    if f:
        f.close()

Library Vigilance: Understand how the libraries you use manage resources like connections. Check their documentation for proper usage patterns.
Celery’s maxtasksperchild: Setting maxtasksperchild for Celery workers causes them to restart after processing a certain number of tasks. This can mitigate the impact of slow leaks but doesn’t fix the underlying bug. It’s a workaround, not a solution.
Code Reviews: Specifically look for resource management patterns during code reviews.
Monitoring: Continuously monitor FD usage for your worker processes in production.

Beyond `strace`: Complementary Tools and Considerations

lsof: As mentioned, lsof -p <PID> provides a snapshot of all files and network sockets opened by a process. It’s excellent for seeing the current state of open FDs.
/proc/[pid]/fd and /proc/[pid]/fdinfo: These kernel interfaces (detailed in proc(5)) provide raw information about open FDs. fdinfo can give more details like the flags and position for file FDs.
Python’s ResourceWarning: Python can emit ResourceWarning for some unclosed resources like files. Ensure these warnings are enabled and logged, especially during development and testing. However, they don’t cover all types of FD leaks (e.g., those from C extensions).
Performance Impact of strace: strace significantly slows down the traced process because it intercepts every system call. Use it judiciously in production, ideally on a single problematic worker or for limited durations. The -c option for summarizing can be less invasive for getting an overview.
eBPF (Extended Berkeley Packet Filter): For more advanced, lower-overhead tracing in modern Linux kernels, tools based on eBPF (like bpftrace or BCC tools) offer powerful alternatives. They have a steeper learning curve but can be less intrusive for production diagnostics.

Conclusion

File descriptor leaks in long-running Python Celery workers can be elusive but ultimately debilitating. strace provides an unparalleled, low-level view of a worker’s system interactions, enabling engineers to pinpoint the exact operations responsible for leaking FDs. By systematically monitoring FD counts, identifying target PIDs, carefully invoking strace with appropriate filters, and meticulously analyzing its output, you can effectively diagnose and resolve these critical resource leaks.

Combined with preventative best practices like consistent use of context managers and vigilant resource handling in your Python code, mastering strace equips you to ensure the stability and reliability of your Celery-based applications. While strace itself is a diagnostic tool, the insights it provides are crucial for driving the necessary code changes to build more robust and resilient systems.