adllm Insights logo adllm Insights logo

Creating LLDB Python Scripts for Inspecting C++20 Coroutine State in Multi-Threaded Apps

Published on by The adllm Team. Last modified: . Tags: C++20 coroutines LLDB Python debugging multi-threading on-device ml low-level debugging

C++20 coroutines provide a powerful way to write asynchronous code with a more sequential, readable syntax. However, their unique execution model, where state is saved off the main stack and execution can suspend and resume, introduces new challenges for debugging. Standard debugger commands often struggle to provide a clear picture of a coroutine’s internal state (locals, promise, suspension point), especially in multi-threaded applications where coroutines can be created, resumed, and destroyed across different threads.

This article dives into creating custom LLDB Python scripts to deeply inspect the state of C++20 coroutines. We’ll explore how to access coroutine frames, interpret promise objects, attempt to retrieve local variables, and navigate the complexities introduced by multi-threading, empowering you to build effective debugging tools tailored to your asynchronous C++ applications.

Understanding C++20 Coroutine Internals (Briefly)

To script effectively, a basic understanding of coroutine mechanics is essential:

  • Coroutine Frame: When a coroutine suspends, its state (local variables that need to persist, parameters, current suspension point, and the promise object) is typically stored in a dynamically allocated region called the “coroutine frame.” This frame is usually on the heap. Its exact layout is compiler-dependent. More on coroutine theory at cppreference.com.
  • std::coroutine_handle<>: A non-owning, low-level handle to a coroutine frame. It’s used to resume a suspended coroutine (.resume()) or destroy its frame (.destroy()). This handle is the primary way scripts will begin to access coroutine state.
  • promise_type: A user-defined type associated with every coroutine. It’s created by the compiler as part of the coroutine frame and allows the coroutine to communicate results (or exceptions) and customize behavior (e.g., initial and final suspension points via initial_suspend() and final_suspend()).

Challenges in Debugging Coroutines with Standard Tools

Traditional debuggers often fall short with C++20 coroutines because:

  • Obscured State: The coroutine frame is not on the standard call stack when suspended. frame variable might not show all relevant locals, especially those whose lifetime spans a co_await.
  • Compiler-Dependent Layout: The internal structure of the coroutine frame is an implementation detail of the compiler (Clang, GCC, MSVC), making generic inspection tools difficult to build without specific knowledge.
  • No “Async Stack Trace”: Standard backtraces (bt) only show the physical call stack, not the chain of coroutine continuations that led to the current suspension.
  • Multi-threading Complexity:
    • Identifying which thread a specific coroutine_handle is currently active on or associated with.
    • Tracking coroutines that are created on one thread and resumed on another.
    • Understanding the state of shared data accessed by concurrently running coroutines.
  • Evolving Tooling: Native debugger support for deep coroutine inspection is still maturing. While improving, as noted in discussions like LLVM Issue #69309, custom scripts often provide more immediate and tailored solutions.

Leveraging LLDB Python Scripting

LLDB, the debugger from the LLVM project, provides a powerful Python scripting interface that allows developers to extend its capabilities. This is ideal for coroutine inspection. The lldb Python module provides classes to interact with the debugger and the debugged process:

  • lldb.debugger: The main debugger object.
  • SBTarget: Represents the executable and its associated images.
  • SBProcess: Represents the running process.
  • SBThread: Represents a thread within the process.
  • SBFrame: Represents a call stack frame.
  • SBValue: Represents the value of a variable or expression.
  • SBType: Represents a type.
  • SBAddress: Represents an address in memory.

Using these APIs, we can programmatically access memory, interpret data structures, and create custom commands. The official LLDB Python scripting documentation is an excellent resource.

Core Scripting Techniques for Coroutine Inspection

Let’s outline the key steps an LLDB Python script would take.

1. Accessing the coroutine_handle

Typically, you’ll start with an SBValue representing a std::coroutine_handle variable in your C++ code.

1
2
3
4
5
6
7

# Assuming 'frame' is an lldb.SBFrame object
# and 'handle_name' is the C++ variable name of the coroutine_handle
coro_handle_sbvalue = frame.FindVariable(handle_name)

if not coro_handle_sbvalue.IsValid():
    print(f"Error: Could not find coroutine_handle variable '{handle_name}'")
    return

2. Locating the Coroutine Frame Address

The std::coroutine_handle stores a pointer to the coroutine frame. The method address_of() on the handle gives the address of the handle object itself, while h.address() (where h is a std::coroutine_handle) returns the raw address of the coroutine frame.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# coro_handle_sbvalue is an SBValue for std::coroutine_handle<P>
# First, ensure it's not a null handle
if coro_handle_sbvalue.GetValueAsUnsigned() == 0:
    print("Coroutine handle is null.")
    return

# The coroutine_handle itself often directly holds the frame address.
# Accessing it might involve calling its .address() method or
# understanding its internal structure if direct member access is needed.
# For a std::coroutine_handle h, h.address() gives a void*.
# In LLDB, we can try to evaluate this:
target = lldb.debugger.GetSelectedTarget()
frame_addr_expr = f"(void*){coro_handle_sbvalue.GetName()}.address()"
frame_addr_sbvalue = frame.EvaluateExpression(frame_addr_expr)

if not frame_addr_sbvalue.IsValid() or \
   frame_addr_sbvalue.GetValueAsUnsigned() == 0:
    print(f"Failed to get frame address from {coro_handle_sbvalue.GetName()}")
    return

coro_frame_address = frame_addr_sbvalue.GetValueAsUnsigned()
print(f"Coroutine frame address: 0x{coro_frame_address:x}")

Note: The exact way a std::coroutine_handle stores the frame pointer can be ABI-specific, but h.address() is the standard C++ way to obtain it.

3. Interpreting the Coroutine Frame

Once you have the coro_frame_address, you need to cast it to the actual compiler-generated coroutine frame type. This type name is usually mangled or follows a compiler-specific pattern (e.g., MyFunction.Frame). Finding this exact type name might require inspecting debug symbols or compiler documentation (like Clang’s DebuggingCoroutines.html).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# Assume 'actual_coro_frame_type_name' is the string name of the
# compiler-generated frame type for your coroutine.
# This is the most challenging part to generalize.
# For Clang, it might be something like "foo()::Frame", where foo is your
# coroutine.
actual_coro_frame_type = target.FindFirstType(actual_coro_frame_type_name)

if not actual_coro_frame_type.IsValid():
    print(f"Error: Could not find frame type '{actual_coro_frame_type_name}'")
    return

# Create an SBValue from the raw address and the determined type
coro_frame_sbvalue = target.CreateValueFromAddress(
    "coro_frame",
    lldb.SBAddress(coro_frame_address, target),
    actual_coro_frame_type
)

if not coro_frame_sbvalue.IsValid():
    print("Error: Could not create SBValue for coroutine frame.")
    return

Accessing the promise_type Object

The promise_type object is a member of the coroutine frame. Its name is often promise or __promise (compiler-dependent).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# Try common names for the promise member
promise_member_names = ["promise", "__promise"]
promise_sbvalue = None
for name in promise_member_names:
    promise_sbvalue = coro_frame_sbvalue.GetChildMemberWithName(name)
    if promise_sbvalue and promise_sbvalue.IsValid():
        break

if not promise_sbvalue or not promise_sbvalue.IsValid():
    print("Error: Could not find promise object in coroutine frame.")
    return

print(f"Promise object type: {promise_sbvalue.GetType().GetName()}")
print(f"Promise object address: {promise_sbvalue.GetAddress()}")

# Now you can inspect members of your custom promise_type
# For example, if your promise_type has a member 'my_custom_state':
# custom_state_val = promise_sbvalue.GetChildMemberWithName("my_custom_state")
# if custom_state_val and custom_state_val.IsValid():
#     print(f"Promise.my_custom_state: {custom_state_val.GetValue()}")

4. Retrieving Coroutine Local Variables

Accessing local variables stored in the coroutine frame is highly dependent on DWARF information quality and compiler optimizations.

  • If DWARF is good, LLDB’s frame variable might show some locals when stopped inside the coroutine (before it suspends for the first time or after resumption).
  • For variables that live across co_await points and are part of the frame, direct access via LLDB Python scripting might involve:
    • Iterating children of coro_frame_sbvalue if DWARF correctly describes them as members.
    • Knowing specific member names or offsets (fragile, compiler-dependent).

This area often requires experimentation. The Clang documentation on Debugging Coroutines provides insights, though it may be specific to certain compiler versions or focus on GDB.

5. Building an “Async Stack Trace”

If your promise_type is designed to store a std::coroutine_handle to its awaiting coroutine (the “continuation”), you can recursively walk this chain.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

// In C++:
struct MyPromise {
    // ... other members
    std::coroutine_handle<> awaiting_coroutine; // Stored by await_suspend

    // In your Awaiter's await_suspend:
    // void await_suspend(std::coroutine_handle<> h) {
    //   this_promise.awaiting_coroutine = h; // 'this_promise' is this coroutine's promise
    //   // ... schedule resumption ...
    // }
};

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

def print_async_stack_trace(coro_frame_addr_str, target, result_builder, depth=0):
    # (Similar logic as above to get coro_frame_sbvalue and promise_sbvalue)
    # ...
    # continuation_handle_sbval = promise_sbvalue.GetChildMemberWithName("awaiting_coroutine")
    # if continuation_handle_sbval and \
    #    continuation_handle_sbval.GetValueAsUnsigned() != 0:
    #     # Get the frame address of the awaiting coroutine
    #     expr = f"(void*){continuation_handle_sbval.GetName()}.address()"
    #     next_frame_addr_val = target.GetProcess().GetSelectedThread() \
    #                               .GetSelectedFrame().EvaluateExpression(expr)
    #     if next_frame_addr_val and \
    #        next_frame_addr_val.GetValueAsUnsigned() != 0:
    #         result_builder.Printf(f"{'  ' * depth} Awaited by: ...")
    #         # Need to resolve its frame type and recursively call
    #         # print_async_stack_trace(hex(next_frame_addr_val.GetValueAsUnsigned()),
    #         #                        target, result_builder, depth + 1)
    pass # Placeholder for full recursive logic

This requires careful design of your promise_type and awaiters.

Handling Multi-Threaded Scenarios

When dealing with multiple threads:

  • Iterate Threads: Use process = target.GetProcess() and then loop through process.threads (or process.GetThreadAtIndex(i)).
  • Thread-Specific State: For each SBThread, you can get its current frames (thread.GetFrameAtIndex(j)). You can then check if the function in a frame is a known coroutine or if coroutine_handle variables are present on its stack.
  • Identify Active Coroutine: Pinpointing which coroutine is currently executing on a thread involves checking the Program Counter (PC) and function name of the top stack frame. If it’s a coroutine’s resume function, you’re inside that coroutine.
  • Global Coroutine Lists: If your application maintains global lists or maps of active coroutine handles, your script can access these global variables to get starting points for inspection.
1
2
3
4
5
6
7
8
9

# Example: Iterate threads and print top frame function
# process = target.GetProcess()
# for i in range(process.GetNumThreads()):
#     thread = process.GetThreadAtIndex(i)
#     frame = thread.GetFrameAtIndex(0) # Top frame
#     if frame.IsValid():
#         func_name = frame.GetFunctionName()
#         print(f"Thread {thread.GetThreadID()}: Top frame function: {func_name}")
#         # Further logic to check if func_name is a coroutine body/resume

Practical LLDB Script Development

Creating Custom LLDB Commands

Package your Python functions into LLDB commands for easy use.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

# In your LLDB Python script file (e.g., coro_inspector.py)
import lldb

def inspect_coroutine_command(debugger, command, result, internal_dict):
    """
    LLDB command to inspect a C++20 coroutine.
    Usage: inspect_coro <coroutine_handle_variable_name_in_current_frame>
    """
    target = debugger.GetSelectedTarget()
    if not target:
        result.SetError("Invalid target.")
        return

    process = target.GetProcess()
    if not process:
        result.SetError("Invalid process.")
        return

    thread = process.GetSelectedThread()
    if not thread:
        result.SetError("Invalid thread.")
        return

    frame = thread.GetSelectedFrame()
    if not frame:
        result.SetError("Invalid frame.")
        return

    if not command: # 'command' holds the arguments to your LLDB command
        result.SetError("Usage: inspect_coro <coroutine_handle_var_name>")
        return

    handle_name = command # The argument is the variable name

    # --- Combine logic from sections above ---
    # 1. Get coro_handle_sbvalue = frame.FindVariable(handle_name)
    # 2. Get coro_frame_address from it
    # 3. Find actual_coro_frame_type_name (this is tricky, may need hints)
    # 4. Create coro_frame_sbvalue from address and type
    # 5. Get promise_sbvalue from coro_frame_sbvalue
    # 6. Print details from promise_sbvalue to 'result' (e.g., result.Printf(...))
    # Example:
    result.Printf(f"Inspecting coroutine handle: {handle_name}\n")
    # ... (actual inspection logic here) ...
    result.Printf("Inspection logic placeholder.\n")


# This function is called when the script is loaded by LLDB
def __lldb_init_module(debugger, internal_dict):
    # Add the command to LLDB
    # The first argument is the command name in LLDB
    # The second is the Python function that implements it
    # The third is a help string
    debugger.HandleCommand(
        'command script add -f coro_inspector.inspect_coroutine_command inspect_coro'
    )
    print('LLDB command "inspect_coro" registered.')

Load this script in LLDB: (lldb) command script import /path/to/coro_inspector.py.

Error Handling and Iterative Development

  • Always check if SBValue, SBFrame, etc., objects are valid (.IsValid()).
  • Use SBError objects returned by some API calls.
  • Use (lldb) script to enter LLDB’s embedded Python interpreter for quick tests.
  • print() statements in your script go to the LLDB console, aiding debugging.

Example: A Simple C++ Coroutine and Inspector Idea

Consider this C++ coroutine:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

#include <coroutine>
#include <iostream>
#include <thread> // For std::this_thread for demo

struct MySimpleTask {
    struct promise_type {
        int internal_value = 0;
        std::coroutine_handle<> continuation = nullptr;

        MySimpleTask get_return_object() {
            return MySimpleTask{
                std::coroutine_handle<promise_type>::from_promise(*this)
            };
        }
        std::suspend_always initial_suspend() noexcept { return {}; }
        std::suspend_always final_suspend() noexcept {
            // In a real scenario, might resume continuation here
            if (continuation) {
                // Not resuming in this simple example to keep handle alive
                // continuation.resume();
            }
            return {};
        }
        void return_void() {}
        void unhandled_exception() { std::terminate(); }

        // Custom function to be called by awaiter
        void set_awaiting_coroutine(std::coroutine_handle<> h) {
            continuation = h;
        }
    };

    std::coroutine_handle<promise_type> h_;
    explicit MySimpleTask(std::coroutine_handle<promise_type> h) : h_(h) {}
    // ... (operator co_await, resume functions etc.)
    bool resume() { if (h_ && !h_.done()) { h_.resume(); return true; } return false; }
    int get_promise_value() { if(h_) return h_.promise().internal_value; return -1; }
};

MySimpleTask my_coro_func(int start_val) {
    std::cout << "Coroutine started with: " << start_val << std::endl;
    co_await std::suspend_always{}; // Suspend for inspection

    // Access promise to change value
    // This requires knowing how to get the current coroutine's promise_type.
    // For simplicity, we'll imagine our script reads/writes this.
    // In a real coroutine, you'd get a reference to the promise via
    // std::coroutine_handle<promise_type>::from_promise(*this) in promise_type
    // or by other means if your framework provides it.
    // Here, we'll assume the script will modify promise().internal_value.

    std::cout << "Coroutine resumed. Thread ID: "
              << std::this_thread::get_id() << std::endl;
    co_return;
}

Your LLDB script (inspect_coro command) would aim to:

  1. Take the name of a MySimpleTask variable (e.g., task_obj).
  2. Access task_obj.h_ (the coroutine_handle).
  3. Get the frame address from task_obj.h_.address().
  4. Find the frame type for my_coro_func (e.g., my_coro_func.Frame).
  5. Cast and get the promise member.
  6. Print promise.internal_value and promise.continuation.

Advanced Considerations and Future

  • Compiler ABI: Reliance on compiler-specific frame layouts makes scripts potentially fragile across different compilers or even versions of the same compiler.
  • Optimizations: Heavily optimized coroutines might have different frame layouts or elided variables. Always debug with optimizations turned down (-O0 -g) initially.
  • DWARF and Native Support: As compilers and debuggers improve DWARF emission and native coroutine support (see proposals like P2073R0), some script functionalities might become built-in. However, custom scripts will likely remain valuable for framework-specific logic or advanced visualization.

Conclusion

Debugging C++20 coroutines, especially in multi-threaded environments, presents unique hurdles. LLDB’s Python scripting interface offers a robust mechanism to overcome these, allowing developers to create bespoke tools for inspecting coroutine state, understanding execution flow, and ultimately, building more reliable asynchronous applications. While the learning curve for both coroutine internals and LLDB scripting can be steep, the ability to tailor your debugger to your specific needs is a powerful asset. As native tooling evolves, the core principles learned from scripting will remain valuable for understanding and debugging complex asynchronous systems.