Implementing Custom HttpContent for Binary RPC with .NET HttpClient

When building distributed systems, especially those requiring high performance and low overhead, binary Remote Procedure Call (RPC) protocols often become the communication backbone. While System.Net.Http.HttpClient is the standard for HTTP communication in .NET, its default content types like StringContent or JsonContent are designed for text-based data. To send custom binary payloads, such as those generated by Protocol Buffers, MessagePack, or other proprietary binary serializers, you need a more tailored approach: implementing a custom System.Net.Http.HttpContent.

This article provides a deep dive into creating and utilizing custom HttpContent derivatives for binary RPC. We’ll explore the necessary overrides, best practices for efficient serialization, and how to integrate this with HttpClient for seamless and performant binary data exchange. This technique gives developers fine-grained control over the serialization process directly into the HTTP request stream.

Understanding the Core Components

Before diving into the implementation, let’s clarify the key .NET components involved.

• System.Net.Http.HttpContent: This abstract base class represents an HTTP entity body and content headers. It’s the foundation for all types of content that can be sent or received via HttpClient. Common derived classes include StringContent, ByteArrayContent, and StreamContent. For custom serialization logic, you’ll inherit from HttpContent. You can find its documentation on Microsoft Learn.
• Binary RPC Protocols: These protocols define how data is structured and exchanged in a binary format. Unlike text-based formats like JSON or XML, binary formats are typically more compact and faster to parse, making them ideal for performance-sensitive applications. Examples include Google’s Protocol Buffers (Protobuf) and MessagePack.
• System.Net.Http.HttpClient: The workhorse for sending HTTP requests and receiving responses in .NET. It uses HttpContent objects to manage the request body. See more at Microsoft Learn.

The primary goal of a custom HttpContent for binary RPC is to serialize a .NET object directly into the HTTP request’s output stream using the chosen binary format and to correctly set associated HTTP headers like Content-Type.

Crafting Your Custom Binary HttpContent

Creating a custom HttpContent involves inheriting from the base class and overriding a few key methods.

Key Methods to Override

1. SerializeToStreamAsync(Stream stream, TransportContext context): This is the most crucial method. Your implementation will serialize the data object directly into the stream provided. This stream is the actual request stream that HttpClient sends to the server. It’s vital to use asynchronous operations here.
2. TryComputeLength(out long length): This method attempts to calculate the content’s length in bytes.
   • If you can determine the exact length beforehand (e.g., some serializers can calculate this), set length and return true. This allows HttpClient to set the Content-Length header, which can be more efficient.
   • If the length is unknown or expensive to calculate upfront (common with true streaming), set length = -1 (or 0) and return false. HttpClient will then typically use Transfer-Encoding: chunked.
3. SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken) (Optional but Recommended): Overriding this newer overload (available in .NET Standard 2.1+ and .NET Core 2.1+) allows your serialization logic to be cancellable, which is good practice for robust applications. If not overridden, the base implementation calls the two-argument overload.

Setting the Content-Type Header

The Content-Type header informs the server how to interpret the request body. You must set this appropriately for your binary protocol. This is typically done in the constructor of your custom HttpContent class using the Headers.ContentType property, assigning it a System.Net.Http.Headers.MediaTypeHeaderValue instance. For example, for Protobuf, it might be application/x-protobuf, or for MessagePack, application/x-msgpack, or a custom MIME type specific to your RPC protocol. You can find MediaTypeHeaderValue documentation on Microsoft Learn.

Constructor Design

The constructor of your custom class will typically accept the object to be serialized and any necessary serializer instances or options. It’s also the ideal place to set the Content-Type header.

Step-by-Step Implementation Example

Let’s build a generic CustomBinaryContent<T> that can use an arbitrary binary serializer.

1. Define a Sample Request Object and Serializer Interface

First, let’s define a simple data object our HttpContent will serialize, and an interface for our binary serializer. This abstraction allows CustomBinaryContent<T> to work with any binary serialization library.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

// Example request data structure
public class MyRpcRequest
{
    public int RequestId { get; set; }
    public string? Payload { get; set; }
    public DateTime Timestamp { get; set; }

    // Override ToString for simple logging in the serializer example
    public override string ToString() => 
        $"Id: {RequestId}, Payload: '{Payload}', Time: {Timestamp:O}";
}

// Conceptual interface for a binary serializer
public interface IBinarySerializer
{
    Task SerializeAsync<T>(Stream stream, T item, 
        CancellationToken cancellationToken = default);
    
    // Optional: Only if your serializer can pre-calculate length easily
    // and without significant overhead.
    bool TryComputeLength<T>(T item, out long length); 
}

2. Implement the Custom HttpContent

Now, we create CustomBinaryContent<T>, inheriting from HttpContent and implementing the core logic.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

using System;
using System.IO;
using System.Net; // Required for TransportContext
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading;
using System.Threading.Tasks;

public class CustomBinaryContent<T> : HttpContent
{
    private readonly T _value;
    private readonly IBinarySerializer _serializer;

    // Optional: for caching pre-serialized data if TryComputeLength buffers.
    // Use with caution due to memory implications.
    private byte[]? _bufferedData; 

    public CustomBinaryContent(T value, IBinarySerializer serializer, 
                               string contentType)
    {
        if (value == null) // Or value is default for struct
            throw new ArgumentNullException(nameof(value));
        if (serializer == null)
            throw new ArgumentNullException(nameof(serializer));
        if (string.IsNullOrEmpty(contentType))
            throw new ArgumentNullException(nameof(contentType));

        _value = value;
        _serializer = serializer;
        // Parse the content type string into a MediaTypeHeaderValue
        Headers.ContentType = MediaTypeHeaderValue.Parse(contentType);
    }

    // Main serialization method with CancellationToken support
    protected override async Task SerializeToStreamAsync(Stream stream, 
                                               TransportContext? context,
                                               CancellationToken cancellationToken)
    {
        // It's good practice to flush headers before potentially long serialization.
        // This ensures headers are sent promptly, preventing some timeout issues.
        // See: https://ayende.com/blog/190989-A/
        await stream.FlushAsync(cancellationToken).ConfigureAwait(false);

        if (_bufferedData != null)
        {
            // If data was pre-buffered (e.g., by TryComputeLength), write it
            await stream.WriteAsync(_bufferedData, 0, _bufferedData.Length, 
                                    cancellationToken).ConfigureAwait(false);
        }
        else
        {
            // Otherwise, serialize directly to the output stream
            await _serializer.SerializeAsync(stream, _value, cancellationToken)
                             .ConfigureAwait(false);
        }
    }

    // Fallback for older frameworks or if the CancellationToken overload isn't called
    protected override Task SerializeToStreamAsync(Stream stream, 
                                               TransportContext? context)
    {
        // Delegate to the CancellationToken-enabled overload
        return SerializeToStreamAsync(stream, context, CancellationToken.None);
    }

    protected override bool TryComputeLength(out long length)
    {
        // Attempt 1: Ask the serializer if it can compute length easily.
        if (_serializer.TryComputeLength(_value, out long calculatedLength))
        {
            length = calculatedLength;
            return true;
        }

        // Attempt 2 (Use with caution due to memory/performance impact): 
        // Serialize to a temporary MemoryStream to get length.
        // This buffers the entire object in memory. Not ideal for large objects.
        // if (_bufferedData == null)
        // {
        //     using (var tempStream = new MemoryStream())
        //     {
        //         // Note: This makes TryComputeLength effectively synchronous
        //         // if the SerializeAsync blocks or is awaited with .Result.
        //         // This is generally an anti-pattern for TryComputeLength.
        //         _serializer.SerializeAsync(tempStream, _value, 
        //                                    CancellationToken.None)
        //                    .GetAwaiter().GetResult(); 
        //         _bufferedData = tempStream.ToArray();
        //     }
        // }
        // length = _bufferedData.Length;
        // return true;

        // If length cannot be determined without significant overhead or buffering,
        // return false. HttpClient will use chunked transfer encoding.
        length = -1; // Or 0, depending on preference for "unknown"
        return false;
    }
}

Key points in CustomBinaryContent<T>:

• The constructor takes the value, serializer, and Content-Type string (which it parses).
• SerializeToStreamAsync calls stream.FlushAsync() first to ensure headers are sent promptly. This critical step can prevent certain types of request hangs, as detailed in various networking discussions (e.g., by Ayende Rahien). Then, it uses the provided IBinarySerializer to write the object to the stream.
• The ConfigureAwait(false) is used on await calls to avoid capturing the synchronization context, a best practice in library and non-UI code.
• TryComputeLength first asks the IBinarySerializer. The commented-out section shows how one could buffer to compute length, but it highlights the downsides. Returning false is often the best choice for true streaming or when length calculation is costly.

3. Using the Custom HttpContent with HttpClient

Here’s how you might implement a concrete serializer (for demonstration purposes, not a real binary one) and use CustomBinaryContent<T>:

First, an example IBinarySerializer implementation. In a real application, this would use a library like Google.Protobuf or MessagePack-CSharp.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

// Example concrete serializer (uses simple text for demonstration).
// Replace with actual binary serialization logic (e.g., Protobuf, MessagePack).
public class MyDemoBinarySerializer : IBinarySerializer
{
    public async Task SerializeAsync<TItem>(Stream stream, TItem item, 
                                CancellationToken cancellationToken = default)
    {
        // This is NOT binary, just for demonstration of the HttpContent structure.
        // In a real scenario, use your binary serializer here.
        // e.g., for Protobuf: ((Google.Protobuf.IMessage)item).WriteTo(stream);
        // e.g., for MessagePack: 
        // await MessagePack.MessagePackSerializer.SerializeAsync(stream, item, 
        //                                        cancellationToken: cancellationToken);
        using var writer = new StreamWriter(stream, System.Text.Encoding.UTF8, 
                                            bufferSize: 1024, leaveOpen: true);
        await writer.WriteAsync($"Serialized: {item?.ToString()}")
                    .ConfigureAwait(false);
        // Crucial: Ensure data is flushed from StreamWriter's internal buffer
        // to the underlying stream before SerializeToStreamAsync returns.
        await writer.FlushAsync().ConfigureAwait(false); 
    }

    public bool TryComputeLength<TItem>(TItem item, out long length)
    {
        // Placeholder: Most real binary serializers might not offer a trivial
        // TryComputeLength without serializing. Protobuf has CalculateSize().
        // For this demo, we'll say we don't know.
        length = -1;
        return false;
    }
}

Now, an ApiClient class demonstrates using our custom content with HttpClient.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

public class ApiClient
{
    private readonly HttpClient _httpClient;
    private readonly IBinarySerializer _serializer;

    public ApiClient(HttpClient httpClient)
    {
        _httpClient = httpClient ?? throw new ArgumentNullException(nameof(httpClient));
        // In a real app, inject IBinarySerializer or choose a specific one.
        _serializer = new MyDemoBinarySerializer(); 
    }

    public async Task SendRpcRequestAsync(MyRpcRequest requestData, 
                                          string endpointUrl)
    {
        var content = new CustomBinaryContent<MyRpcRequest>(
            requestData,
            _serializer,
            // Example custom MIME type. Use standard types if applicable
            // (e.g., "application/x-protobuf", "application/x-msgpack").
            "application/x-my-custom-binary" 
        );

        Console.WriteLine($"Sending request to {endpointUrl} with " +
                          $"Content-Type: {content.Headers.ContentType}");

        // Example: POST request
        HttpResponseMessage response = await _httpClient.PostAsync(
            endpointUrl, 
            content
        ).ConfigureAwait(false);

        response.EnsureSuccessStatusCode(); // Throws if not a success code.
        
        Console.WriteLine("RPC request sent successfully. " + 
                          $"Status: {response.StatusCode}");
        // Process the response (deserialization of response content is a 
        // separate concern, potentially using another HttpContent or helpers).
    }
}

This ApiClient creates an instance of CustomBinaryContent<MyRpcRequest> and sends it via HttpClient.PostAsync. The Content-Type is set to a custom MIME type, but you should use standard types like application/x-protobuf or application/x-msgpack if your binary format is one of these.

Essential Considerations and Best Practices

• Asynchronous All The Way: Always use async and await for I/O operations within SerializeToStreamAsync to prevent blocking threads and maintain application responsiveness. Use ConfigureAwait(false) diligently in library-like code.
• Content-Length Accuracy: If TryComputeLength returns true, the provided length must be absolutely accurate. Incorrect lengths can lead to truncated requests or server timeouts. If in doubt, return false and let the server handle chunked encoding.
• Streaming Large Payloads: For very large objects, avoid buffering the entire serialized payload in memory just to calculate its length (i.e., inside TryComputeLength). Design SerializeToStreamAsync to write data incrementally and have TryComputeLength return false.
• Header Finalization with stream.FlushAsync(): Calling await stream.FlushAsync().ConfigureAwait(false); at the beginning of SerializeToStreamAsync is crucial. It ensures that HTTP headers are sent before the (potentially lengthy) serialization process begins.
• Error Handling: Implement robust error handling within your serialization logic. Exceptions during serialization should typically be allowed to propagate unless specific handling is required.
• CancellationToken: Honor the CancellationToken passed to SerializeToStreamAsync (and to your underlying serializer if it supports cancellation) to allow the operation to be gracefully cancelled.
• Immutability: While HttpContent instances are often single-use with HttpClient, design your content to be effectively immutable after creation if there’s any chance of reuse or retry logic that might re-read it (though retries often create new HttpRequestMessage instances).

Alternative Approaches

While custom HttpContent offers maximum control, consider these alternatives:

1. ByteArrayContent / StreamContent with Pre-serialization:
   • Approach: Serialize your object to a byte[] or MemoryStream first, then wrap it with ByteArrayContent or StreamContent.
   • Pros: Simpler to implement if you already have the serialized byte array or stream. Content-Length is handled automatically by these types.
   • Cons: Requires buffering the entire serialized object in memory, which is inefficient for large objects and negates true streaming benefits. You still need to set the ContentType header manually on the content object’s Headers property.
2. Full RPC Frameworks (e.g., gRPC):
   • Approach: Use a comprehensive framework like gRPC, which is built on HTTP/2 and typically uses Protobuf by default.
   • Pros: Provides a complete RPC solution including code generation, efficient transport, and built-in serialization/deserialization. Optimized for performance and various communication patterns (unary, streaming).
   • Cons: Introduces a larger framework and specific architectural patterns. May be overkill if you only need to send a binary payload over a standard HTTP/1.1 POST to a non-gRPC endpoint, or if the server expects a different binary protocol.

Conclusion

Implementing a custom HttpContent derivative in .NET is a powerful technique for sending binary RPC payloads efficiently using HttpClient. It grants developers precise control over the serialization process, enabling direct streaming of binary data and proper handling of HTTP headers like Content-Type and Content-Length. By following best practices such as asynchronous implementation, careful length calculation (or opting for chunked encoding), and prompt header flushing, you can build high-performance, robust client applications that communicate effectively with binary RPC services. This approach is invaluable when standard content types fall short and you need optimal performance and control for your binary data exchange over HTTP.