adllm Insights logo adllm Insights logo

Secure Egress: mTLS Between Linkerd and External Services with Custom CAs

Published on by The adllm Team. Last modified: . Tags: Linkerd mTLS Service Mesh Kubernetes Custom CA Security Egress

Linkerd, a lightweight and security-focused service mesh, excels at providing transparent mutual TLS (mTLS) for traffic within your Kubernetes cluster. However, many applications also need to securely communicate with services running outside the cluster – external databases, partner APIs, or legacy systems. When these external services require mTLS and utilize certificates signed by a custom Certificate Authority (CA) not part of public trust stores, configuring Linkerd’s sidecar proxies to establish trust becomes a critical task.

This comprehensive guide walks experienced engineers through the process of configuring mTLS between Linkerd-meshed applications and external services secured by custom CAs. We’ll cover the core concepts, provide step-by-step instructions with practical code examples, and discuss common pitfalls and debugging techniques.

The Challenge: External mTLS with Custom CAs

By default, Linkerd’s sidecar proxy (linkerd-proxy), when acting as a client to an external service, validates the server’s certificate against a standard set of well-known public CAs (similar to a web browser). If the external service presents a certificate signed by your organization’s internal CA or any other private CA, the Linkerd proxy will fail the TLS handshake, unable to verify the server’s identity.

To resolve this, we must explicitly instruct the Linkerd proxy to trust this specific custom CA. This involves making the custom CA certificate available to the proxy within the application’s pod and ensuring the proxy uses it when establishing an mTLS connection to the designated external service.

Core Concepts

  • mTLS (Mutual TLS): Both client and server authenticate each other using X.509 certificates. The client (Linkerd proxy) verifies the external server’s certificate, and the external server verifies the client’s certificate (if required by the server; this guide focuses on the Linkerd proxy trusting the external server).
  • Linkerd Proxy: The sidecar injected into your application pods. It intercepts outbound traffic and can originate TLS connections.
  • Custom CA: A private Certificate Authority whose root certificate is not in standard public trust stores.
  • System Trust Store: An operating system-level repository of trusted CA certificates. Many applications, including proxies, can be configured to use this store.

Solution Overview: Leveraging the System Trust Store

A robust and common pattern to enable the Linkerd proxy to trust a custom CA for egress connections is to add the custom CA certificate to the pod’s system trust store. The Linkerd proxy, like many network clients, often respects the system’s CA certificates. This involves:

  1. Storing the Custom CA Certificate: Securely store your custom CA certificate (PEM-encoded) in a Kubernetes Secret.
  2. Mounting the Secret: Mount this Secret as a volume into your application pod.
  3. Using an Init Container: An initContainer runs before your main application container. It copies the custom CA certificate from the mounted volume into the appropriate system directory for CA certificates and updates the system trust store.
  4. Verification: The Linkerd proxy in the same pod will then (typically) recognize this newly trusted CA when initiating TLS connections to external services that present certificates signed by it.

Step-by-Step Configuration

Let’s walk through an example. Assume you have an external service my-external-service.example.com:443 that requires mTLS and uses a certificate signed by your organization’s custom CA.

1. Prepare Your Custom CA Certificate

Ensure you have your custom CA certificate in PEM format. For this guide, let’s assume its content is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

-----BEGIN CERTIFICATE-----
MIIEXTCCAsWgAwIBAgIJAO0OVoVi1ABRMA0GCSqGSIb3DQEBCwUAMIGPMQswCQYD
VQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5j
aXNjbzEZMBcGA1UECgwQTXkgT3JnYW5pemF0aW9uMRswGQYDVQQLDBJEZXZFbmdp
bmVlcmluZyBUZWFtMR0wGwYDVQQDDBRteWN1c3RvbS1jYS5leGFtcGxlLmNvbTAe
Fw0yNTAxMDEwMDAwMDBaFw0zNTAxMDEwMDAwMDBaMIGPMQswCQYDVQQGEwJVUzET
MBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzEZMBcG
A1UECgwQTXkgT3JnYW5pemF0aW9uMRswGQYDVQQLDBJEZXZFbmdpbmVlcmluZyBU
ZWFtMR0wGwYDVQQDDBRteWN1c3RvbS1jYS5leGFtcGxlLmNvbTCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBAN3gXg5P7sr1j9M5vzjOaV8CVGmKM8s8N0mC
OUzN8b3yGqZasSSK8HwOKnBDN+2x0JBhmVddxTtSJfNIh3rLo8rsbsA93ZMyN0Yw
7SOj2UuP2+h5Y0V78YQfAIFrgMVjoMM0E62h4JqLBE5gKbjAbYx2LgD5AR8hYWW1
eQnUD8EDP1P2EMnd8tyejftP7HLsZTnCVWPLX4U7Qhhpb8pXGgm248882Jm0gD5c
G0WjD+p0NMpPSqWH8tav7XbPZGx2kKZepe4N2yd7MvYOShJUfPPM04L6nvoShYbg
G1iDB3dCt8O9z6bCx0UjV3z7HBao2nED6XlAP0xev8kHhWfD8wMCAwEAAaOBjTCB
ijAdBgNVHQ4EFgQU8ADGHYGt0DpaOBAUND6WfQVROLLgwgYcGA1UdIwSBfjB8gBTw
AMYdga3QOlo4EBA0PpZ9BU0S0LihaKRnMGUMQswCQYDVQQGEwJVUzETMBEGA1UE
CAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzEZMBcGA1UECgwQ
TXkgT3JnYW5pemF0aW9uMRswGQYDVQQLDBJEZXZFbmdpbmVlcmluZyBUZWFtMR0w
GwYDVQQDDBRteWN1c3RvbS1jYS5leGFtcGxlLmNvbYIJAO0OVoVi1ABRMAwGA1Ud
EwEB/wQCMAAwDQYJKoZIhvcNAQELBQADggEBAEW2x4o9OUgVAnOQtJcDEq5nzzjV
iK6mE9OMzhOcqV+nJBNkQfJblR8a0vA8yPMhzNLCsT2UDLTCSTToOTPYO0v5tFp8
bsgPC5gRdgHZrAPM396GfPfF8bBqE8SK3GZzYdIcNUTEhUXUJbLoAmQg5kEuQnCs
vG83qi8Cwc3hB9keg16YAe1W/KHP7oW0yY2i05QyULUAGYyAqGuqgHnLNRr2YZZU
u082r2V2PvgQzh9gXk3hs5NVSUCrkIZ4BNKe6DS8SNj9v8oX0qjrFsNqL1wU7Yh0
q3s+hV3yA0+jPsUPGFoDNUFXdLTlZIAp5DKWi862P9QPiKmqtM9t3R2n+oU=
-----END CERTIFICATE-----
# Note: This is a self-signed placeholder. Replace with your actual CA cert.

Save this content as custom-ca.pem.

2. Create a Kubernetes Secret

Create a Kubernetes Secret to store this CA certificate. This makes it securely available to your pods.

1
2
3

kubectl create secret generic custom-ca-secret \
  --from-file=ca.crt=./custom-ca.pem \
  -n your-namespace

Replace ‘your-namespace’ with the namespace of your application```

This command creates a secret named custom-ca-secret with a key ca.crt containing the PEM content.

3. Deploy Your Application with an Init Container

Now, define your application’s Deployment. We’ll include an initContainer that uses an image containing the ca-certificates package (like alpine or ubuntu) to update the pod’s trust store.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78

# app-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-consuming-external-service
  namespace: your-namespace # Ensure this matches the secret's namespace
  labels:
    app: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
      annotations:
        linkerd.io/inject: enabled # Enable Linkerd sidecar injection
    spec:
      initContainers:
      - name: update-ca-certificates
        # Use an image that has 'update-ca-certificates' command
        # Alpine is small and has 'ca-certificates' package
        # but requires apk add. Debian/Ubuntu based images often have it.
        # Let's use a debian-base for simplicity of pre-installed tool.
        image: debian:stable-slim # Contains update-ca-certificates
        command:
        - /bin/sh
        - -c
        - |
          set -e;
          echo "Copying custom CA to system store...";
          cp /mnt/custom-ca/ca.crt \
             /usr/local/share/ca-certificates/custom-external-ca.crt;
          echo "Updating CA certificates...";
          update-ca-certificates;
          echo "CA update complete.";
        volumeMounts:
        - name: custom-ca-volume
          mountPath: /mnt/custom-ca
          readOnly: true
        securityContext:
          # Required if your init container needs root to write to
          # /usr/local/share/ca-certificates or run update-ca-certificates
          runAsUser: 0
      containers:
      - name: my-app
        image: curlimages/curl:latest # Example app using curl
        # Command to periodically try connecting to the external service
        # Replace with your actual application's command and logic
        command: ["/bin/sh", "-c"]
        args:
        - |
          while true; do
            echo "Attempting to connect to external service...";
            curl -v --cacert /mnt/custom-ca/ca.crt \
                 https://my-external-service.example.com;
            # The --cacert above is for curl to explicitly trust.
            # Linkerd proxy should automatically use system CAs.
            # For testing Linkerd's trust, you might remove --cacert
            # if the proxy uses the updated system store effectively.
            #
            # A more direct test of Linkerd proxy (if it uses system CAs):
            # curl -v https://my-external-service.example.com;
            # This relies on Linkerd proxy handling TLS and using the
            # updated system trust store.
            echo "Sleeping for 30 seconds...";
            sleep 30;
          done
        volumeMounts:
        - name: custom-ca-volume # Optional for app, but good for curl testing
          mountPath: /mnt/custom-ca
          readOnly: true
      volumes:
      - name: custom-ca-volume
        secret:
          secretName: custom-ca-secret

Key points in this deployment:

  • linkerd.io/inject: enabled: Ensures the Linkerd proxy is injected.
  • initContainers:
    • The update-ca-certificates init container runs first.
    • It mounts the custom-ca-secret via custom-ca-volume.
    • It copies ca.crt to /usr/local/share/ca-certificates/custom-external-ca.crt.
    • It runs update-ca-certificates, which rebuilds the system’s list of trusted CAs.
    • securityContext.runAsUser: 0 is often needed for update-ca-certificates and writing to system directories. Ensure your PodSecurityPolicies/Standards allow this if applicable.
  • Application Container (my-app):
    • In this example, it uses curlimages/curl to attempt connections to https://my-external-service.example.com.
    • The curl command with --cacert /mnt/custom-ca/ca.crt explicitly tells curl to use the custom CA. This is useful for direct testing from the application container.
    • For the Linkerd proxy to handle the mTLS using the updated system trust store, the curl command without --cacert would rely on the transparent mTLS provided by Linkerd. If the Linkerd proxy successfully uses the updated system CA store, this “naked” curl call should succeed.
  • Volumes: Defines custom-ca-volume to provide the ca.crt from custom-ca-secret.

After applying this deployment (kubectl apply -f app-deployment.yaml -n your-namespace), the Linkerd proxy in the my-app pods should be able to establish mTLS connections to my-external-service.example.com by trusting its custom CA.

Alternative: Using LINKERD2_PROXY_TLS_TRUST_ANCHORS_PEM

Linkerd documentation also specifies that the LINKERD2_PROXY_TLS_TRUST_ANCHORS_PEM environment variable can be set on the proxy container to provide the PEM-encoded CA certificate(s) directly. This is an alternative to modifying the system trust store.

You can set this environment variable for the injected linkerd-proxy using the config.linkerd.io/proxy-env annotation on the pod spec:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Partial pod spec template
# ...
    metadata:
      annotations:
        linkerd.io/inject: enabled
        # This annotation is tricky for multi-line PEMs.
        # The PEM content needs to be properly formatted as a single string,
        # often by replacing newlines with '\n' or using YAML block scalars
        # if the annotation processor supports it.
        # Example with a very short, single-line placeholder for brevity:
        config.linkerd.io/proxy-env: |
          LINKERD2_PROXY_TLS_TRUST_ANCHORS_PEM=\
          "-----BEGIN CERTIFICATE-----\nMIIC...\n-----END CERTIFICATE-----"
# ...

Challenges with proxy-env for PEMs:

  • Multi-line Strings: Kubernetes annotations are flat strings. Injecting a multi-line PEM certificate directly into an annotation value in YAML can be cumbersome and error-prone. You might need to process the PEM into a single-line string with literal \n characters.
  • Size Limits: Annotations have size limits, which might be an issue for large CA bundles.

For these reasons, the init container method modifying the system trust store is often more robust for larger or multiple CA certificates. If your CA PEM is small and you can manage the string formatting, proxy-env offers a more direct configuration approach without needing an init container with root privileges.

Verification and Debugging

  1. Check Pod Logs:

    • Inspect the init container logs:
      1

      kubectl logs <your-pod-name> -n your-namespace -c update-ca-certificates
      Ensure it completed successfully.
    • Inspect your application container logs:
      1

      kubectl logs <your-pod-name> -n your-namespace -c my-app
      Look for successful connection attempts from curl.
    • Inspect the Linkerd proxy logs:
      1

      kubectl logs <your-pod-name> -n your-namespace -c linkerd-proxy
      Look for messages related to TLS handshakes or certificate validation errors. Increase verbosity if needed (e.g., by setting config.linkerd.io/proxy-log-level: debug annotation).

  2. linkerd tap: Use linkerd tap to observe traffic from your application pod:

    1
2

    linkerd tap deploy/my-app-consuming-external-service -n your-namespace \
  --to svc/my-external-service.example.com # Adjust target if needed

    This can show you details about the TLS handshake, including the server identity used.

  3. openssl s_client from within the pod: If your application container (or a debug sidecar) has openssl, you can test manually:

    1
2
3
4
5
6
7
8
9

    kubectl exec -it <your-pod-name> -n your-namespace -c my-app -- /bin/sh

# Inside the pod:
# Test with the CA explicitly provided to openssl
openssl s_client -connect my-external-service.example.com:443 \
                 -CAfile /mnt/custom-ca/ca.crt

# Test relying on system trust store (if openssl uses it)
openssl s_client -connect my-external-service.example.com:443

Common Pitfalls

  • Incorrect CA Certificate: Ensure the exact root CA (or intermediate CAs if the server doesn’t send the full chain and the root is what you trust) that signed the external service’s certificate is used.
  • Namespace Mismatch: The Secret and the Deployment using it must be in the same Kubernetes namespace.
  • Permissions for Init Container: The init container might need root privileges (runAsUser: 0) to write to system CA directories and run update-ca-certificates.
  • External Service Name Mismatch: The hostname used by your application to connect (e.g., my-external-service.example.com) must match a Subject Alternative Name (SAN) or the Common Name (CN) in the external service’s certificate.
  • Linkerd Proxy Not Using System Store: While common, verify that your Linkerd version and configuration indeed lead the proxy to respect the updated system CA list. If not, the LINKERD2_PROXY_TLS_TRUST_ANCHORS_PEM environment variable method becomes more critical.
  • Firewall/NetworkPolicy Issues: Ensure network connectivity to the external service is allowed from your pod.

Conclusion

Securing egress traffic to external services with mTLS is a vital aspect of a comprehensive security posture. By configuring your Linkerd-meshed applications to trust custom Certificate Authorities, you can extend Linkerd’s powerful security benefits beyond your cluster boundaries. The init container pattern provides a robust way to manage custom CA trust for the Linkerd proxy, ensuring that your applications can reliably and securely communicate with external dependencies. Always refer to the latest Linkerd official documentation for features and configuration specifics related to your Linkerd version.