In this article we describe the process of configuring Containerd client to connect to a Sonatype Nexus container registry proxy/mirror. This allows for faster, and secure pulling of container images since Nexus will cache frequently used images thus reducing dependency and direct hist on external registries. Another practical use case of Nexus caching is in air-gapped network infrastructures, which are common in corporate environments. In this kind of setup, Nexus instance will have internet access ( i.e access to external container registries), while Kubernetes nodes remain isolated and disconnected from the internet.

We have created this guide to provide a step-by-step instructions on integrating Nexus registry proxy with containerd.

Pre-requisites

The guide has been created with the following assumptions:

1. You already have Nexus sever installed and configured – (We will consider creating a dedicated post)
2. Proxied container registries configured on Nexus. See Docker hub registry example.
3. Containerd is installed and running
4. Machines running containerd have connectivity to Nexus server

Gather Necessary Information

Before you get started, collect the following information from your setup.

• Version of containerd installed on your machines

$ containerd --version
containerd github.com/containerd/containerd 1.7.12 

• Sonatype Nexus Registry Proxy FQDN (e.g., nexus.example.net), or IP address
• Confirm whether Nexus is configured with SSL(HTTPS) or HTPP
• List of container registries to proxy (e.g., docker.io, ghcr.io, etc.).

Registry Configuration on Containerd

Configuring registries for containerd and its clients (ctr, crictl, or kubectl) is done by specifying a hosts.toml file for each desired registry host in a configuration directory.

The system-wide configuration file for containerd is /etc/containerd/config.toml. Confirm that the following lines exist in the config:

• In containerd 1.x

version = 2

[plugins."io.containerd.grpc.v1.cri".registry]
   config_path = "/etc/containerd/certs.d"

• In containerd 2.x

version = 3

[plugins."io.containerd.cri.v1.images".registry]
   config_path = "/etc/containerd/certs.d"

Creating registry host namespace

A registry host namespace is a path to the hosts.toml file specified by the registry host name, or ip address, and an optional port identifier. When making a pull request for an image the format is typically as follows:

pull [registry_host_name|IP address][:port][/v2][/org_path]<image_name>[:tag|@DIGEST]

The registry host namespace portion is [registry_host_name|IP address][:port]. Example tree for docker.io:

$ tree /etc/containerd/certs.d
/etc/containerd/certs.d
└── docker.io
    └── hosts.toml

In docker example, create the directory if doesn’t exist already

sudo mkdir -p /etc/containerd/certs.d/docker.io

Create hosts.toml file inside the created docker.io directory

sudo vim /etc/containerd/certs.d/docker.io/hosts.toml

Configure nexus registry access like below:

server = "https://docker.io"
[host."https://nexux.example.net"]
capabilities = ["pull", "resolve"]

If the nexus server is using a self-signed SSL certificate, you can choose to bypass the TLS verification:

server = "https://docker.io"
[host."https://nexux.example.net"]
capabilities = ["pull", "resolve"]
skip_verify = true

If pushing to the registry is allowed, then passing push to the capabilities list will be valid.

capabilities = ["pull", "resolve", "push"]

Using custom SSL CA file

server = "https://docker.io"
[host."https://192.168.1.12:8080"]
capabilities = ["pull", "resolve"]
ca = "nexus-proxy.crt" #  Or absolute path /etc/containerd/certs.d/docker.io/exus-proxy.crt

Setup Default Mirror for All Registries

It’s possible to have a default configuration for mirror regardless of the intended registry. The upstream registry will automatically be used after all defined hosts have been tried.

This is achieved by placing hosts.toml in the /etc/containerd/certs.d/_default directory.

sudo mkdir /etc/containerd/certs.d/_default
sudo vim /etc/containerd/certs.d/_default/hosts.toml

Edit the file and add correct configurations for nexus registries.

Explanation of used fields in hosts.toml

For each registry host namespace directory in your registry config_path you may include a hosts.toml configuration file. The following root level toml fields apply to the registry host namespace:

server field

The server field specifies the default server for this registry host namespace. If server is not specified then the image’s registry host namespace will automatically be used.

server = "https://docker.io"

capabilities field

This optional setting specifies the operations a host can perform. It represents the registry host’s capabilities, and you should include only the values that are relevant.

capabilities =  ["pull", "resolve", "push"]

• pushing – This capability should only be performed on an upstream source, not on a mirror.
• resolve – A process of converting a name into a digest, should be treated as a trusted operation. It must be performed only by a trusted host or, preferably, by a secure process that can verify the provenance of the mapping.

A public mirror should never be trusted to perform a resolve action.

Registry Type	Pull	Resolve	Push
Public Registry	yes	yes	yes
Private Registry	yes	yes	yes
Public Mirror	yes	no	no
Private Mirror	yes	yes	no

ca field

The ca (Certificate Authority Certification) can be configured as a path or an array of paths, each pointing to a CA file used for authenticating with the registry namespace.

ca = "/etc/certs/mirror.pem"

Or

ca = ["/etc/certs/test-1-ca.pem", "/etc/certs/special.pem"]

This setup allows the system to use multiple Certificate Authority (CA) files for authentication with the registry namespace.

client field

The client certificates are configured as follows

• a path:

client = "/etc/certs/client.pem"

• an array of paths:

client = ["/etc/certs/client-1.pem", "/etc/certs/client-2.pem"]

• an array of pairs of paths:

client = [["/etc/certs/client.cert", "/etc/certs/client.key"],["/etc/certs/client.pem", ""]]

skip_verify field

Setting this to true skips verification of the registry’s certificate chain and hostname. However, this should only be used for testing purposes or alongside other secure methods of verifying connections. By default, this is set to false.

skip_verify = false

override_path field

Use the override_path setting to specify that the host’s API root endpoint is defined in the URL path rather than adhering to the standard API specification. This is particularly useful for non-compliant OCI registries that lack the /v2 prefix. By default, this option is set to false.

override_path = true

host field(s) (in the toml table format)

Entries like [host]."https://namespace" and [host]."http://namespace" in the hosts.toml configuration define registry namespaces that act as substitutes for the default registry host namespace. These hosts are often referred to as mirrors, as they may store copies of container images and artifacts intended for retrieval from the default registry.

Each mirror namespace is configured similarly to the default registry namespace, with the key difference being that the server is not explicitly defined in the host description; instead, it is embedded within the namespace itself. Below are a few examples of how to configure host mirror namespaces for this registry host namespace:

[host."https://mirror.registry"]
  capabilities = ["pull"]
  ca = "/etc/certs/mirror.pem"
  skip_verify = false
  [host."https://mirror.registry".header]
    x-custom-2 = ["value1", "value2"]

[host."https://mirror-bak.registry/us"]
  capabilities = ["pull"]
  skip_verify = true

[host."http://mirror.registry"]
  capabilities = ["pull"]

[host."https://test-1.registry"]
  capabilities = ["pull", "resolve", "push"]
  ca = ["/etc/certs/test-1-ca.pem", "/etc/certs/special.pem"]
  client = [["/etc/certs/client.cert", "/etc/certs/client.key"],["/etc/certs/client.pem", ""]]

[host."https://test-2.registry"]
  client = "/etc/certs/client.pem"

[host."https://test-3.registry"]
  client = ["/etc/certs/client-1.pem", "/etc/certs/client-2.pem"]

[host."https://non-compliant-mirror.registry/v2/upstream"]
  capabilities = ["pull"]
  override_path = true

Verify it’s working

After configurations of registry mirrors, you can restart containerd

sudo systemctl restart containerd

Try pull an image from the nexus registry

sudo ctr images pull nexus.example.net/nginx:latest

Or run sample application deployment if using kubernetes

kubectl run nginx --image=nginx:latest

The same can also be done from yaml manifest

$ vim rocky-linux.yaml
apiVersion: v1
kind: Pod
metadata:
  name: rocky-9-test
spec:
  containers:
    - name: rocky-linux-9
      image: rockylinux:9
      command: ["/bin/bash", "-c", "while true; do sleep 3600; done"]

Then apply the manifest to create kubernetes resources:

kubectl apply -f  rocky-linux.yaml

Login to nexus and navigate to the correct registry folder.

Confirm the used images in your container creation process were cached in Nexus.

Conclusion

In conclusion, by setting up containerd to use the Sonatype Nexus Registry Proxy you save on bandwidth, have faster deployments, and avoid image duplications using caching. Additionally, the approach of configuring a trusted, and efficient proxy for container registries enhances security and efficiency for modern and scalable containerized infrastructures.