Nginx is an open source, and high-performance web server application that can do load balancing, caching, reverse proxy, and more. Nginx is known to be reliable, fast, efficient, and stable modern web server. It can handle thousands of requests in seconds without consuming much hardware resources in the server. You can use Nginx as:

• Web Server: To server your website static and dynamic content.
• Reverse Proxy: It acts as a gateway between servers and clients. Example is Node.js and Python backend applications proxying.
• Load Balancer: It does the distribution of traffic coming in across multiple backend servers for efficiency and better resource utilization
• Caching: It stores content that’s accessed more frequently to increase response times

Nginx as TCP Load Balancer

Nginx can also be used as a TCP Load Balancer. In this mode it distributes the incoming TCP traffic on configured port across defined backend servers. This is commonly used to load balance traffic for services like mail servers, databases, logging servers, among others.

In this article we shall configure Nginx as Load balancer to our Kubernetes API servers (control plane) listening on port 6443.

Set up Nginx as Kubernetes API Load Balancer

Follow the following steps to set up Nginx as a Load Balancer to your Kubernetes API server.

1 – Install Nginx

Begin with the installation of Nginx web server in your system.

Install Nginx on Ubuntu / Debian

Ensure the necessary dependencies are installed before adding the repository.

sudo apt update
sudo apt install curl gnupg2 ca-certificates lsb-release ubuntu-keyring

Import GPG key used in package signing.

curl  -fsSL https://nginx.org/keys/nginx_signing.key|sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/nginx.gpg

Enable Nginx official APT repository

• On Ubuntu systems

echo "deb http://nginx.org/packages/mainline/ubuntu `lsb_release -cs` nginx"|sudo tee /etc/apt/sources.list.d/nginx.list

• On Debian systems

echo "deb http://nginx.org/packages/mainline/debian `lsb_release -cs` nginx"|sudo tee /etc/apt/sources.list.d/nginx.list

Install nginx package using apt command

sudo apt update && sudo apt install nginx

Install Nginx on Enterprise Linux systems – Rocky / AlmaLinux / CentOS Stream

Disable Nginx AppStream repository.

sudo dnf module disable -y nginx

Add official Nginx RPM repository which provide latest packages.

sudo tee /etc/yum.repos.d/nginx.repo<<EOF
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/rhel/\$releasever/\$basearch/
gpgcheck=0
enabled=1
EOF

Update DNF package index

$ sudo dnf makecache
nginx repo                                                                                                                                                                                                                                         670 kB/s |  34 kB     00:00
Rocky Linux 9 - BaseOS                                                                                                                                                                                                                              20 kB/s | 4.1 kB     00:00
Rocky Linux 9 - AppStream                                                                                                                                                                                                                           22 kB/s | 4.5 kB     00:00
Rocky Linux 9 - Extras                                                                                                                                                                                                                              14 kB/s | 2.9 kB     00:00
Metadata cache created.

Install Nginx from the repository added:

sudo dnf -y install nginx

Hit y key when asked to confirm installation of nginx package.

Dependencies resolved.
===================================================================================================================================================================================================================================================================================
 Package                                                               Architecture                                                Version                                                                    Repository                                                      Size
===================================================================================================================================================================================================================================================================================
Installing:
 nginx                                                                 x86_64                                                      1:1.26.1-2.el8.ngx                                                         nginx                                                          962 k
Installing dependencies:
 compat-openssl11                                                      x86_64                                                      1:1.1.1k-4.el9_0                                                           appstream                                                      1.5 M

Transaction Summary
===================================================================================================================================================================================================================================================================================
Install  2 Packages

Total size: 2.4 M
Installed size: 6.9 M
Is this ok [y/N]: y

2 – Configure Nginx as Load Balancer for Kubernetes API

Open the default nginx configuration file.

sudo vim /etc/nginx/nginx.conf

Add the following line after the ending } – “add to very end of file”

include /etc/nginx/tcpconf.d/*.conf;

Here is a sample complete configuration file after adding the line.

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

include /etc/nginx/tcpconf.d/*.conf;

Create a directory that will store Nginx configuration files for TCP load balancing.

 sudo mkdir /etc/nginx/tcpconf.d

Now we can create the configuration file specific to Kubernetes Load balancing

sudo vim /etc/nginx/tcpconf.d/kubernetes_tcp.conf

These are the contents added to the configuration file; adjust to your ideal environment setup.

stream {
    upstream kube_apiservers {
        least_conn;
        server 10.25.25.11:6443 weight=1 max_fails=3 fail_timeout=10s;
        server 10.25.25.12:6443 weight=1 max_fails=3 fail_timeout=10s;
        server 10.25.25.13:6443 weight=1 max_fails=3 fail_timeout=10s;
    }
    server {
        listen 6443;
        proxy_pass kube_apiservers;
        proxy_timeout 10m;
        proxy_connect_timeout 1m;
    }
}

Validate Nginx configuration file to confirm there are no syntax errors:

$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Enable nginx service and restart after making the changes.

sudo systemctl enable nginx && sudo systemctl restart nginx

Check status of the service to ensure it’s running.

systemctl status nginx

3 – Allowing the port in firewall

For RHEL based systems with firewalld

sudo firewall-cmd --add-port=6443/tcp --permanent
sudo firewall-cmd --reload

For Debian based systems with UFW:

sudo ufw allow 6443/tcp
sudo ufw reload

4 – Labeling port on SELinux

If you have SELinux in enforcing operation mode, label port 6443 for TCP traffic with Nginx by running the following command:

sudo dnf install policycoreutils-python-utils python3-semanage
sudo semanage port -a -t http_port_t -p tcp 6443

Explanations to the options used:

• -a: Used to add a new port
• -t http_port_t: Specify SELinux type for HTTP traffic, which allows NGINX to use this port.
• -p tcp: Specifies the protocol used

To validate run:

sudo semanage port -l | grep 6443

Also allow Nginx http server to make network connections.

sudo setsebool -P httpd_can_network_connect 1

Verify the setting, you should see --> on

sudo getsebool httpd_can_network_connect
httpd_can_network_connect --> on

Restart nginx service

sudo systemctl restart nginx

You can identify any denied operations by checking audit logs if you encounter issues with SELinux

sudo ausearch -m avc -ts recent

5 – Test Kubernetes port

Check if nginx is bound on port 6443

$ sudo ss -tunelp|grep 6443
tcp   LISTEN 0      511          0.0.0.0:6443      0.0.0.0:*    users:(("nginx",pid=53534,fd=7),("nginx",pid=53533,fd=7),("nginx",pid=53532,fd=7)) ino:351840 sk:1002 cgroup:/system.slice/nginx.service <->

If you have telnet client you can use it to validate connection:

$ telnet  127.0.0.1 6443
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.

From this point you can proceed to bootstrap your Kubernetes cluster if doing a new installation. Remember to define an A record and map DNS name to the load balancer IP address for easy updating or changing of load balancer at later stage. Avoid binding your Kubernetes control plan endpoint to an IP address.

Offloading Kubernetes administration tasks enables you to focus on growing your business and brand. Allow CloudSpinx Engineers to handle your Kubernetes clusters, and you give attention to the business growth!