K3s is a lightweight Kubernetes distribution by Rancher. It uses k3d to create containerized k3s clusters. With Docker, we can spin up a multi-node k3s cluster on a single machine. K3d helps to run k3s inside a Docker container.
For this guide, I am going to be using Ubuntu based Linux system for any examples. Let’s go ahead to install Docker CE on our server. Update your system before you proceed.
sudo apt update && sudo apt upgrade -yIf a reboot is required let the system be restarted.
[ -e /var/run/reboot-required ] && sudo rebootStep 1: Install Docker Runtime
The simplest way of installing Docker in your system is the scripted method.
curl -fsSL get.docker.com -o get-docker.sh
sudo sh get-docker.shTo check your version of Docker run the following commands:
$ docker --version
Docker version 27.1.1, build 6312585Service is started automatically after installation:
$ systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-08-09 16:07:54 UTC; 5min ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 2005 (dockerd)
Tasks: 8
Memory: 24.6M
CPU: 451ms
CGroup: /system.slice/docker.service
└─2005 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Aug 09 16:07:53 ubuntu-cloudspinx-com dockerd[203788]: time="2024-08-09T16:07:53.474770680Z" level=info msg="Starting up"
Aug 09 16:07:53 ubuntu-cloudspinx-com dockerd[203788]: time="2024-08-09T16:07:53.475379494Z" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
Aug 09 16:07:53 ubuntu-cloudspinx-com dockerd[203788]: time="2024-08-09T16:07:53.744487436Z" level=info msg="Loading containers: start."
Aug 09 16:07:53 ubuntu-cloudspinx-com dockerd[203788]: time="2024-08-09T16:07:53.798373584Z" level=info msg="Firewalld: created docker-forwarding policy"
Aug 09 16:07:54 ubuntu-cloudspinx-com dockerd[203788]: time="2024-08-09T16:07:54.272908647Z" level=info msg="Firewalld: interface docker0 already part of docker zone, returning"
Aug 09 16:07:54 ubuntu-cloudspinx-com dockerd[203788]: time="2024-08-09T16:07:54.363245984Z" level=info msg="Loading containers: done."
Aug 09 16:07:54 ubuntu-cloudspinx-com dockerd[203788]: time="2024-08-09T16:07:54.375650104Z" level=info msg="Docker daemon" commit=cc13f95 containerd-snapshotter=false storage-driver=overlay2 version=27.1.1
Aug 09 16:07:54 ubuntu-cloudspinx-com dockerd[203788]: time="2024-08-09T16:07:54.375753181Z" level=info msg="Daemon has completed initialization"
Aug 09 16:07:54 ubuntu-cloudspinx-com dockerd[203788]: time="2024-08-09T16:07:54.405335136Z" level=info msg="API listen on /run/docker.sock"
Aug 09 16:07:54 ubuntu-cloudspinx-com systemd[1]: Started docker.service - Docker Application Container Engine.Confirm installation by checking version:
$ docker --version
Docker version 24.0.5, build ced0996Step 2: Install k3d on your Linux system
There are a number of options you can use to install k3s inside docker (k3d). See below installation steps for your particular Linux operating system.
sudo -iUsing installation script to get the latest version:
# With wget
wget -q -O - https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bash
# Or with curl
curl -s https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bashInstall on macOS with Homebrew:
brew install k3dFor Arch Linux you can use yay:
yay -S rancher-k3d-binConfirm installed version
# k3d version
k3d version v5.7.3
k3s version v1.30.3-k3s1 (default)Once you have installed, you can run k3d help to see what you can do with k3d
# k3d help
https://k3d.io/
k3d is a wrapper CLI that helps you to easily create k3s clusters inside docker.
Nodes of a k3d cluster are docker containers running a k3s image.
All Nodes of a k3d cluster are part of the same docker network.
Usage:
k3d [flags]
k3d [command]
Available Commands:
cluster Manage cluster(s)
completion Generate completion scripts for [bash, zsh, fish, powershell | psh]
config Work with config file(s)
help Help about any command
image Handle container images.
kubeconfig Manage kubeconfig(s)
node Manage node(s)
registry Manage registry/registries
version Show k3d and default k3s version
Flags:
-h, --help help for k3d
--timestamps Enable Log timestamps
--trace Enable super verbose output (trace logging)
--verbose Enable verbose output (debug logging)
--version Show k3d and default k3s version
Use "k3d [command] --help" for more information about a command.Step 3: Create a Kubernetes cluster with k3d
- Creating a cluster with single agent (worker node)
To create a cluster named ‘k8skdcluster‘ with a single node, run the command below:
# k3d cluster create k8skdcluster
INFO[0000] Prep: Network
INFO[0000] Created network 'k3d-k8skdcluster'
INFO[0000] Created image volume k3d-k8skdcluster-images
INFO[0000] Starting new tools node...
INFO[0000] Pulling image 'ghcr.io/k3d-io/k3d-tools:5.7.3'
INFO[0001] Creating node 'k3d-k8skdcluster-server-0'
INFO[0001] Starting node 'k3d-k8skdcluster-tools'
INFO[0002] Pulling image 'docker.io/rancher/k3s:v1.30.3-k3s1'
INFO[0006] Creating LoadBalancer 'k3d-k8skdcluster-serverlb'
INFO[0006] Pulling image 'ghcr.io/k3d-io/k3d-proxy:5.7.3'
INFO[0009] Using the k3d-tools node to gather environment information
INFO[0009] HostIP: using network gateway 172.18.0.1 address
INFO[0009] Starting cluster 'k8skdcluster'
INFO[0009] Starting servers...
INFO[0009] Starting node 'k3d-k8skdcluster-server-0'
INFO[0013] All agents already running.
INFO[0013] Starting helpers...
INFO[0013] Starting node 'k3d-k8skdcluster-serverlb'
INFO[0019] Injecting records for hostAliases (incl. host.k3d.internal) and for 2 network members into CoreDNS configmap...
INFO[0021] Cluster 'k8skdcluster' created successfully!
INFO[0021] You can now use it like this:
kubectl cluster-info- Creating a cluster with more than one agent
You can remove current cluster or use different name:
k3d cluster delete k8skdclusterTo create a k3d cluster with more than one worker nodes, run the command below:
$ k3d cluster create k8skdcluster --agents 3
INFO[0000] Prep: Network
INFO[0000] Created network 'k3d-k8skdcluster'
INFO[0000] Created image volume k3d-k8skdcluster-images
INFO[0000] Starting new tools node...
INFO[0000] Starting node 'k3d-k8skdcluster-tools'
INFO[0001] Creating node 'k3d-k8skdcluster-server-0'
INFO[0001] Creating node 'k3d-k8skdcluster-agent-0'
INFO[0001] Creating node 'k3d-k8skdcluster-agent-1'
INFO[0001] Creating node 'k3d-k8skdcluster-agent-2'
INFO[0001] Creating LoadBalancer 'k3d-k8skdcluster-serverlb'
INFO[0001] Using the k3d-tools node to gather environment information
INFO[0001] HostIP: using network gateway 172.18.0.1 address
INFO[0001] Starting cluster 'k8skdcluster'
INFO[0001] Starting servers...
INFO[0001] Starting node 'k3d-k8skdcluster-server-0'
INFO[0005] Starting agents...
INFO[0005] Starting node 'k3d-k8skdcluster-agent-2'
INFO[0005] Starting node 'k3d-k8skdcluster-agent-0'
INFO[0005] Starting node 'k3d-k8skdcluster-agent-1'
INFO[0011] Starting helpers...
INFO[0011] Starting node 'k3d-k8skdcluster-serverlb'
INFO[0018] Injecting records for hostAliases (incl. host.k3d.internal) and for 5 network members into CoreDNS configmap...
INFO[0020] Cluster 'k8skdcluster' created successfully!
INFO[0020] You can now use it like this:
kubectl cluster-infoWhere –agents flag specifies the number of agents you need. In this case, I have specified 3. Remember to export config. Once its created you can list clusters:
# k3d cluster list
NAME SERVERS AGENTS LOADBALANCER
k8skdcluster 1/1 3/3 trueStep 4: Managing k3s Cluster
List clusters with k3d
# k3d cluster list
NAME SERVERS AGENTS LOADBALANCER
k8skdcluster 1/1 0/0 trueList nodes with k3d
# k3d node list
NAME ROLE CLUSTER STATUS
k3d-k8skdcluster-server-0 server k8skdcluster running
k3d-k8skdcluster-serverlb loadbalancer k8skdcluster runningStep 5: Manage k3s Cluster with kubectl
To be able to use Kubectl in K3s cluster, we need to install Kubectl first.
Install Kubectl on Linux
Run the command below to install the latest version.
curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl"Make the kubectl binary executable.
chmod +x ./kubectlMove the binary in to your PATH.
mv ./kubectl /usr/local/bin/kubectlTo install Kubectl on MacOS, use the below commands
brew install kubectl
# or
brew install kubernetes-cliYou can now run k3s cluster commands with kubectl. For example, get all pods
# kubectl get pods --all-namespaces
kube-system coredns-576bfc4dc7-p5vxt 1/1 Running 0 115s
kube-system helm-install-traefik-crd-kt8ql 0/1 Completed 0 115s
kube-system helm-install-traefik-llswz 0/1 Completed 1 115s
kube-system local-path-provisioner-6795b5f9d8-rwqtt 1/1 Running 0 115s
kube-system metrics-server-557ff575fb-ms6ht 1/1 Running 0 115s
kube-system svclb-traefik-b7daaafb-nw5fv 2/2 Running 0 103s
kube-system traefik-5fb479b77-grjc8 1/1 Running 0 103sList nodes in the cluster
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k3d-k8skdcluster-server-0 Ready control-plane,master 2m33s v1.30.3+k3s1List nodes
# k3d node list
NAME ROLE CLUSTER STATUS
k3d-newcluster-agent-0 agent newcluster running
k3d-newcluster-agent-1 agent newcluster running
k3d-newcluster-server-0 server newcluster running
k3d-newcluster-serverlb loadbalancer newcluster runningConfirm running containers
# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1111327ef68b ghcr.io/k3d-io/k3d-proxy:5.6.0 "/bin/sh -c nginx-pr…" 3 minutes ago Up 3 minutes 80/tcp, 0.0.0.0:32913->6443/tcp k3d-newcluster-serverlb
976dbc89fbd0 rancher/k3s:v1.27.4-k3s1 "/bin/k3d-entrypoint…" 3 minutes ago Up 3 minutes k3d-newcluster-server-0Get cluster info
# kubectl cluster-info
Kubernetes master is running at https://0.0.0.0:43467
CoreDNS is running at https://0.0.0.0:43467/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://0.0.0.0:43467/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.Get all services in the cluster
# kubectl -n kube-system get all
NAME READY STATUS RESTARTS AGE
pod/metrics-server-7b4f8b595-qdfsc 1/1 Running 0 33m
pod/helm-install-traefik-ztlj8 0/1 Completed 0 33m
pod/svclb-traefik-vmwr5 2/2 Running 0 31m
pod/svclb-traefik-787b2 2/2 Running 0 31m
pod/svclb-traefik-4l4q2 2/2 Running 0 31m
pod/coredns-66c464876b-v9m94 1/1 Running 0 33m
pod/traefik-5dd496474-sp9hl 1/1 Running 0 31m
pod/local-path-provisioner-7ff9579c6-mnccm 1/1 Running 4 33m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 33m
service/metrics-server ClusterIP 10.43.237.220 <none> 443/TCP 33m
service/traefik-prometheus ClusterIP 10.43.116.77 <none> 9100/TCP 31m
service/traefik LoadBalancer 10.43.178.246 172.21.0.4 80:30172/TCP,443:30736/TCP 31m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/svclb-traefik 3 3 3 3 3 <none> 31m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/metrics-server 1/1 1 1 33m
deployment.apps/coredns 1/1 1 1 33m
deployment.apps/traefik 1/1 1 1 31m
deployment.apps/local-path-provisioner 1/1 1 1 33m
NAME DESIRED CURRENT READY AGE
replicaset.apps/metrics-server-7b4f8b595 1 1 1 33m
replicaset.apps/coredns-66c464876b 1 1 1 33m
replicaset.apps/traefik-5dd496474 1 1 1 31m
replicaset.apps/local-path-provisioner-7ff9579c6 1 1 1 33m
NAME COMPLETIONS DURATION AGE
job.batch/helm-install-traefik 1/1 94s 33m
To delete a cluster with k3d, use command syntax:
# k3d cluster delete <cluster-name>
INFO[0001] Deleting cluster 'newcluster'
INFO[0005] Deleted k3d-newcluster-serverlb
INFO[0008] Deleted k3d-newcluster-server-0
INFO[0008] Deleting cluster network '40d1445aaac90b4aa8c2db5c2547d3bbfa7662941808df0721373050ffea9c18'
INFO[0009] Deleting image volume 'k3d-newcluster-images'
INFO[0009] Removing cluster details from default kubeconfig...
INFO[0009] Removing standalone kubeconfig file (if there is one)...
INFO[0009] Successfully deleted cluster newcluster! Enjoy working with K3s cluster in Docker containers and contact CloudSpinx for any container related issues.
