DayOne
Journal
Today I Learned

Day 3 - How to setup k8s on Vagrant

Introduction

Just straight to the tutorial without any introduction. My introduction only:

If you want to learn how to setup multi-nodes k8s cluster, you come to the right place

1 - Preparation - Installing Vagrant

You need to install vagrant. Depending on your OS, you can visit the official documentation here to get the installation guideline. I’m using Arch btw, so I’m going to show you how to install vagrant using Arch.

  1. Install vagrant from AUR using yay. Run this command: yay -S vagrant
  2. Install required plugins. Run this command: vagrant plugin install vagrant-vbguest vagrant-share

2 - Preparation - Vagrantfile

Create project directory and create Vagrantfile

mkdir -p <your-project-path>
cd <your-project-path>
touch Vagrantfile
code . # Depending on your IDE

You can visit official documentation to get initial Vagrantfile or you may run init command:

vagrant init

It will give you the Vagrantfile initial template. For me, I have my own template that you can copy it below

# -*- mode: ruby -*-
# vi: set ft=ruby :

# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|
  # The most common configuration options are documented and commented below.
  # For a complete reference, please see the online documentation at
  # https://docs.vagrantup.com.

  # Every Vagrant development environment requires a box. You can search for
  # boxes at https://vagrantcloud.com/search.
  config.vm.box = "ubuntu/focal64"

  config.vm.provision "shell", inline: <<-SHELL
    # Install containerd
    for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done

    # Add Docker's official GPG key:
    sudo apt-get update
    sudo apt-get install ca-certificates curl
    sudo install -m 0755 -d /etc/apt/keyrings
    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
    sudo chmod a+r /etc/apt/keyrings/docker.asc

    # Add the repository to Apt sources:
    echo \
      "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
      $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
      sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    sudo apt-get update

    sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

    # Install kubeadm
    sudo apt-get update
    sudo apt-get install -y apt-transport-https ca-certificates curl gpg

    curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

    echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

    sudo apt-get update
    sudo apt-get install -y kubelet kubeadm kubectl
    sudo apt-mark hold kubelet kubeadm kubectl

    sudo systemctl enable --now kubelet
  SHELL

  config.vm.provider "virtualbox" do |vb|
    vb.memory = 2048
    vb.cpus = 2
  end

  # Define Master
  config.vm.define "master" do |master|

    master.vm.hostname = "master"
    master.vm.network "private_network", ip: "192.168.56.11"
    master.vm.network "forwarded_port", guest: 8080, host: 8080
    master.vm.network "forwarded_port", guest: 9090, host: 9090
    master.vm.network "forwarded_port", guest: 3000, host: 3000
  end

  # Define Worker1
  config.vm.define "worker1" do |worker1|

    worker1.vm.hostname = "worker1"
    worker1.vm.network "private_network", ip: "192.168.56.12"

  end

  # Define Worker2
  config.vm.define "worker2" do |worker2|

    worker2.vm.hostname = "worker2"
    worker2.vm.network "private_network", ip: "192.168.56.13"

  end
end

Explanation

I create one master node and two workers (worker1 and worker2) as you can see at the Vagrantfile as:

config.vm.define "master" do |master|
config.vm.define "worker1" do |worker1|
config.vm.define "worker2" do |worker2|

In master node, I add network configuration for forwarding a port so that I can do kubectl port-forward svc/my-service hostPort:targetPort later when I want to test my app

I add initial script to install necessary packages during vm creation. You can see at the section config.vm.provision

After that, you can enter your vagrant VM using vagrant ssh <vm-name>. You need to run each of ssh command on different terminal.

vagrant ssh master
vagrant ssh worker1
vagrant ssh worker2

3 - Install kubeadm, kubectl, and kubelet

Notes: when you enter vagrant vm for the first time, you may face something strange such as your backspace doesn’t work to do deleting and CTRL+L is not working for clearing the screen. Then you may need to run this command below at the beginning:
# First time enter the VM
echo 'export TERM=xterm' >> ~/.bashrc
echo 'alias k=kubectl' >> ~/.bashrc
source ~/.bashrc

For each node, you may need to create a new bash file called pre-install.sh and copy this code below:

### PRE-INSTALL for each node ####
#!/bin/bash

for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo usermod -aG docker $USER

containerd config default | sed 's/SystemdCgroup = false/SystemdCgroup = true/' | sudo tee /etc/containerd/config.toml

sudo systemctl restart containerd

sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sudo sysctl --system

### PRE-INSTALL for each node ####

Master node

For master node, you need to run this command:

#### INSTALL ONLY ON MASTER NODE ####
sudo kubeadm init --pod-network-cidr=<pod-network-cidr> --upload-certs --apiserver-advertise-address=<your-master-ip-address> --node-name=master

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

For pod-network-cidr, if you are planning to use Flannel for the CNI, you may need to use default value:

10.244.0.0/16

For your-master-ip-address, you can get the ip address using this command

ip addr

# See the value of inet for enp0s8 interface, usually starts with 192.*.*.*

Finally, you need to edit kubelet service so that your pod can be reachable from master node. You can run this following command:

# Edit kubelet if internal IP for each node is the same
sudo vim /lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Find this line ExecStart=/usr/bin/kubelet $KUBELET_KUBEADM_ARGS
# Then add --node=<your-node-ip>
# To get your node ip, use the same way using `ip addr` command and find
# the enp0s8 interface

# Then restart the service
sudo systemctl daemon-reexec
sudo systemctl restart kubelet

Worker node

For worker node, you only need to do two things:

  1. Run the kubeadm join from the output of init run at the master node. If you forget to copy the kubeadm join command, you can get the command by using running this command on master node
# Run this on master node to get kubeadm join command
kubeadm token create --print-join-command

Then run the kubeadm join command on your worker node. Refer to this documentation

kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>

Then edit the kubelet using the same way as master node

# Edit kubelet if internal IP for each node is the same
sudo vim /lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Find this line ExecStart=/usr/bin/kubelet $KUBELET_KUBEADM_ARGS
# Then add --node=<your-node-ip>
# To get your node ip, use the same way using `ip addr` command and find
# the enp0s8 interface

# Then restart the service
sudo systemctl daemon-reexec
sudo systemctl restart kubelet

4 - Closing

Voila, you have setup k8s on Vagrant virtual machine. Please contact me to my Twitter (X). Thank you