📝

Test - Cluster Upgrade Process

Date

2025/01/07

Q1

This lab tests your skills on upgrading a kubernetes cluster. We have a production cluster with applications running on it. Let us explore the setup first. What is the current version of the cluster?

풀이

kubectl get nodes

현재 버전에 대한 정보가 출력됨

controlplane ~ ➜  kubectl get nodes
NAME           STATUS   ROLES           AGE   VERSION
controlplane   Ready    control-plane   24m   v1.30.0
node01         Ready    <none>          23m   v1.30.0
Bash
복사

kubeadm upgrade plan

kubectl get nodes로도 버전을 확인할 수 있지만 이는 kubelet의 버전이지 클러스터의 버전이 아니기 때문에 정확한 판단을 위해서 kubeadm upgrade plan으로 현재 버전 확인

controlplane ~ ➜  kubeadm upgrade plan
...
COMPONENT   NODE           CURRENT   TARGET
kubelet     controlplane   v1.30.0   v1.30.8
kubelet     node01         v1.30.0   v1.30.8
...
Bash
복사

정답

v1.30.0

Q2

How many nodes are part of this cluster? Including controlplane and worker nodes

풀이

controlplane ~ ➜  kubectl get nodes
NAME           STATUS   ROLES           AGE   VERSION
controlplane   Ready    control-plane   32m   v1.30.0
node01         Ready    <none>          31m   v1.30.0
Bash
복사

정답

Q3

How many nodes can host workloads in this cluster? Inspect the applications and taints set on the nodes.

풀이

controlplane ~ ➜  kubectl describe node | grep Taints
Taints:             <none>
Taints:             <none>
Bash
복사

정답

Q4

How many applications are hosted on the cluster? Count the number of deployments in the default namespace.

풀이

controlplane ~ ➜  kubectl get deploy
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
blue   5/5     5            5           16m
Bash
복사

정답

Q5

What nodes are the pods hosted on?

풀이

controlplane ~ ➜  kubectl get po -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
blue-fffb6db8d-6hpfz   1/1     Running   0          17m   10.244.0.4   controlplane   <none>           <none>
blue-fffb6db8d-8mw8g   1/1     Running   0          17m   10.244.1.2   node01         <none>           <none>
blue-fffb6db8d-9bjzv   1/1     Running   0          17m   10.244.1.3   node01         <none>           <none>
blue-fffb6db8d-9qmc7   1/1     Running   0          17m   10.244.0.5   controlplane   <none>           <none>
blue-fffb6db8d-q6pbh   1/1     Running   0          17m   10.244.1.4   node01         <none>           <none>
Bash
복사

정답

controlplane, node01

Q6

You are tasked to upgrade the cluster. Users accessing the applications must not be impacted, and you cannot provision new VMs. What strategy would you use to upgrade the cluster?

정답

Upgrade one node at a time while moving the workloads to the other

Q7

What is the latest version available for an upgrade with the current version of the kubeadm tool installed?

풀이

controlplane ~ ➜  kubeadm upgrade plan
...
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE           CURRENT   TARGET
kubelet     controlplane   v1.30.0   v1.30.8
kubelet     node01         v1.30.0   v1.30.8

Upgrade to the latest version in the v1.30 series:

COMPONENT                 NODE           CURRENT    TARGET
kube-apiserver            controlplane   v1.30.0    v1.30.8
kube-controller-manager   controlplane   v1.30.0    v1.30.8
kube-scheduler            controlplane   v1.30.0    v1.30.8
kube-proxy                               1.30.0     v1.30.8
CoreDNS                                  v1.10.1    v1.11.1
etcd                      controlplane   3.5.12-0   3.5.12-0
...
Bash
복사

정답

v1.30.8

Q8

We will be upgrading the controlplane node first. Drain the controlplane node of workloads and mark it UnSchedulable

풀이 과정

kubectl drain controlplane --ignore-daemonsets
Bash
복사

Q9

Upgrade the controlplane components to exact version v1.31.0

Upgrade the kubeadm tool (if not already), then the controlplane components, and finally the kubelet. Practice referring to the Kubernetes documentation page.

풀이 과정

k8s 패키지 설치를 위한 APT Repository 설정

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
Bash
복사

업그레이드 버전 결정 

sudo apt update
sudo apt-cache madison kubeadm

   kubeadm | 1.31.4-1.1 | https://pkgs.k8s.io/core:/stable:/v1.31/deb  Packages
   kubeadm | 1.31.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.31/deb  Packages
   kubeadm | 1.31.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.31/deb  Packages
   kubeadm | 1.31.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.31/deb  Packages
   kubeadm | 1.31.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.31/deb  Packages
Bash
복사

1.31.0 버전으로 업그레이드 하라는 문제이므로 1.31.0-1.1 사용

kubeadm 업그레이드 

sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.31.0-1.1' && \
sudo apt-mark hold kubeadm
Bash
복사

plan 확인 

업그레이드 가능 버전이 v1.31.4로 변경됨을 확인

sudo kubeadm upgrade plan

...
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE           CURRENT   TARGET
kubelet     controlplane   v1.30.0   v1.31.4
kubelet     node01         v1.30.0   v1.31.4

Upgrade to the latest stable version:

COMPONENT                 NODE           CURRENT    TARGET
kube-apiserver            controlplane   v1.30.0    v1.31.4
kube-controller-manager   controlplane   v1.30.0    v1.31.4
kube-scheduler            controlplane   v1.30.0    v1.31.4
kube-proxy                               1.30.0     v1.31.4
CoreDNS                                  v1.10.1    v1.11.1
etcd                      controlplane   3.5.12-0   3.5.15-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.31.4

Note: Before you can perform this upgrade, you have to update kubeadm to v1.31.4.
...
Bash
복사

클러스터 업그레이드

v1.31.4까지 업그레이드 가능하지만 문제에 맞게 v1.31.0으로 업그레이드

kubeadm upgrade apply v1.31.0

...
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.31.0". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
Bash
복사

kubelet & kubectl 업그레이드 

sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.31.0-1.1' kubectl='1.31.0-1.1' && \
sudo apt-mark hold kubelet kubectl
Bash
복사

kubelet 재실행

sudo systemctl daemon-reload
sudo systemctl restart kubelet
Bash
복사

재실행 후 완료까지 약간의 시간 소요

확인

controlplane ~ ➜  kubectl get nodes
NAME           STATUS                     ROLES           AGE   VERSION
controlplane   Ready,SchedulingDisabled   control-plane   62m   v1.31.0
node01         Ready                      <none>          61m   v1.30.0
Bash
복사

Documents

Upgrading kubeadm clusters

This page explains how to upgrade a Kubernetes cluster created with kubeadm from version 1.30.x to version 1.31.x, and from version 1.31.x to 1.31.y (where y > x). Skipping MINOR versions when upgrading is unsupported. For more details, please visit Version Skew Policy. To see information about upgrading clusters created using older versions of kubeadm, please refer to following pages instead: Upgrading a kubeadm cluster from 1.29 to 1.30 Upgrading a kubeadm cluster from 1.

https://v1-31.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

pkgs.k8s.io: Introducing Kubernetes Community-Owned Package Repositories

On behalf of Kubernetes SIG Release, I am very excited to introduce the Kubernetes community-owned software repositories for Debian and RPM packages: pkgs.k8s.io! The new package repositories are replacement for the Google-hosted package repositories (apt.kubernetes.io and yum.kubernetes.io) that we've been using since Kubernetes v1.5. This blog post contains information about these new package repositories, what does it mean to you as an end user, and how to migrate to the new repositories.

https://v1-31.docs.kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/

Q10

Mark the controlplane node as "Schedulable" again

풀이

kubectl uncordon controlplane
Bash
복사

Q11

Next is the worker node. Drain the worker node of the workloads and mark it UnSchedulable

풀이

kubectl drain node01 --ignore-daemonsets
Bash
복사

Q12

Upgrade the worker node to the exact version v1.31.0

풀이 과정

Drain worker node 

kubectl drain node01 --ignore-daemonsets
Bash
복사

node01에 접속 

Bash
복사

k8s 패키지 설치를 위한 APT Repository 설정

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
Bash
복사

노드 업그레이드

sudo kubeadm upgrade node
Bash
복사

kubelet & kubectl 업그레이드

sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.31.0-1.1' kubectl='1.31.0-1.1' && \
sudo apt-mark hold kubelet kubectl
Bash
복사

kubelet 재시작

sudo systemctl daemon-reload
sudo systemctl restart kubelet
Bash
복사

controlplane으로 돌아와 확인

controlplane ~ ➜  kubectl get nodes
NAME           STATUS                     ROLES           AGE     VERSION
controlplane   Ready                      control-plane   76m     v1.31.0
node01         Ready,SchedulingDisabled   <none>          76m     v1.31.0
Bash
복사

Documents

Upgrading Linux nodes

This page explains how to upgrade a Linux Worker Nodes created with kubeadm. Before you begin You need to have shell access to all the nodes, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. To check the version, enter kubectl version. Familiarize yourself with the process for upgrading the rest of your kubeadm cluster.

https://v1-31.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/upgrading-linux-nodes/

Q13

Remove the restriction and mark the worker node as schedulable again.

풀이

controlplane ~ ➜  kubectl get nodes
NAME           STATUS   ROLES           AGE   VERSION
controlplane   Ready    control-plane   76m   v1.31.0
node01         Ready    <none>          75m   v1.31.0
Bash
복사