Search
๐Ÿ“

Test - Backup and Restore Methods

Date
2025/01/07
Category
Devops
Tag
Kubernetes
CKA
Lab

Q1

We have a working Kubernetes cluster with a set of web applications running. Let us first explore the setup. How many deployments exist in the cluster in defaultย namespace?

ํ’€์ด ๊ณผ์ •

controlplane ~ โžœ kubectl get deploy NAME READY UP-TO-DATE AVAILABLE AGE blue 3/3 3 3 38s red 2/2 2 2 38s
Bash
๋ณต์‚ฌ

์ •๋‹ต

2

Q2

What is the version of ETCD running on the cluster?
Check the ETCD Pod or Process

ํ’€์ด ๊ณผ์ •

1.
Pod detail๋กœ ํ™•์ธ
controlplane ~ โžœ kubectl -n kube-system describe pod etcd-controlplane | grep Image Image: registry.k8s.io/etcd:3.5.15-0 Image ID: registry.k8s.io/etcd@sha256:a6dc63e6e8cfa0307d7851762fa6b629afb18f28d8aa3fab5a6e91b4af60026a
Bash
๋ณต์‚ฌ
2.
log๋กœ ํ™•์ธ
controlplane ~ โžœ kubectl -n kube-system logs etcd-controlplane | grep -i 'etcd-version' {"level":"info","ts":"2025-01-07T15:05:55.915197Z","caller":"embed/etcd.go:310","msg":"starting an etcd server","etcd-version":"3.5.15","git-sha":"9a5533382","go-version":"go1.21.12","go-os":"linux","go-arch":"amd64","max-cpu-set":16,"max-cpu-available":16,"member-initialized":false,"name":"controlplane","data-dir":"/var/lib/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":10000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://192.168.183.211:2380"],"listen-peer-urls":["https://192.168.183.211:2380"],"advertise-client-urls":["https://192.168.183.211:2379"],"listen-client-urls":["https://127.0.0.1:2379","https://192.168.183.211:2379"],"listen-metrics-urls":["http://127.0.0.1:2381"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"controlplane=https://192.168.183.211:2380","initial-cluster-state":"new","initial-cluster-token":"etcd-cluster","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
Bash
๋ณต์‚ฌ

์ •๋‹ต

3.5.15

Q3

At what address can you reach the ETCD cluster from the controlplane node?
Check the ETCD Service configuration in the ETCD POD

ํ’€์ด ๊ณผ์ •

1.
Pod detail๋กœ ํ™•์ธ
controlplane ~ โžœ kubectl -n kube-system describe pod etcd-controlplane | grep listen-client --listen-client-urls=https://127.0.0.1:2379,https://192.168.231.167:2379
Bash
๋ณต์‚ฌ
2.
manifest ํŒŒ์ผ๋กœ ํ™•์ธ
controlplane ~ โžœ cat /etc/kubernetes/manifests/etcd.yaml | grep listen-client - --listen-client-urls=https://127.0.0.1:2379,https://192.168.183.211:2379
Bash
๋ณต์‚ฌ

์ •๋‹ต

https://127.0.0.1:2379

Q4

Where is the ETCD server certificate file located?
Note this path down as you will need to use it later

ํ’€์ด ๊ณผ์ •

1.
Pod detail๋กœ ํ™•์ธ
controlplane ~ โžœ kubectl -n kube-system describe pod etcd-controlplane | grep cert --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt --peer-client-cert-auth=true /etc/kubernetes/pki/etcd from etcd-certs (rw) etcd-certs:
Bash
๋ณต์‚ฌ
2.
manifest ํŒŒ์ผ๋กœ ํ™•์ธ
controlplane ~ โœ– cat /etc/kubernetes/manifests/etcd.yaml | grep cert - --cert-file=/etc/kubernetes/pki/etcd/server.crt - --client-cert-auth=true - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt - --peer-client-cert-auth=true name: etcd-certs name: etcd-certs
Bash
๋ณต์‚ฌ

์ •๋‹ต

/etc/kubernetes/pki/etcd/server.crt

Q5

Where is the ETCD CA Certificate file located?
Note this path down as you will need to use it later.

ํ’€์ด ๊ณผ์ •

1.
Pod detail๋กœ ํ™•์ธ
controlplane ~ โžœ cat /etc/kubernetes/manifests/etcd.yaml | grep ca - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt priorityClassName: system-node-critical
Bash
๋ณต์‚ฌ
2.
manifest ํŒŒ์ผ๋กœ ํ™•์ธ
controlplane ~ โžœ cat /etc/kubernetes/manifests/etcd.yaml | grep ca - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt priorityClassName: system-node-critical
Bash
๋ณต์‚ฌ

์ •๋‹ต

/etc/kubernetes/pki/etcd/ca.crt

Q6

The master node in our cluster is planned for a regular maintenance reboot tonight. While we do not anticipate anything to go wrong, we are required to take the necessary backups. Take a snapshot of theย ETCDย database using the built-inย snapshotย functionality.
Store the backup file at locationย /opt/snapshot-pre-boot.db

ํ’€์ด

# kubectl describe -n kube-system pod etcd-controlplane ๋˜๋Š” # cat /etc/kubernetes/manifests/etcd.yaml ๋ฅผ ํ†ตํ•ด ์„ค์ •๊ฐ’ ํ™•์ธํ•˜๋ฉฐ ๊ฐ’ ๋Œ€์ž… ETCDCTL_API=3 etcdctl snapshot save /opt/snapshot-pre-boot.db \ --endpoints=127.0.0.1:2379 \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --key=/etc/kubernetes/pki/etcd/server.key
Bash
๋ณต์‚ฌ

Q8

Wake up! We have a conference call! After the reboot the master nodes came back online, but none of our applications are accessible. Check the status of the applications on the cluster. What's wrong?

ํ’€์ด ๊ณผ์ •

controlplane ~ โžœ kubectl get all NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 172.20.0.1 <none> 443/TCP 36s
Bash
๋ณต์‚ฌ

์ •๋‹ต

All of the above (Pod, Deployment, Service ์•„๋ฌด๊ฒƒ๋„ ์กด์žฌํ•˜์ง€ ์•Š์Œ)

Q9

Luckily we took a backup. Restore the original state of the cluster using the backup file.

ํ’€์ด

1.
์Šค๋ƒ…์ƒท์„ ํ†ตํ•ด ๋ณต๊ตฌ ์ง„ํ–‰
ETCDCTL_API=3 etcdctl snapshot restore /opt/snapshot-pre-boot.db --data-dir=/var/lib/etcd-from-backup
Bash
๋ณต์‚ฌ
2.
๋ณต์› ์‹œ data-dir์„ ์ƒˆ๋กœ ์ง€์ •ํ•˜๊ธฐ ๋•Œ๋ฌธ์— ๊ทธ์— ๋งž์ถฐ etcd.yaml ํŒŒ์ผ ๋ณผ๋ฅจ ์„ค์ •
static pod์ด๊ธฐ ๋•Œ๋ฌธ์— yaml ํŒŒ์ผ๋งŒ ๋ณ€๊ฒฝํ•˜๋ฉด ์ ์šฉ๋จ
vi /etc/kubernetes/manifests/etcd.yaml volumes: - hostPath: path: /var/lib/etcd-from-backup # ์ƒˆ๋กœ์šด data-dir์— ๋งž๊ฒŒ ๊ฒฝ๋กœ ๋ณ€๊ฒฝ type: DirectoryOrCreate name: etcd-data
Bash
๋ณต์‚ฌ
3.
ํ™•์ธ
ํ™•์ธ๊นŒ์ง€ ์•ฝ๊ฐ„์˜ ์‹œ๊ฐ„ ์†Œ์š”๋˜๋ฉฐ, etcd-controlplane pod๊ฐ€ ์ƒˆ๋กœ ์‹œ์ž‘๋œ ๊ฒƒ์„ ๋ณผ ์ˆ˜ ์žˆ๋‹ค. ๋˜ํ•œ kube-apiserver, kube-controller-manager, kube-scheduler๋„ ์žฌ์‹œ์ž‘๋œ ๊ฒƒ์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ๋‹ค.
controlplane ~ โžœ kubectl get po -n kube-system NAME READY STATUS RESTARTS AGE coredns-77d6fd4654-2bxrw 1/1 Running 0 13m coredns-77d6fd4654-d79rs 1/1 Running 0 13m etcd-controlplane 1/1 Running 0 23s kube-apiserver-controlplane 1/1 Running 1 (4m17s ago) 13m kube-controller-manager-controlplane 1/1 Running 1 (6m19s ago) 13m kube-proxy-dqb5r 1/1 Running 0 13m kube-scheduler-controlplane 1/1 Running 1 (6m14s ago) 13m
Bash
๋ณต์‚ฌ
์‚ฌ๋ผ์กŒ๋˜ ์›Œํฌ๋กœ๋“œ ๋ณต๊ตฌ ์ƒํƒœ ํ™•์ธ
controlplane ~ โžœ kubectl get po -n kube-system NAME READY STATUS RESTARTS AGE coredns-77d6fd4654-2bxrw 1/1 Running 0 13m coredns-77d6fd4654-d79rs 1/1 Running 0 13m etcd-controlplane 1/1 Running 0 23s kube-apiserver-controlplane 1/1 Running 1 (4m17s ago) 13m kube-controller-manager-controlplane 1/1 Running 1 (6m19s ago) 13m kube-proxy-dqb5r 1/1 Running 0 13m kube-scheduler-controlplane 1/1 Running 1 (6m14s ago) 13m
Bash
๋ณต์‚ฌ