📝

Test - Backup and Restore Methods 2

Date

2025/01/07

Q3

How many clusters are defined in the kubeconfig on the student-node?

You can make use of the kubectl config command.

풀이 과정

kubectl config view 명령을 통해 현재 node에서 관리되는 클러스터를 확인할 수 있다

student-node ~ ➜  kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://cluster1-controlplane:6443
  name: cluster1
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://192.1.3.9:6443
  name: cluster2
contexts:
- context:
    cluster: cluster1
    user: cluster1
  name: cluster1
- context:
    cluster: cluster2
    user: cluster2
  name: cluster2
current-context: cluster1
kind: Config
preferences: {}
users:
- name: cluster1
  user:
    client-certificate-data: DATA+OMITTED
    client-key-data: DATA+OMITTED
- name: cluster2
  user:
    client-certificate-data: DATA+OMITTED
    client-key-data: DATA+OMITTED
Bash
복사

정답

Q4

How many nodes (both controlplane and worker) are part of cluster1?

풀이 과정

student-node ~ ➜  kubectl config use-context cluster1
Switched to context "cluster1".

student-node ~ ➜  kubectl get nodes
NAME                    STATUS   ROLES           AGE   VERSION
cluster1-controlplane   Ready    control-plane   78m   v1.29.0
cluster1-node01         Ready    <none>          78m   v1.29.0
Bash
복사

정답

Q5

What is the name of the controlplane node in cluster2?

풀이 과정

student-node ~ ➜  kubectl config use-context cluster2
Switched to context "cluster2".

student-node ~ ➜  kubectl get nodes
NAME                    STATUS   ROLES           AGE   VERSION
cluster2-controlplane   Ready    control-plane   80m   v1.29.0
cluster2-node01         Ready    <none>          79m   v1.29.0
Bash
복사

정답

cluster2-controlplane

Q7

How is ETCD configured for cluster1?

Remember, you can access the clusters from student-node using the kubectl tool. You can also ssh to the cluster nodes from the student-node.

풀이 과정

클러스터 내부에서 pod로 실행되는 ETCD는 Stacked ETCD이다. 해당 클러스터에서는 static pod로 etcd가 실행되고 있으므로 Stacked 이다.

방법 1) Pod 확인

student-node ~ ➜  kubectl config use-context cluster1
Switched to context "cluster1".

student-node ~ ➜  kubectl get po -n kube-system | grep etcd
etcd-cluster1-controlplane                      1/1     Running   0             82m
Bash
복사

방법 2) manifest 파일 확인

student-node ~ ➜  ssh cluster1-controlplane

cluster1-controlplane ~ ➜  ls /etc/kubernetes/manifests/ | grep etcd
etcd.yaml
Bash
복사

정답

Stacked ETCD

Q8

How is ETCD configured for cluster2?

Remember, you can access the clusters from student-node using the kubectl tool. You can also ssh to the cluster nodes from the student-node.

풀이 과정

방법 1) Pod 확인

student-node ~ ➜  kubectl config use-context cluster2
Switched to context "cluster2".

student-node ~ ➜  kubectl get po -n kube-system | grep etcd
Bash
복사

방법 2) manifest 파일 확인

student-node ~ ➜  ssh cluster2-controlplane

cluster2-controlplane ~ ➜  ls /etc/kubernetes/manifests/ | grep etcd
Bash
복사

etcd가 클러스터 내부에 위치하고 있지 않으므로 Stacked ETCD는 아니다. 그렇다면 ‘ETCD가 없는가’에 대해 확인해야 한다. ETCD에 대한 프로세스가 있는지에 대한 확인이 필요하다. kube-apiserver에서 external etcd를 확인할 수 있었다.

cluster2-controlplane ~ ➜  ps -ef | grep etcd
root        2881    2474  0 Jan07 ?        00:03:57 kube-apiserver --advertise-address=192.1.3.9 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.pem --etcd-certfile=/etc/kubernetes/pki/etcd/etcd.pem --etcd-keyfile=/etc/kubernetes/pki/etcd/etcd-key.pem --etcd-servers=https://192.1.3.17:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
root       11661   11576  0 01:18 pts/0    00:00:00 grep etcd
Bash
복사

정답

External ETCD

Q9

What is the IP address of the External ETCD datastore used in cluster2?

풀이 과정

위에서 확인한 etcd에 대한 kube-apiserver 프로세스에서 etcd-server에 대한 정보를 확인할 수 있다. 프로세스로 확인하는 방법 이외의 방법으로는 kube-apiserver pod의 세부사항을 확인하면 된다. describe 명령어로 확인하거나 manifest를 확인하면 된다.

방법 1) Pod detail로 확인

student-node ~ ➜  kubectl config use-context cluster2
Switched to context "cluster2".

student-node ~ ➜  kubectl describe -n kube-system po kube-apiserver-cluster2-controlplane | grep etcd-server
      --etcd-servers=https://192.1.3.17:2379
Bash
복사

방법2) manifest 확인

student-node ~ ➜  ssh cluster2-controlplane

cluster2-controlplane ~ ➜  cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd-server
    - --etcd-servers=https://192.1.3.17:2379
Bash
복사

정답

192.1.3.17 (그때그때 다름)

Q10

What is the default data directory used the for ETCD datastore used in cluster1?

Remember, this cluster uses a Stacked ETCD topology.

풀이 과정

방법 1) Pod detail로 확인

student-node ~ ➜  kubectl config use-context cluster2
Switched to context "cluster2".

student-node ~ ➜  kubectl describe -n kube-system po kube-apiserver-cluster2-controlplane | grep etcd-server
      --etcd-servers=https://192.1.3.17:2379
Bash
복사

방법2) manifest 확인

student-node ~ ➜  ssh cluster1-controlplane

cluster1-controlplane ~ ➜  cat /etc/kubernetes/manifests/etcd.yaml | grep data-dir
    - --data-dir=/var/lib/etcd
Bash
복사

정답

/var/lib/etcd

Q12

What is the default data directory used the for ETCD datastore used in cluster2?

Remember, this cluster uses a External ETCD topology.

풀이 과정

외부 ETCD는 pod나 manifest가 클러스터 내부에 없기 때문에 위에서 진행한 방법들로는 확인할 수 없다. Q11에서 제공된대로 etcd-server로 ssh 접속을 통해 확인해야 한다.

student-node ~ ➜  ssh etcd-server

etcd-server ~ ➜  ps -ef | grep etcd | grep --color=auto data-dir
etcd         824       1  0 Jan07 ?        00:01:50 /usr/local/bin/etcd --name etcd-server --data-dir=/var/lib/etcd-data --cert-file=/etc/etcd/pki/etcd.pem --key-file=/etc/etcd/pki/etcd-key.pem --peer-cert-file=/etc/etcd/pki/etcd.pem --peer-key-file=/etc/etcd/pki/etcd-key.pem --trusted-ca-file=/etc/etcd/pki/ca.pem --peer-trusted-ca-file=/etc/etcd/pki/ca.pem --peer-client-cert-auth --client-cert-auth --initial-advertise-peer-urls https://192.1.3.17:2380 --listen-peer-urls https://192.1.3.17:2380 --advertise-client-urls https://192.1.3.17:2379 --listen-client-urls https://192.1.3.17:2379,https://127.0.0.1:2379 --initial-cluster-token etcd-cluster-1 --initial-cluster etcd-server=https://192.1.3.17:2380 --initial-cluster-state new
Bash
복사

정답

/var/lib/etcd-data

Q13

How many nodes are part of the ETCD cluster that etcd-server is a part of?

풀이 과정

etcdctl member list 명령으로 cluster node들을 확인할 수 있다. 클러스터 정보를 위해 인증이 필요하기 때문에 snapshot을 할 때와 같이 각각의 플래그에 대한 정보를 기입한다. 이는 위에서 확인했던 etcd 프로세스 결과값으로 확인 가능하다.

etcd-server ~ ➜  ETCDCTL_API=3 etcdctl member list \
--endpoints=127.0.0.1:2379 \
--cert=/etc/etcd/pki/etcd.pem \
--cacert=/etc/etcd/pki/ca.pem \
--key=/etc/etcd/pki/etcd-key.pem

77a603763678b2f4, started, etcd-server, https://192.1.3.17:2380, https://192.1.3.17:2379, false
Bash
복사

정답

Q14

Take a backup of etcd on cluster1 and save it on the student-node at the path /opt/cluster1.db

풀이

snapshot 생성

student-node ~ ➜  ssh cluster1-controlplane

cluster1-controlplane ~ ➜  ETCDCTL_API=3 etcdctl snapshot save /opt/cluster1.db \
--endpoints=127.0.0.1:2379 \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--key=/etc/kubernetes/pki/etcd/server.key

Snapshot saved at /opt/cluster1.db
Bash
복사

student-node로 파일 복사

student-node ~ ➜  scp cluster1-controlplane:/opt/cluster1.db /opt/cluster1.db
cluster1.db                   100% 2200KB 134.4MB/s   00:00    

student-node ~ ➜  ls /opt
cluster1.db
Bash
복사

Q15

An ETCD backup for cluster2 is stored at /opt/cluster2.db. Use this snapshot file to carryout a restore on cluster2 to a new path /var/lib/etcd-data-new.

Once the restore is complete, ensure that the controlplane components on cluster2 are running.

The snapshot was taken when there were objects created in the critical namespace on cluster2. These objects should be available post restore.

풀이

etcd-server로 cluster2.db 스냅샷 복사

student-node ~ ➜  scp /opt/cluster2.db etcd-server:/root
cluster2.db                100% 2320KB 209.3MB/s   00:00
Bash
복사

스냅샷을 통해 복원

etcd-server ~ ➜  ETCDCTL_API=3 etcdctl snapshot restore /root/cluster2.db \
--data-dir=/var/lib/etcd-data-new

{"level":"info","ts":1736301547.4107833,"caller":"snapshot/v3_snapshot.go:296","msg":"restoring snapshot","path":"/root/cluster2.db","wal-dir":"/var/lib/etcd-data-new/member/wal","data-dir":"/var/lib/etcd-data-new","snap-dir":"/var/lib/etcd-data-new/member/snap"}
{"level":"info","ts":1736301547.429816,"caller":"mvcc/kvstore.go:388","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":10635}
{"level":"info","ts":1736301547.4367092,"caller":"membership/cluster.go:392","msg":"added member","cluster-id":"cdf818194e3a8c32","local-member-id":"0","added-peer-id":"8e9e05c52164694d","added-peer-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":1736301547.4433842,"caller":"snapshot/v3_snapshot.go:309","msg":"restored snapshot","path":"/root/cluster2.db","wal-dir":"/var/lib/etcd-data-new/member/wal","data-dir":"/var/lib/etcd-data-new","snap-dir":"/var/lib/etcd-data-new/member/snap"}
Bash
복사

data-dir 경로 업데이트

해당 ETCD는 External ETCD로, pod 형태가 아닌 서비스로 실행되기 때문에 systemd에서 확인해야 한다. 따라서 /etc/systemd/system/etcd.service 파일을 수정하면 되는데 해당 경로를 확인하는 것은 systemctl status etcd 를 통해 etcd의 서비스 파일 경로를 확인할 수 있다.

vi /etc/systemd/system/etcd.service

[Unit]
Description=etcd key-value store
Documentation=https://github.com/etcd-io/etcd
After=network.target

[Service]
User=etcd
Type=notify
ExecStart=/usr/local/bin/etcd \
  --name etcd-server \
  --data-dir=/var/lib/etcd-data-new \   # 해당 경로 업데이트
...
Bash
복사

권한 업데이트 

etcd-server ~ ➜  chown -R etcd:etcd /var/lib/etcd-data-new
Bash
복사

etcd service 재시작

systemctl daemon-reload 
systemctl restart etcd
Bash
복사

확인

apiserver, scheduler, controller-manager가 Restore 과정을 통해 재시작되었음을 확인할 수 있다.

student-node ~ ➜  kubectl get po -n kube-system
NAME                                            READY   STATUS    RESTARTS       AGE
coredns-69f9c977-4kjqc                          1/1     Running   0              46m
coredns-69f9c977-wvn8z                          1/1     Running   0              46m
kube-apiserver-cluster2-controlplane            1/1     Running   7 (4m8s ago)   46m
kube-controller-manager-cluster2-controlplane   1/1     Running   1 (10m ago)    46m
kube-proxy-ghcdq                                1/1     Running   0              45m
kube-proxy-tpjh6                                1/1     Running   0              46m
kube-scheduler-cluster2-controlplane            1/1     Running   1 (10m ago)    46m
weave-net-75x5z                                 2/2     Running   1 (46m ago)    46m
weave-net-t6bn2                                 2/2     Running   0              45m
Bash
복사

TroubleShooting

권한을 업데이트하지 않고 진행했을 경우 아래와 같이 etcd 서비스가 재시작 되지 않으며, 동작하고 있지 않음을 볼 수 있다.

etcd-server ~ ➜  systemctl restart etcd
Job for etcd.service failed because the control process exited with error code.
See "systemctl status etcd.service" and "journalctl -xe" for details.

etcd-server ~ ✖ systemctl status etcd
● etcd.service - etcd key-value store
   Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Wed 2025-01-08 02:13:13 UTC; 83
9ms ago
     Docs: https://github.com/etcd-io/etcd
  Process: 2083 ExecStart=/usr/local/bin/etcd --name etcd-server --data-dir=/var/lib/etcd-data
-new --cert-file=/etc/etcd/pki/etcd.pem --key-file=/etc/etcd/pki/etcd-key.pem --peer-cert-file
=/etc/etcd/pki/etcd.pem --peer-key-file=/etc/etcd/pki/etcd-key.pem --trusted-ca-file=/etc/etcd
/pki/ca.pem --peer-trusted-ca-file=/etc/etcd/pki/ca.pem --peer-client-cert-auth --client-cert-
auth --initial-advertise-peer-urls https://192.2.20.6:2380 --listen-peer-urls https://192.2.20
.6:2380 --advertise-client-urls https://192.2.20.6:2379 --listen-client-urls https://192.2.20.
6:2379,https://127.0.0.1:2379 --initial-cluster-token etcd-cluster-1 --initial-cluster etcd-se
rver=https://192.2.20.6:2380 --initial-cluster-state new (code=exited, status=1/FAILUR
E)
 Main PID: 2083 (code=exited, status=1/FAILURE)
Bash
복사

문제 해결을 위해journalctl을 통해 에러 로그를 확인해보았을 때, ETCD에 지정된 데이터 디렉토리를 찾지 못했거나 해당 디렉토리에 접근할 수 없어 etcd 서비스가 실패됨을 확인할 수 있었다.

Jan 08 02:15:09 etcd-server etcd[2326]: error listing data dir: /var/l
ib/etcd-data-new
Jan 08 02:15:09 etcd-server systemd[1]: Failed to start etcd key-value
 store.
Bash
복사

따라서 etcd-data-new의 권한을 확인하였고, 소유 권한을 root → etcd 변경을 통해 실행을 완료했다.

etcd-server ~ ➜  ls -l /var/lib | grep etcd-data
drwx------ 1 etcd etcd 4096 Jan  8 01:36 etcd-data
drwx------ 3 root root 4096 Jan  8 02:06 etcd-data-new

etcd-server ~ ➜  chown -R etcd:etcd /var/lib/etcd-data-new
etcd-server ~ ➜  systemctl daemon-reload
etcd-server ~ ➜  systemctl restart etcd
Bash
복사