Search
๐Ÿ“

Test - View Certificate Details

Date
2025/01/14
Category
Devops
Tag
Kubernetes
CKA
Lab

Q12

Kubectl suddenly stops responding to your commands. Check it out! Someone recently modified theย /etc/kubernetes/manifests/etcd.yamlย file

ํ’€์ด

1.
kubectl ์ž‘๋™ํ™•์ธ
โ€ข
kubectl์ด ์ž‘๋™ํ•˜์ง€ ์•Š๋Š”๋‹ค๋Š” ๊ฒƒ์€ kube-apiserver, etcd-server์— ๋ฌธ์ œ๊ฐ€ ์žˆ๋‹ค๋Š” ๋œป์ด๋‹ค.
$ controlplane ~ โœ– kubectl get po The connection to the server controlplane:6443 was refused - did you specify the right host or port?
Bash
๋ณต์‚ฌ
2.
kube-apiserver์™€ etcd์— ๋Œ€ํ•œ ์ปจํ…Œ์ด๋„ˆ ํ™•์ธ
โ€ข
kubeadm์œผ๋กœ ํด๋Ÿฌ์Šคํ„ฐ๋ฅผ ๊ตฌ์„ฑํ–ˆ์„ ๊ฒฝ์šฐ Pod๋กœ ์ปดํฌ๋„ŒํŠธ๋“ค์ด ๋ฐฐํฌ๋˜๊ธฐ ๋•Œ๋ฌธ์— kubectl์ด ๋™์ž‘ํ•˜์ง€ ์•Š์œผ๋ฉด Pod์˜ Log๋ฅผ ํ™•์ธํ•  ์ˆ˜ ์—†๋‹ค.
$ controlplane ~ โžœ crictl ps -a | grep kube-apiserver 8ae9a07114306 604f5db92eaa8 59 seconds ago Exited kube-apiserver 5 a2a0ad40b4e4f kube-apiserver-controlplane $ controlplane ~ โžœ crictl logs 8ae9a07114306 ... W0114 07:50:45.959183 1 logging.go:55] [core] [Channel #2 SubChannel #4]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused" W0114 07:50:47.204132 1 logging.go:55] [core] [Channel #1 SubChannel #3]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused" F0114 07:50:51.314580 1 instance.go:225] Error creating leases: error creating storage factory: context deadline exceeded
Bash
๋ณต์‚ฌ
โ†’ 127.0.0.1:2379์— ๋Œ€ํ•œ ์ ‘๊ทผ ์—๋Ÿฌ๊ฐ€ ๋ฐœ์ƒํ•˜๊ณ  ์žˆ๋‹ค. ์ฆ‰ ETCD๋ฅผ ํ™•์ธํ•ด์•ผ ํ•œ๋‹ค
$ controlplane ~ โžœ crictl ps -a | grep etcd 1cf6df5a713ae 2e96e5913fc06 About a minute ago Exited etcd 6 7db3425fb37c6 etcd-controlplane $ controlplane ~ โžœ crictl logs 1cf6df5a713ae ... {"level":"info","ts":"2025-01-14T07:53:05.002032Z","caller":"etcdmain/main.go:50","msg":"successfully notified init daemon"} {"level":"fatal","ts":"2025-01-14T07:53:05.002038Z","caller":"etcdmain/etcd.go:219","msg":"listener failed","error":"open /etc/kubernetes/pki/etcd/server-certificate.crt: no such file or directory","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:219\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:267"}
Bash
๋ณต์‚ฌ
โ†’ /etc/kubernetes/pki/etcd/server-certificate.crt ํŒŒ์ผ์ด ์—†๋‹ค๋Š” ์—๋Ÿฌ๋ฅผ ํ™•์ธํ•  ์ˆ˜ ์žˆ๋‹ค. ๋”ฐ๋ผ์„œ etcd์˜ ์ธ์ฆ์„œ๋ฅผ ํ™•์ธํ•ด์•ผ ํ•œ๋‹ค.
3.
etcd ์ธ์ฆ์„œ ํ™•์ธ
$ controlplane ~ โžœ ls /etc/kubernetes/pki/etcd/*.crt /etc/kubernetes/pki/etcd/ca.crt /etc/kubernetes/pki/etcd/peer.crt /etc/kubernetes/pki/etcd/healthcheck-client.crt /etc/kubernetes/pki/etcd/server.crt
Bash
๋ณต์‚ฌ
โ†’ etcd์˜ ํ˜ธ์ŠคํŠธ ์ธ์ฆ์„œ๋Š” /etc/kubernetes/pki/etcd/server.crt ์ด๋‹ค. ๋”ฐ๋ผ์„œ etcd์˜ ์ธ์ฆ์„œ ๊ฒฝ๋กœ๋ฅผ ๋ณ€๊ฒฝํ•œ๋‹ค.
4.
etcd.yaml ํŒŒ์ผ ์ˆ˜์ •
vi /etc/kubernetes/manifests/etcd.yaml # ๊ฒฝ๋กœ ์ˆ˜์ • - --cert-file=/etc/kubernetes/pki/etcd/server.crt
Bash
๋ณต์‚ฌ
5.
์ž‘๋™ ํ™•์ธ
etcd.yaml ํŒŒ์ผ์ด ๋ณ€๊ฒฝ๋˜์—ˆ๊ธฐ ๋•Œ๋ฌธ์— kube-apiserver๊ฐ€ ๋‹ค์‹œ ์‹œ์ž‘๋˜๋Š”๋ฐ ์‹œ๊ฐ„์ด ๊ฑธ๋ฆฌ๋ฉฐ ์™„๋ฃŒ๋œ ์ดํ›„์—๋Š” kubectl ๋ช…๋ น์ด ๊ฐ€๋Šฅํ•ด์ง„๋‹ค.
# apiserver ์ปจํ…Œ์ด๋„ˆ ์ƒํƒœ ํ™•์ธ crictl ps -a | grep kube-apiserver # etcd ์ปจํ…Œ์ด๋„ˆ ์ƒํƒœ ํ™•์ธ crictl ps -a | grep etcd # kubectl ๋™์ž‘ ํ™•์ธ kubectl get po
Bash
๋ณต์‚ฌ

Q13

The kube-api server stopped again! Check it out. Inspect the kube-api server logs and identify the root cause and fix the issue.

ํ’€์ด

12๋ฒˆ ๋ฌธ์ œ์™€ ๋˜‘๊ฐ™์€ ๋ฐฉ์‹์œผ๋กœ ํ’€์ด๋ฅผ ํ•˜๋ฉด ๋œ๋‹ค. ๋ฌธ์ œ์—์„œ kube-apiserver๊ฐ€ ์ค‘์ง€๋˜์—ˆ๊ธฐ ๋•Œ๋ฌธ์— kubectl์€ ๋™์ž‘์„ ํ•˜์ง€ ์•Š์œผ๋‹ˆ ์ปจํ…Œ์ด๋„ˆ์— ์ง์ ‘ ์ ‘๊ทผํ•ด์•ผ ํ•œ๋‹ค.
1.
kube-apiserver์™€ etcd์— ๋Œ€ํ•œ ์ปจํ…Œ์ด๋„ˆ ํ™•์ธ
$ controlplane ~ โœ– crictl ps -a | grep kube-apiserver 946de2fa8aa97 604f5db92eaa8 48 seconds ago Exited kube-apiserver 2 8dd35b937e301 kube-apiserver-controlplane $ controlplane ~ โžœ crictl logs 946 ... W0114 08:09:41.514449 1 logging.go:55] [core] [Channel #1 SubChannel #3]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority" F0114 08:09:44.072821 1 instance.go:225] Error creating leases: error creating storage factory: context deadline exceeded
Bash
๋ณต์‚ฌ
โ†’ 127.0.0.1:2379์— ๋Œ€ํ•œ CA๊ฐ€ ์ž˜๋ชป ์„ค์ •๋˜์—ˆ์Œ์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ๋‹ค
$ controlplane ~ โžœ crictl ps -a | grep etcd 581cb851e1e51 2e96e5913fc06 12 minutes ago Running etcd 0 874f426ac7c1d etcd-controlplane $ controlplane ~ โžœ crictl logs 581 ... {"level":"warn","ts":"2025-01-14T08:13:15.994300Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:53996","server-name":"","error":"remote error: tls: bad certificate"} {"level":"warn","ts":"2025-01-14T08:13:17.566497Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:54012","server-name":"","error":"remote error: tls: bad certificate"}
Bash
๋ณต์‚ฌ
โ†’ ETCD๋Š” ์ •์ƒ์ ์œผ๋กœ ์ž‘๋™ํ•˜๊ณ  ์žˆ์œผ๋ฉฐ ์™ธ๋ถ€์—์„œ ์ ‘๊ทผ์ด ์ด๋ค„์ง€๊ณ  ์žˆ๋Š”๋ฐ ์ธ์ฆ์„œ ๋ฌธ์ œ๋กœ ์ธํ•ด ์—ฐ๊ฒฐ์„ ๊ฑฐ๋ถ€ํ•˜๊ณ  ์žˆ๋‹ค
2.
kube-apiserver์˜ ์˜ต์…˜ ํ™•์ธ
$ controlplane ~ โžœ cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd - --etcd-cafile=/etc/kubernetes/pki/ca.crt - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key - --etcd-servers=https://127.0.0.1:2379 $ controlplane ~ โžœ ls /etc/kubernetes/pki | grep etcd apiserver-etcd-client.crt apiserver-etcd-client.key etcd $ controlplane ~ โžœ ls /etc/kubernetes/pki/etcd ca.crt healthcheck-client.crt peer.crt server.crt ca.key healthcheck-client.key peer.key server.key
Bash
๋ณต์‚ฌ
โ†’ etcd๋Š” etcd์˜ ca ์ธ์ฆ์„œ๋ฅผ ๋”ฐ๋กœ ๊ฐ€์ง€๊ณ  ์žˆ๋Š”๋ฐ kube-apiserver์˜ ์˜ต์…˜์—๋Š” ํ•ด๋‹น ์ธ์ฆ์„œ๋ฅผ ์‚ฌ์šฉํ•˜๊ณ  ์žˆ์ง€ ์•Š๋‹ค
3.
kube-apiserver.yaml ํŒŒ์ผ ์ˆ˜์ •
vi /etc/kubernetes/manifests/kube-apiserver.yaml # ํ•ด๋‹น ๊ฒฝ๋กœ๋กœ ์ˆ˜์ • - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
Bash
๋ณต์‚ฌ
4.
์ž‘๋™ ํ™•์ธ
โ€ข
kube-apiserver๊ฐ€ ์žฌ์‹คํ–‰๋˜๋ฉด ์—ฐ๊ฒฐ๊นŒ์ง€ ์•ฝ๊ฐ„์˜ ์‹œ๊ฐ„์ด ์†Œ์š”๋œ๋‹ค
$ controlplane ~ โžœ crictl ps -a | grep kube-apiserver 376fd539871dd 604f5db92eaa8 About a minute ago Running kube-apiserver 0 6cf103f38f651 kube-apiserver-controlplane $ controlplane ~ โžœ kubectl get po No resources found in default namespace.
Bash
๋ณต์‚ฌ
โ†’ ์ปจํ…Œ์ด๋„ˆ ์ƒํƒœ ํ™•์ธ ๋ฐ ์ž‘๋™ ํ™•์ธ