Q12
Kubectl suddenly stops responding to your commands. Check it out! Someone recently modified theย /etc/kubernetes/manifests/etcd.yamlย file
ํ์ด
1.
kubectl ์๋ํ์ธ
โข
kubectl์ด ์๋ํ์ง ์๋๋ค๋ ๊ฒ์ kube-apiserver, etcd-server์ ๋ฌธ์ ๊ฐ ์๋ค๋ ๋ป์ด๋ค.
$ controlplane ~ โ kubectl get po
The connection to the server controlplane:6443 was refused - did you specify the right host or port?
Bash
๋ณต์ฌ
2.
kube-apiserver์ etcd์ ๋ํ ์ปจํ
์ด๋ ํ์ธ
โข
kubeadm์ผ๋ก ํด๋ฌ์คํฐ๋ฅผ ๊ตฌ์ฑํ์ ๊ฒฝ์ฐ Pod๋ก ์ปดํฌ๋ํธ๋ค์ด ๋ฐฐํฌ๋๊ธฐ ๋๋ฌธ์ kubectl์ด ๋์ํ์ง ์์ผ๋ฉด Pod์ Log๋ฅผ ํ์ธํ ์ ์๋ค.
$ controlplane ~ โ crictl ps -a | grep kube-apiserver
8ae9a07114306 604f5db92eaa8 59 seconds ago Exited kube-apiserver 5 a2a0ad40b4e4f kube-apiserver-controlplane
$ controlplane ~ โ crictl logs 8ae9a07114306
...
W0114 07:50:45.959183 1 logging.go:55] [core] [Channel #2 SubChannel #4]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0114 07:50:47.204132 1 logging.go:55] [core] [Channel #1 SubChannel #3]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
F0114 07:50:51.314580 1 instance.go:225] Error creating leases: error creating storage factory: context deadline exceeded
Bash
๋ณต์ฌ
โ 127.0.0.1:2379์ ๋ํ ์ ๊ทผ ์๋ฌ๊ฐ ๋ฐ์ํ๊ณ ์๋ค. ์ฆ ETCD๋ฅผ ํ์ธํด์ผ ํ๋ค
$ controlplane ~ โ crictl ps -a | grep etcd
1cf6df5a713ae 2e96e5913fc06 About a minute ago Exited etcd 6 7db3425fb37c6 etcd-controlplane
$ controlplane ~ โ crictl logs 1cf6df5a713ae
...
{"level":"info","ts":"2025-01-14T07:53:05.002032Z","caller":"etcdmain/main.go:50","msg":"successfully notified init daemon"}
{"level":"fatal","ts":"2025-01-14T07:53:05.002038Z","caller":"etcdmain/etcd.go:219","msg":"listener failed","error":"open /etc/kubernetes/pki/etcd/server-certificate.crt: no such file or directory","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:219\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:267"}
Bash
๋ณต์ฌ
โ /etc/kubernetes/pki/etcd/server-certificate.crt ํ์ผ์ด ์๋ค๋ ์๋ฌ๋ฅผ ํ์ธํ ์ ์๋ค. ๋ฐ๋ผ์ etcd์ ์ธ์ฆ์๋ฅผ ํ์ธํด์ผ ํ๋ค.
3.
etcd ์ธ์ฆ์ ํ์ธ
$ controlplane ~ โ ls /etc/kubernetes/pki/etcd/*.crt
/etc/kubernetes/pki/etcd/ca.crt /etc/kubernetes/pki/etcd/peer.crt
/etc/kubernetes/pki/etcd/healthcheck-client.crt /etc/kubernetes/pki/etcd/server.crt
Bash
๋ณต์ฌ
โ etcd์ ํธ์คํธ ์ธ์ฆ์๋ /etc/kubernetes/pki/etcd/server.crt ์ด๋ค. ๋ฐ๋ผ์ etcd์ ์ธ์ฆ์ ๊ฒฝ๋ก๋ฅผ ๋ณ๊ฒฝํ๋ค.
4.
etcd.yaml ํ์ผ ์์
vi /etc/kubernetes/manifests/etcd.yaml
# ๊ฒฝ๋ก ์์
- --cert-file=/etc/kubernetes/pki/etcd/server.crt
Bash
๋ณต์ฌ
5.
์๋ ํ์ธ
etcd.yaml ํ์ผ์ด ๋ณ๊ฒฝ๋์๊ธฐ ๋๋ฌธ์ kube-apiserver๊ฐ ๋ค์ ์์๋๋๋ฐ ์๊ฐ์ด ๊ฑธ๋ฆฌ๋ฉฐ ์๋ฃ๋ ์ดํ์๋ kubectl ๋ช
๋ น์ด ๊ฐ๋ฅํด์ง๋ค.
# apiserver ์ปจํ
์ด๋ ์ํ ํ์ธ
crictl ps -a | grep kube-apiserver
# etcd ์ปจํ
์ด๋ ์ํ ํ์ธ
crictl ps -a | grep etcd
# kubectl ๋์ ํ์ธ
kubectl get po
Bash
๋ณต์ฌ
Q13
The kube-api server stopped again! Check it out. Inspect the kube-api server logs and identify the root cause and fix the issue.
ํ์ด
12๋ฒ ๋ฌธ์ ์ ๋๊ฐ์ ๋ฐฉ์์ผ๋ก ํ์ด๋ฅผ ํ๋ฉด ๋๋ค. ๋ฌธ์ ์์ kube-apiserver๊ฐ ์ค์ง๋์๊ธฐ ๋๋ฌธ์ kubectl์ ๋์์ ํ์ง ์์ผ๋ ์ปจํ
์ด๋์ ์ง์ ์ ๊ทผํด์ผ ํ๋ค.
1.
kube-apiserver์ etcd์ ๋ํ ์ปจํ
์ด๋ ํ์ธ
$ controlplane ~ โ crictl ps -a | grep kube-apiserver
946de2fa8aa97 604f5db92eaa8 48 seconds ago Exited kube-apiserver 2 8dd35b937e301 kube-apiserver-controlplane
$ controlplane ~ โ crictl logs 946
...
W0114 08:09:41.514449 1 logging.go:55] [core] [Channel #1 SubChannel #3]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
F0114 08:09:44.072821 1 instance.go:225] Error creating leases: error creating storage factory: context deadline exceeded
Bash
๋ณต์ฌ
โ 127.0.0.1:2379์ ๋ํ CA๊ฐ ์๋ชป ์ค์ ๋์์์ ํ์ธํ ์ ์๋ค
$ controlplane ~ โ crictl ps -a | grep etcd
581cb851e1e51 2e96e5913fc06 12 minutes ago Running etcd 0 874f426ac7c1d etcd-controlplane
$ controlplane ~ โ crictl logs 581
...
{"level":"warn","ts":"2025-01-14T08:13:15.994300Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:53996","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2025-01-14T08:13:17.566497Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:54012","server-name":"","error":"remote error: tls: bad certificate"}
Bash
๋ณต์ฌ
โ ETCD๋ ์ ์์ ์ผ๋ก ์๋ํ๊ณ ์์ผ๋ฉฐ ์ธ๋ถ์์ ์ ๊ทผ์ด ์ด๋ค์ง๊ณ ์๋๋ฐ ์ธ์ฆ์ ๋ฌธ์ ๋ก ์ธํด ์ฐ๊ฒฐ์ ๊ฑฐ๋ถํ๊ณ ์๋ค
2.
kube-apiserver์ ์ต์
ํ์ธ
$ controlplane ~ โ cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd
- --etcd-cafile=/etc/kubernetes/pki/ca.crt
- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
- --etcd-servers=https://127.0.0.1:2379
$ controlplane ~ โ ls /etc/kubernetes/pki | grep etcd
apiserver-etcd-client.crt
apiserver-etcd-client.key
etcd
$ controlplane ~ โ ls /etc/kubernetes/pki/etcd
ca.crt healthcheck-client.crt peer.crt server.crt
ca.key healthcheck-client.key peer.key server.key
Bash
๋ณต์ฌ
โ etcd๋ etcd์ ca ์ธ์ฆ์๋ฅผ ๋ฐ๋ก ๊ฐ์ง๊ณ ์๋๋ฐ kube-apiserver์ ์ต์
์๋ ํด๋น ์ธ์ฆ์๋ฅผ ์ฌ์ฉํ๊ณ ์์ง ์๋ค
3.
kube-apiserver.yaml ํ์ผ ์์
vi /etc/kubernetes/manifests/kube-apiserver.yaml
# ํด๋น ๊ฒฝ๋ก๋ก ์์
- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
Bash
๋ณต์ฌ
4.
์๋ ํ์ธ
โข
kube-apiserver๊ฐ ์ฌ์คํ๋๋ฉด ์ฐ๊ฒฐ๊น์ง ์ฝ๊ฐ์ ์๊ฐ์ด ์์๋๋ค
$ controlplane ~ โ crictl ps -a | grep kube-apiserver
376fd539871dd 604f5db92eaa8 About a minute ago Running kube-apiserver 0 6cf103f38f651 kube-apiserver-controlplane
$ controlplane ~ โ kubectl get po
No resources found in default namespace.
Bash
๋ณต์ฌ
โ ์ปจํ
์ด๋ ์ํ ํ์ธ ๋ฐ ์๋ ํ์ธ