microk8sのクラスターを仮想マシン(LXD)で作る
はじめに
Kubernetesの軽量版？であるmicrok8sがクラスタ組めるようになったと聞いて実際にやってみました。
Kubernetesのクラスタはhardwayかkubeadmだと思ってたんですけどね・・・
https://ubuntu.com/blog/introducing-ha-microk8s-the-ultra-reliable-minimal-kubernetes
インストール
まずはmicrok8sの入れ物となる仮想マシンを作ります。
なお、ここで作成される仮想マシンはネットワークがブリッジされています。
lxc init ubuntu:20.04 microk8s-0 -p network_bridge --vm
lxc init ubuntu:20.04 microk8s-1 -p network_bridge --vm
lxc init ubuntu:20.04 microk8s-2 -p network_bridge --vm
lxc config device override microk8s-0 root size=32GiB
lxc config device override microk8s-1 root size=32GiB
lxc config device override microk8s-2 root size=32GiB
lxc config set microk8s-0 limits.cpu 4
lxc config set microk8s-1 limits.cpu 4
lxc config set microk8s-2 limits.cpu 4
lxc config set microk8s-0 limits.memory 8GB
lxc config set microk8s-1 limits.memory 8GB
lxc config set microk8s-2 limits.memory 8GB
lxc start microk8s-0
lxc start microk8s-1
lxc start microk8s-2
仮想マシン内にmicrok8sをインストールします。
lxc exec microk8s-0 -- snap install microk8s --classic --channel=1.19/stable
lxc exec microk8s-1 -- snap install microk8s --classic --channel=1.19/stable
lxc exec microk8s-2 -- snap install microk8s --classic --channel=1.19/stable
完全に起動が終わると以下のようにネットワークができています。
+--------------------+---------+-----------------------------+----------------------------------------------+-----------------+-----------+
| microk8s-0 | RUNNING | 192.168.1.60 (enp5s0) | fdda:f0f7:c6cc:0:216:3eff:fed8:7ebf (enp5s0) | VIRTUAL-MACHINE | 0 |
| | | 10.1.33.64 (vxlan.calico) | | | |
+--------------------+---------+-----------------------------+----------------------------------------------+-----------------+-----------+
| microk8s-1 | RUNNING | 192.168.1.61 (enp5s0) | fdda:f0f7:c6cc:0:216:3eff:fe40:4ea0 (enp5s0) | VIRTUAL-MACHINE | 0 |
| | | 10.1.35.128 (vxlan.calico) | | | |
+--------------------+---------+-----------------------------+----------------------------------------------+-----------------+-----------+
| microk8s-2 | RUNNING | 192.168.1.62 (enp5s0) | fdda:f0f7:c6cc:0:216:3eff:fea3:3726 (enp5s0) | VIRTUAL-MACHINE | 0 |
| | | 10.1.100.128 (vxlan.calico) | | | |
+--------------------+---------+-----------------------------+----------------------------------------------+-----------------+-----------+
インストール後に、joinコードを吐き出させます。
lxc exec microk8s-0 -- microk8s add-node
From the node you wish to join to this cluster, run the following:
microk8s join 192.168.1.60:25000/eddce18b61f46c88f5326018b072bcc4
If the node you are adding is not reachable through the default interface you can use one of the following:
microk8s join 192.168.1.60:25000/eddce18b61f46c88f5326018b072bcc4
microk8s join 10.1.33.64:25000/eddce18b61f46c88f5326018b072bcc4
joinコマンドを実行してクラスターを作成します。結構時間がかかるので一台ずつやりましょう。
lxc exec microk8s-1 -- microk8s join 192.168.1.60:25000/eddce18b61f46c88f5326018b072bcc4
Contacting cluster at 192.168.1.60
Waiting for this node to finish joining the cluster. ..
一度joinするとその認証コードは無効になるので、もう一度吐き出させます。
lxc exec microk8s-0 -- microk8s add-node
From the node you wish to join to this cluster, run the following:
microk8s join 192.168.1.60:25000/b43e320c3a806dc4b05836bc3d884ddf
If the node you are adding is not reachable through the default interface you can use one of the following:
microk8s join 192.168.1.60:25000/b43e320c3a806dc4b05836bc3d884ddf
microk8s join 10.1.33.64:25000/b43e320c3a806dc4b05836bc3d884ddf
吐き出した認証コードを実行します。
lxc exec microk8s-2 -- microk8s join 192.168.1.60:25000/b43e320c3a806dc4b05836bc3d884ddf
Contacting cluster at 192.168.1.60
Waiting for this node to finish joining the cluster. ..
どのノードでも良いので起動させます。起動すると高可用性モードで動いていることがわかります。
lxc exec microk8s-0 -- microk8s.status
microk8s is running
high-availability: yes
datastore master nodes: 192.168.1.60:19001 192.168.1.61:19001 192.168.1.62:19001
datastore standby nodes: none
addons:
enabled:
ha-cluster # Configure high availability on the current node
disabled:
ambassador # Ambassador API Gateway and Ingress
cilium # SDN, fast with full network policy
dashboard # The Kubernetes dashboard
dns # CoreDNS
fluentd # Elasticsearch-Fluentd-Kibana logging and monitoring
gpu # Automatic enablement of Nvidia CUDA
helm # Helm 2 - the package manager for Kubernetes
helm3 # Helm 3 - Kubernetes package manager
host-access # Allow Pods connecting to Host services smoothly
ingress # Ingress controller for external access
istio # Core Istio service mesh services
jaeger # Kubernetes Jaeger operator with its simple config
knative # The Knative framework on Kubernetes.
kubeflow # Kubeflow for easy ML deployments
linkerd # Linkerd is a service mesh for Kubernetes and other frameworks
metallb # Loadbalancer for your Kubernetes cluster
metrics-server # K8s Metrics Server for API access to service metrics
multus # Multus CNI enables attaching multiple network interfaces to pods
prometheus # Prometheus operator for monitoring and logging
rbac # Role-Based Access Control for authorisation
registry # Private image registry exposed on localhost:32000
storage # Storage class; allocates storage from host directory
ノードは以下のようになっています。
起動直後はNOTReadyになったりならなかったりと状態がフラフラするので、apiserverの上にHAが乗っている感じですかね？
lxc exec microk8s-0 -- microk8s.kubectl get nodes
NAME STATUS ROLES AGE VERSION
microk8s-2 Ready <none> 7m28s v1.19.2-34+1b3fa60b402c1c
microk8s-0 Ready <none> 19m v1.19.2-34+1b3fa60b402c1c
microk8s-1 Ready <none> 11m v1.19.2-34+1b3fa60b402c1c
動作確認
まずはポッドが名前解決ができるようにDNSを作ります。
lxc exec microk8s-0 -- microk8s.enable dns
Enabling DNS
Applying manifest
serviceaccount/coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
clusterrole.rbac.authorization.k8s.io/coredns created
clusterrolebinding.rbac.authorization.k8s.io/coredns created
Restarting kubelet
Adding argument --cluster-domain to nodes.
Configuring node 192.168.1.62
Configuring node 192.168.1.60
Configuring node 192.168.1.61
Adding argument --cluster-dns to nodes.
Configuring node 192.168.1.62
Configuring node 192.168.1.60
Configuring node 192.168.1.61
Restarting nodes.
Configuring node 192.168.1.62
Configuring node 192.168.1.60
Configuring node 192.168.1.61
DNS is enabled
名前解決をしてみます
lxc exec microk8s-0 -- microk8s.kubectl run test --image busybox:1.28 --restart Never -it --rm -- nslookup google.com
Server: 10.152.183.10
Address 1: 10.152.183.10 kube-dns.kube-system.svc.cluster.local
Name: google.com
Address 1: 2404:6800:4004:810::200e nrt12s28-in-x0e.1e100.net
Address 2: 172.217.175.46 nrt20s19-in-f14.1e100.net
pod "test" deleted
よきかな(詠嘆)
最後にnginxでも動かして終わりにします。
$ lxc exec microk8s-0 -- microk8s.kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
$ lxc exec microk8s-0 -- microk8s.kubectl get pods -l app=nginx -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-6799fc88d8-zmsrm 1/1 Running 0 32s 10.1.100.130 microk8s-2 <none> <none>
POD_NAME=$(lxc exec microk8s-0 -- microk8s.kubectl get pods -l app=nginx -o jsonpath="{.items[0].metadata.name}")
lxc exec microk8s-0 -- microk8s.kubectl exec -ti $POD_NAME -- nginx -v
nginx version: nginx/1.19.3
ついでにロードバランサーも設置します。
とりあえず、DHCP範囲外を少し割り当てます。
lxc exec microk8s-0 -- microk8s.enable metallb
Enabling MetalLB
Enter each IP address range delimited by comma (e.g. '10.64.140.43-10.64.140.49,192.168.0.105-192.168.0.111'): 192.168.1.70-192.168.1.80
Applying Metallb manifest
namespace/metallb-system created
secret/memberlist created
podsecuritypolicy.policy/controller created
podsecuritypolicy.policy/speaker created
serviceaccount/controller created
serviceaccount/speaker created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
role.rbac.authorization.k8s.io/config-watcher created
role.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/config-watcher created
rolebinding.rbac.authorization.k8s.io/pod-lister created
daemonset.apps/speaker created
deployment.apps/controller created
configmap/config created
MetalLB is enabled
ロードバランサでアクセスするようにして・・・
~$ lxc exec microk8s-0 -- microk8s.kubectl expose deployment nginx --port 80 --type LoadBalancer
service/nginx exposed
~$ lxc exec microk8s-0 -- microk8s.kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 40m
nginx LoadBalancer 10.152.183.218 192.168.1.70 80:31290/TCP 36s
やったぜ
問題としては常時CPUが持っていかれることですかね
やっぱり単一ノードでやることではない
おまけ
ダッシュボードでも表示させてみましょうか。立ち上がるまで待機
$ lxc exec microk8s-0 -- microk8s.enable dashboard
~$ lxc exec microk8s-0 -- microk8s kubectl -n kube-system get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-node-rlnw2 1/1 Running 0 51m 192.168.1.62 microk8s-2 <none> <none>
calico-node-2c6b6 1/1 Running 0 56m 192.168.1.61 microk8s-1 <none> <none>
coredns-86f78bb79c-7fg86 1/1 Running 0 38m 10.1.35.131 microk8s-1 <none> <none>
calico-kube-controllers-847c8c99d-lbpps 1/1 Running 0 64m 10.1.33.66 microk8s-0 <none> <none>
calico-node-nf2lc 1/1 Running 0 57m 192.168.1.60 microk8s-0 <none> <none>
kubernetes-dashboard-7ffd448895-5hl7p 1/1 Running 0 82s 10.1.35.132 microk8s-1 <none> <none>
dashboard-metrics-scraper-6c4568dc68-fm7mk 1/1 Running 0 82s 10.1.33.68 microk8s-0 <none> <none>
metrics-server-8bbfb4bdb-kll4c 1/1 Running 0 84s 10.1.33.69 microk8s-0 <none> <none>
トークンを表示させて・・・
$ lxc exec microk8s-0 -- microk8s kubectl -n kube-system describe secret $(lxc exec microk8s-0 -- microk8s kubectl -n kube-system get secret | grep default-token | awk '{print $1}')
Name: default-token-2q7fj
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name: default
kubernetes.io/service-account.uid: 944534e9-f674-4834-832d-f3415f25b631
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1103 bytes
namespace: 11 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IkVzMUhwOU略
外部から見れるようにして・・・
$ lxc exec microk8s-0 -- microk8s.kubectl expose deployment kubernetes-dashboard --name=kubernetes-dashboard-lb --port 443 --target-port=8443 --type LoadBalancer -n kube-system
トークン貼り付けて・・・
入れました
あー良き
さてノード間はどう通信しているのだろうか
うーんこれは、ロードバランサ無い？apiserverだけで完結してるのかなるほど・・・（わかっていない）
~$ lxc exec microk8s-0 -- lsof -i -P -n | grep IPv4
calico-no 307 root 6u IPv4 51612 0t0 TCP 127.0.0.1:9099 (LISTEN)
calico-no 307 root 19u IPv4 334238 0t0 TCP 192.168.1.60:55316->10.152.183.1:443 (ESTABLISHED)
kubelet 339 root 17u IPv4 226172 0t0 TCP 127.0.0.1:10248 (LISTEN)
kubelet 339 root 19u IPv4 334901 0t0 TCP 127.0.0.1:56110->127.0.0.1:16443 (ESTABLISHED)
systemd-n 373 systemd-network 21u IPv4 206498 0t0 UDP 192.168.1.60:68
systemd-r 375 systemd-resolve 12u IPv4 8489 0t0 UDP 127.0.0.53:53
systemd-r 375 systemd-resolve 13u IPv4 8490 0t0 TCP 127.0.0.53:53 (LISTEN)
sshd 462 root 3u IPv4 11811 0t0 TCP *:22 (LISTEN)
kube-sche 1009 root 7u IPv4 252109 0t0 TCP 127.0.0.1:10251 (LISTEN)
kube-sche 1009 root 10u IPv4 334421 0t0 TCP 127.0.0.1:56122->127.0.0.1:16443 (ESTABLISHED)
kube-cont 1145 root 7u IPv4 252134 0t0 TCP 127.0.0.1:10252 (LISTEN)
kube-cont 1145 root 9u IPv4 334972 0t0 TCP 127.0.0.1:56176->127.0.0.1:16443 (ESTABLISHED)
python3 1446 root 5u IPv4 16225 0t0 TCP *:25000 (LISTEN)
kube-apis 1680 root 15u IPv4 331604 0t0 TCP 192.168.1.60:19001 (LISTEN)
kube-apis 1680 root 16u IPv4 331662 0t0 TCP 192.168.1.60:19001->192.168.1.61:48338 (ESTABLISHED)
kube-apis 1680 root 18u IPv4 331714 0t0 TCP 192.168.1.60:19001->192.168.1.62:35376 (ESTABLISHED)
kube-apis 1680 root 22u IPv4 334889 0t0 TCP 192.168.1.60:60124->192.168.1.62:19001 (ESTABLISHED)
kube-apis 1680 root 23u IPv4 333511 0t0 TCP 192.168.1.60:60110->192.168.1.62:19001 (ESTABLISHED)
kube-apis 1680 root 24u IPv4 334891 0t0 TCP 192.168.1.60:60154->192.168.1.62:19001 (ESTABLISHED)
kube-apis 1680 root 32u IPv4 331744 0t0 TCP 192.168.1.60:60162->192.168.1.62:19001 (ESTABLISHED)
kube-apis 1680 root 33u IPv4 334902 0t0 TCP 192.168.1.60:60164->192.168.1.62:19001 (ESTABLISHED)
kube-apis 1680 root 41u IPv4 334914 0t0 TCP 192.168.1.60:60196->192.168.1.62:19001 (ESTABLISHED)
kube-apis 1680 root 180u IPv4 333671 0t0 TCP 192.168.1.60:39208->10.1.33.68:8000 (ESTABLISHED)
kube-apis 1680 root 184u IPv4 334576 0t0 TCP 192.168.1.60:38500->10.152.183.78:443 (ESTABLISHED)
kube-apis 1680 root 188u IPv4 334737 0t0 TCP 192.168.1.60:38516->10.152.183.78:443 (ESTABLISHED)
python3 1714 root 5u IPv4 16225 0t0 TCP *:25000 (LISTEN)
speaker 3596 root 6u IPv4 143894 0t0 TCP 192.168.1.60:7946 (LISTEN)
speaker 3596 root 7u IPv4 143898 0t0 UDP 192.168.1.60:7946
speaker 3596 root 9u IPv4 142008 0t0 TCP 192.168.1.60:7472 (LISTEN)
speaker 3596 root 21u IPv4 332383 0t0 TCP 192.168.1.60:55330->10.152.183.1:443 (ESTABLISHED)
container 3901 root 8u IPv4 26225 0t0 TCP 127.0.0.1:1338 (LISTEN)
container 3901 root 12u IPv4 27970 0t0 TCP 127.0.0.1:44893 (LISTEN)
kube-prox 4045 root 3u IPv4 333610 0t0 TCP 127.0.0.1:56162->127.0.0.1:16443 (ESTABLISHED)
kube-prox 4045 root 7u IPv4 153813 0t0 TCP *:31290 (LISTEN)
kube-prox 4045 root 8u IPv4 281503 0t0 TCP *:30194 (LISTEN)
kube-prox 4045 root 11u IPv4 28959 0t0 TCP 127.0.0.1:10256 (LISTEN)
kube-prox 4045 root 14u IPv4 28961 0t0 TCP 127.0.0.1:10249 (LISTEN)