microk8sのクラスターを仮想マシン(LXD)で作る

はじめに

Kubernetesの軽量版?であるmicrok8sがクラスタ組めるようになったと聞いて実際にやってみました。

Kubernetesのクラスタはhardwayかkubeadmだと思ってたんですけどね・・・

https://ubuntu.com/blog/introducing-ha-microk8s-the-ultra-reliable-minimal-kubernetes

インストール

まずはmicrok8sの入れ物となる仮想マシンを作ります。

なお、ここで作成される仮想マシンはネットワークがブリッジされています。

lxc init ubuntu:20.04 microk8s-0 -p network_bridge --vm 
lxc init ubuntu:20.04 microk8s-1 -p network_bridge --vm 
lxc init ubuntu:20.04 microk8s-2 -p network_bridge --vm 
lxc config device override microk8s-0 root size=32GiB
lxc config device override microk8s-1 root size=32GiB
lxc config device override microk8s-2 root size=32GiB
lxc config set microk8s-0 limits.cpu 4
lxc config set microk8s-1 limits.cpu 4
lxc config set microk8s-2 limits.cpu 4
lxc config set microk8s-0 limits.memory 8GB
lxc config set microk8s-1 limits.memory 8GB
lxc config set microk8s-2 limits.memory 8GB
lxc start microk8s-0
lxc start microk8s-1
lxc start microk8s-2 

仮想マシン内にmicrok8sをインストールします。

lxc exec microk8s-0 -- snap install microk8s --classic --channel=1.19/stable
lxc exec microk8s-1 -- snap install microk8s --classic --channel=1.19/stable
lxc exec microk8s-2 -- snap install microk8s --classic --channel=1.19/stable

完全に起動が終わると以下のようにネットワークができています。

+--------------------+---------+-----------------------------+----------------------------------------------+-----------------+-----------+
| microk8s-0         | RUNNING | 192.168.1.60 (enp5s0)       | fdda:f0f7:c6cc:0:216:3eff:fed8:7ebf (enp5s0) | VIRTUAL-MACHINE | 0         |
|                    |         | 10.1.33.64 (vxlan.calico)   |                                              |                 |           |
+--------------------+---------+-----------------------------+----------------------------------------------+-----------------+-----------+
| microk8s-1         | RUNNING | 192.168.1.61 (enp5s0)       | fdda:f0f7:c6cc:0:216:3eff:fe40:4ea0 (enp5s0) | VIRTUAL-MACHINE | 0         |
|                    |         | 10.1.35.128 (vxlan.calico)  |                                              |                 |           |
+--------------------+---------+-----------------------------+----------------------------------------------+-----------------+-----------+
| microk8s-2         | RUNNING | 192.168.1.62 (enp5s0)       | fdda:f0f7:c6cc:0:216:3eff:fea3:3726 (enp5s0) | VIRTUAL-MACHINE | 0         |
|                    |         | 10.1.100.128 (vxlan.calico) |                                              |                 |           |
+--------------------+---------+-----------------------------+----------------------------------------------+-----------------+-----------+

インストール後に、joinコードを吐き出させます。

lxc exec microk8s-0 -- microk8s add-node
From the node you wish to join to this cluster, run the following:
microk8s join 192.168.1.60:25000/eddce18b61f46c88f5326018b072bcc4
If the node you are adding is not reachable through the default interface you can use one of the following:
 microk8s join 192.168.1.60:25000/eddce18b61f46c88f5326018b072bcc4
 microk8s join 10.1.33.64:25000/eddce18b61f46c88f5326018b072bcc4

joinコマンドを実行してクラスターを作成します。結構時間がかかるので一台ずつやりましょう。

lxc exec microk8s-1 --  microk8s join 192.168.1.60:25000/eddce18b61f46c88f5326018b072bcc4
Contacting cluster at 192.168.1.60
Waiting for this node to finish joining the cluster. ..  

一度joinするとその認証コードは無効になるので、もう一度吐き出させます。

lxc exec microk8s-0 -- microk8s add-node
From the node you wish to join to this cluster, run the following:
microk8s join 192.168.1.60:25000/b43e320c3a806dc4b05836bc3d884ddf
If the node you are adding is not reachable through the default interface you can use one of the following:
 microk8s join 192.168.1.60:25000/b43e320c3a806dc4b05836bc3d884ddf
 microk8s join 10.1.33.64:25000/b43e320c3a806dc4b05836bc3d884ddf

吐き出した認証コードを実行します。

lxc exec microk8s-2 --  microk8s join 192.168.1.60:25000/b43e320c3a806dc4b05836bc3d884ddf
Contacting cluster at 192.168.1.60
Waiting for this node to finish joining the cluster. .. 

どのノードでも良いので起動させます。起動すると高可用性モードで動いていることがわかります。

lxc exec microk8s-0 -- microk8s.status
microk8s is running
high-availability: yes
  datastore master nodes: 192.168.1.60:19001 192.168.1.61:19001 192.168.1.62:19001
  datastore standby nodes: none
addons:
  enabled:
    ha-cluster           # Configure high availability on the current node
  disabled:
    ambassador           # Ambassador API Gateway and Ingress
    cilium               # SDN, fast with full network policy
    dashboard            # The Kubernetes dashboard
    dns                  # CoreDNS
    fluentd              # Elasticsearch-Fluentd-Kibana logging and monitoring
    gpu                  # Automatic enablement of Nvidia CUDA
    helm                 # Helm 2 - the package manager for Kubernetes
    helm3                # Helm 3 - Kubernetes package manager
    host-access          # Allow Pods connecting to Host services smoothly
    ingress              # Ingress controller for external access
    istio                # Core Istio service mesh services
    jaeger               # Kubernetes Jaeger operator with its simple config
    knative              # The Knative framework on Kubernetes.
    kubeflow             # Kubeflow for easy ML deployments
    linkerd              # Linkerd is a service mesh for Kubernetes and other frameworks
    metallb              # Loadbalancer for your Kubernetes cluster
    metrics-server       # K8s Metrics Server for API access to service metrics
    multus               # Multus CNI enables attaching multiple network interfaces to pods
    prometheus           # Prometheus operator for monitoring and logging
    rbac                 # Role-Based Access Control for authorisation
    registry             # Private image registry exposed on localhost:32000
    storage              # Storage class; allocates storage from host directory

ノードは以下のようになっています。

起動直後はNOTReadyになったりならなかったりと状態がフラフラするので、apiserverの上にHAが乗っている感じですかね?

 lxc exec microk8s-0 -- microk8s.kubectl get nodes
NAME         STATUS   ROLES    AGE     VERSION
microk8s-2   Ready    <none>   7m28s   v1.19.2-34+1b3fa60b402c1c
microk8s-0   Ready    <none>   19m     v1.19.2-34+1b3fa60b402c1c
microk8s-1   Ready    <none>   11m     v1.19.2-34+1b3fa60b402c1c

動作確認

まずはポッドが名前解決ができるようにDNSを作ります。

lxc exec microk8s-0 -- microk8s.enable dns
Enabling DNS
Applying manifest
serviceaccount/coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
clusterrole.rbac.authorization.k8s.io/coredns created
clusterrolebinding.rbac.authorization.k8s.io/coredns created
Restarting kubelet
Adding argument --cluster-domain to nodes.
Configuring node 192.168.1.62
Configuring node 192.168.1.60
Configuring node 192.168.1.61
Adding argument --cluster-dns to nodes.
Configuring node 192.168.1.62
Configuring node 192.168.1.60
Configuring node 192.168.1.61
Restarting nodes.
Configuring node 192.168.1.62
Configuring node 192.168.1.60
Configuring node 192.168.1.61
DNS is enabled

名前解決をしてみます

 lxc exec microk8s-0 -- microk8s.kubectl run test --image busybox:1.28 --restart Never -it --rm -- nslookup google.com
Server:    10.152.183.10
Address 1: 10.152.183.10 kube-dns.kube-system.svc.cluster.local
Name:      google.com
Address 1: 2404:6800:4004:810::200e nrt12s28-in-x0e.1e100.net
Address 2: 172.217.175.46 nrt20s19-in-f14.1e100.net
pod "test" deleted

よきかな(詠嘆)

最後にnginxでも動かして終わりにします。

$ lxc exec microk8s-0 -- microk8s.kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
$ lxc exec microk8s-0 -- microk8s.kubectl get pods -l app=nginx -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP             NODE         NOMINATED NODE   READINESS GATES
nginx-6799fc88d8-zmsrm   1/1     Running   0          32s   10.1.100.130   microk8s-2   <none>           <none>
POD_NAME=$(lxc exec microk8s-0 -- microk8s.kubectl get pods -l app=nginx -o jsonpath="{.items[0].metadata.name}")
lxc exec microk8s-0 -- microk8s.kubectl exec -ti $POD_NAME -- nginx -v
nginx version: nginx/1.19.3

ついでにロードバランサーも設置します。

とりあえず、DHCP範囲外を少し割り当てます。

lxc exec microk8s-0 -- microk8s.enable metallb
Enabling MetalLB
Enter each IP address range delimited by comma (e.g. '10.64.140.43-10.64.140.49,192.168.0.105-192.168.0.111'): 192.168.1.70-192.168.1.80
Applying Metallb manifest
namespace/metallb-system created
secret/memberlist created
podsecuritypolicy.policy/controller created
podsecuritypolicy.policy/speaker created
serviceaccount/controller created
serviceaccount/speaker created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
role.rbac.authorization.k8s.io/config-watcher created
role.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/config-watcher created
rolebinding.rbac.authorization.k8s.io/pod-lister created
daemonset.apps/speaker created
deployment.apps/controller created
configmap/config created
MetalLB is enabled

ロードバランサでアクセスするようにして・・・

~$ lxc exec microk8s-0 -- microk8s.kubectl expose deployment nginx --port 80 --type LoadBalancer
service/nginx exposed
~$ lxc exec microk8s-0 -- microk8s.kubectl get svc
NAME         TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)        AGE
kubernetes   ClusterIP      10.152.183.1     <none>         443/TCP        40m
nginx        LoadBalancer   10.152.183.218   192.168.1.70   80:31290/TCP   36s

やったぜ

問題としては常時CPUが持っていかれることですかね

やっぱり単一ノードでやることではない

おまけ

ダッシュボードでも表示させてみましょうか。立ち上がるまで待機

$ lxc exec microk8s-0 -- microk8s.enable dashboard
~$ lxc exec microk8s-0 -- microk8s kubectl -n kube-system get pods -o wide
NAME                                         READY   STATUS    RESTARTS   AGE   IP             NODE         NOMINATED NODE   READINESS GATES
calico-node-rlnw2                            1/1     Running   0          51m   192.168.1.62   microk8s-2   <none>           <none>
calico-node-2c6b6                            1/1     Running   0          56m   192.168.1.61   microk8s-1   <none>           <none>
coredns-86f78bb79c-7fg86                     1/1     Running   0          38m   10.1.35.131    microk8s-1   <none>           <none>
calico-kube-controllers-847c8c99d-lbpps      1/1     Running   0          64m   10.1.33.66     microk8s-0   <none>           <none>
calico-node-nf2lc                            1/1     Running   0          57m   192.168.1.60   microk8s-0   <none>           <none>
kubernetes-dashboard-7ffd448895-5hl7p        1/1     Running   0          82s   10.1.35.132    microk8s-1   <none>           <none>
dashboard-metrics-scraper-6c4568dc68-fm7mk   1/1     Running   0          82s   10.1.33.68     microk8s-0   <none>           <none>
metrics-server-8bbfb4bdb-kll4c               1/1     Running   0          84s   10.1.33.69     microk8s-0   <none>           <none>

トークンを表示させて・・・

$ lxc exec microk8s-0 -- microk8s kubectl -n kube-system describe secret $(lxc exec microk8s-0 -- microk8s kubectl -n kube-system get secret | grep default-token | awk '{print $1}')
Name:         default-token-2q7fj
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: default
              kubernetes.io/service-account.uid: 944534e9-f674-4834-832d-f3415f25b631
Type:  kubernetes.io/service-account-token
Data
====
ca.crt:     1103 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IkVzMUhwOU略

外部から見れるようにして・・・

$ lxc exec microk8s-0 -- microk8s.kubectl expose deployment kubernetes-dashboard --name=kubernetes-dashboard-lb --port 443 --target-port=8443 --type LoadBalancer -n kube-system

トークン貼り付けて・・・

入れました

あー良き

さてノード間はどう通信しているのだろうか

うーんこれは、ロードバランサ無い?apiserverだけで完結してるのかなるほど・・・(わかっていない)

~$ lxc exec microk8s-0 -- lsof -i -P -n  | grep IPv4
calico-no  307            root    6u  IPv4  51612      0t0  TCP 127.0.0.1:9099 (LISTEN)
calico-no  307            root   19u  IPv4 334238      0t0  TCP 192.168.1.60:55316->10.152.183.1:443 (ESTABLISHED)
kubelet    339            root   17u  IPv4 226172      0t0  TCP 127.0.0.1:10248 (LISTEN)
kubelet    339            root   19u  IPv4 334901      0t0  TCP 127.0.0.1:56110->127.0.0.1:16443 (ESTABLISHED)
systemd-n  373 systemd-network   21u  IPv4 206498      0t0  UDP 192.168.1.60:68 
systemd-r  375 systemd-resolve   12u  IPv4   8489      0t0  UDP 127.0.0.53:53 
systemd-r  375 systemd-resolve   13u  IPv4   8490      0t0  TCP 127.0.0.53:53 (LISTEN)
sshd       462            root    3u  IPv4  11811      0t0  TCP *:22 (LISTEN)
kube-sche 1009            root    7u  IPv4 252109      0t0  TCP 127.0.0.1:10251 (LISTEN)
kube-sche 1009            root   10u  IPv4 334421      0t0  TCP 127.0.0.1:56122->127.0.0.1:16443 (ESTABLISHED)
kube-cont 1145            root    7u  IPv4 252134      0t0  TCP 127.0.0.1:10252 (LISTEN)
kube-cont 1145            root    9u  IPv4 334972      0t0  TCP 127.0.0.1:56176->127.0.0.1:16443 (ESTABLISHED)
python3   1446            root    5u  IPv4  16225      0t0  TCP *:25000 (LISTEN)
kube-apis 1680            root   15u  IPv4 331604      0t0  TCP 192.168.1.60:19001 (LISTEN)
kube-apis 1680            root   16u  IPv4 331662      0t0  TCP 192.168.1.60:19001->192.168.1.61:48338 (ESTABLISHED)
kube-apis 1680            root   18u  IPv4 331714      0t0  TCP 192.168.1.60:19001->192.168.1.62:35376 (ESTABLISHED)
kube-apis 1680            root   22u  IPv4 334889      0t0  TCP 192.168.1.60:60124->192.168.1.62:19001 (ESTABLISHED)
kube-apis 1680            root   23u  IPv4 333511      0t0  TCP 192.168.1.60:60110->192.168.1.62:19001 (ESTABLISHED)
kube-apis 1680            root   24u  IPv4 334891      0t0  TCP 192.168.1.60:60154->192.168.1.62:19001 (ESTABLISHED)
kube-apis 1680            root   32u  IPv4 331744      0t0  TCP 192.168.1.60:60162->192.168.1.62:19001 (ESTABLISHED)
kube-apis 1680            root   33u  IPv4 334902      0t0  TCP 192.168.1.60:60164->192.168.1.62:19001 (ESTABLISHED)
kube-apis 1680            root   41u  IPv4 334914      0t0  TCP 192.168.1.60:60196->192.168.1.62:19001 (ESTABLISHED)
kube-apis 1680            root  180u  IPv4 333671      0t0  TCP 192.168.1.60:39208->10.1.33.68:8000 (ESTABLISHED)
kube-apis 1680            root  184u  IPv4 334576      0t0  TCP 192.168.1.60:38500->10.152.183.78:443 (ESTABLISHED)
kube-apis 1680            root  188u  IPv4 334737      0t0  TCP 192.168.1.60:38516->10.152.183.78:443 (ESTABLISHED)
python3   1714            root    5u  IPv4  16225      0t0  TCP *:25000 (LISTEN)
speaker   3596            root    6u  IPv4 143894      0t0  TCP 192.168.1.60:7946 (LISTEN)
speaker   3596            root    7u  IPv4 143898      0t0  UDP 192.168.1.60:7946 
speaker   3596            root    9u  IPv4 142008      0t0  TCP 192.168.1.60:7472 (LISTEN)
speaker   3596            root   21u  IPv4 332383      0t0  TCP 192.168.1.60:55330->10.152.183.1:443 (ESTABLISHED)
container 3901            root    8u  IPv4  26225      0t0  TCP 127.0.0.1:1338 (LISTEN)
container 3901            root   12u  IPv4  27970      0t0  TCP 127.0.0.1:44893 (LISTEN)
kube-prox 4045            root    3u  IPv4 333610      0t0  TCP 127.0.0.1:56162->127.0.0.1:16443 (ESTABLISHED)
kube-prox 4045            root    7u  IPv4 153813      0t0  TCP *:31290 (LISTEN)
kube-prox 4045            root    8u  IPv4 281503      0t0  TCP *:30194 (LISTEN)
kube-prox 4045            root   11u  IPv4  28959      0t0  TCP 127.0.0.1:10256 (LISTEN)
kube-prox 4045            root   14u  IPv4  28961      0t0  TCP 127.0.0.1:10249 (LISTEN)