MicroK8s

This how to guide describes how to deploy the MicroK8s charm and connect with Ceph CSI to consume Ceph storage.

Requirements

• An existing bootstrapped Juju controller on a cloud like AWS, Azure, OpenStack, etc

Deploy MicroK8s

Create a new model called k8s-1, then deploy the microk8s charm. Ensure the specified constraints are enough to account for your workloads:

juju add-model k8s-1
juju deploy microk8s --constraints 'mem=4G root-disk=20G' --channel 1.28/stable

Deploy Ceph

Deploy a Charmed Ceph cluster. In this example, we will deploy a simple Ceph cluster with 1 mon and 6 OSDs:

juju deploy ceph-mon --channel quincy/stable --config monitor-count=1
juju deploy ceph-osd --channel quincy/stable -n 2 --storage osd-devices=3,5G

juju integrate ceph-mon ceph-osd

Please refer to the linked Ceph documentation for more details around configuring and managing a production-grade Ceph cluster.

Deploy Ceph CSI

The Ceph CSI charm is used to deploy Ceph CSI on the MicroK8s cluster so that it can consume storage from our Ceph clsuter.

Deploy the ceph-csi charm. Refer to the Configure Ceph CSI for the available configuration options.

juju deploy ceph-csi --channel 1.28/stable --config namespace=kube-system --config provisioner-replicas=1

Relate ceph-csi with microk8s over the kubernetes-info interface:

juju integrate ceph-csi:kubernetes-info microk8s

Also, relate ceph-csi with ceph-mon over the ceph-client interface:

juju integrate ceph-csi:ceph-client ceph-mon

Wait for everything to settle. After a while, the juju status output should look like this:

Model     Controller  Cloud/Region   Version  SLA          Timestamp
microk8s  zs          zerostack/KHY  3.1.5    unsupported  11:01:26+03:00

App       Version          Status  Scale  Charm     Channel        Rev  Exposed  Message
ceph-csi  v3.7.2,v0,v3...  active      1  ceph-csi  1.28/stable     36  no       Versions: cephfs=v3.7.2, config=v0, rbd=v3.7.2
ceph-mon  17.2.6           active      1  ceph-mon  quincy/stable  183  no       Unit is ready and clustered
ceph-osd  17.2.6           active      2  ceph-osd  quincy/stable  564  no       Unit is ready (3 OSD)
microk8s  1.28.1           active      1  microk8s  1.28/stable    213  no       node is ready

Unit           Workload  Agent  Machine  Public address  Ports      Message
ceph-mon/0*    active    idle   1        172.16.100.11              Unit is ready and clustered
ceph-osd/0*    active    idle   2        172.16.100.124             Unit is ready (3 OSD)
ceph-osd/1     active    idle   3        172.16.100.59              Unit is ready (3 OSD)
microk8s/0*    active    idle   0        172.16.100.217  16443/tcp  node is ready
  ceph-csi/0*  active    idle            172.16.100.217             Unit is ready

Machine  State    Address         Inst id                               Base          AZ    Message
0        started  172.16.100.217  9ef8f282-d39e-46bd-aa49-be56d52e3f6a  ubuntu@22.04  nova  ACTIVE
1        started  172.16.100.11   4e747c87-ad81-4bae-b971-f4539a431818  ubuntu@22.04  nova  ACTIVE
2        started  172.16.100.124  718cceb2-0dd4-4311-9193-f7102b504f84  ubuntu@22.04  nova  ACTIVE
3        started  172.16.100.59   274f87b3-4443-47d0-90e9-b7e38ef142b1  ubuntu@22.04  nova  ACTIVE

(Optional) Deploy CephFS

CephFS supports will enable the cephfs storage class, which can be used to provision ReadWriteMany PVCs on the MicroK8s cluster.

First, deploy ceph-fs and relate with ceph-mon. Please refer to the Ceph documentation for instructions on adding CephFS for a productiong-grade deployment:

juju deploy ceph-fs --channel quincy/stable
juju integrate ceph-fs:ceph-mds ceph-mon

Wait for the deployment to settle, check progress with juju status. The output should eventually look like this:

Model     Controller  Cloud/Region   Version  SLA          Timestamp
microk8s  zs          zerostack/KHY  3.1.5    unsupported  11:04:18+03:00

App       Version          Status  Scale  Charm     Channel        Rev  Exposed  Message
ceph-csi  v3.7.2,v0,v3...  active      1  ceph-csi  1.28/stable     36  no       Versions: cephfs=v3.7.2, config=v0, rbd=v3.7.2
ceph-fs   17.2.6           active      1  ceph-fs   quincy/stable   60  no       Unit is ready
ceph-mon  17.2.6           active      1  ceph-mon  quincy/stable  183  no       Unit is ready and clustered
ceph-osd  17.2.6           active      2  ceph-osd  quincy/stable  564  no       Unit is ready (3 OSD)
microk8s  1.28.1           active      1  microk8s  1.28/stable    213  no       node is ready

Unit           Workload  Agent  Machine  Public address  Ports      Message
ceph-fs/0*     active    idle   4        172.16.100.134             Unit is ready
ceph-mon/0*    active    idle   1        172.16.100.11              Unit is ready and clustered
ceph-osd/0*    active    idle   2        172.16.100.124             Unit is ready (3 OSD)
ceph-osd/1     active    idle   3        172.16.100.59              Unit is ready (3 OSD)
microk8s/0*    active    idle   0        172.16.100.217  16443/tcp  node is ready
  ceph-csi/1*  active    idle            172.16.100.217             Unit is ready

Machine  State    Address         Inst id                               Base          AZ    Message
0        started  172.16.100.217  9ef8f282-d39e-46bd-aa49-be56d52e3f6a  ubuntu@22.04  nova  ACTIVE
1        started  172.16.100.11   4e747c87-ad81-4bae-b971-f4539a431818  ubuntu@22.04  nova  ACTIVE
2        started  172.16.100.124  718cceb2-0dd4-4311-9193-f7102b504f84  ubuntu@22.04  nova  ACTIVE
3        started  172.16.100.59   274f87b3-4443-47d0-90e9-b7e38ef142b1  ubuntu@22.04  nova  ACTIVE
4        started  172.16.100.134  fe8a0e52-dff0-4432-bf6b-fac2cff4087f  ubuntu@22.04  nova  ACTIVE

After deployment is complete, enable CephFS support in the ceph-csi charm:

juju config ceph-csi cephfs-enable=true

Using Ceph

First, you can test the available storage classes in the MicroK8s cluster:

juju exec --unit microk8s/0 -- microk8s kubectl get storageclass

The output should look similar to this:

NAME                 PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
ceph-xfs (default)   rbd.csi.ceph.com      Delete          Immediate           true                   70m
ceph-ext4            rbd.csi.ceph.com      Delete          Immediate           true                   70m
cephfs               cephfs.csi.ceph.com   Delete          Immediate           true                   69m

As an example, let’s create a simple pod with a PVC that uses the ceph-ext4 storage class:

juju ssh microk8s/0 -- sudo microk8s kubectl create -f - <<EOF
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  storageClassName: ceph-ext4
  accessModes: [ReadWriteOnce]
  resources: { requests: { storage: 1Gi } }
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  volumes:
    - name: pvc
      persistentVolumeClaim:
        claimName: my-pvc
  containers:
    - name: nginx
      image: nginx
      ports:
        - containerPort: 80
      volumeMounts:
        - name: pvc
          mountPath: /usr/share/nginx/html
EOF

Shortly, an RBD volume will be created and mounted to the pod. You can verify that everything is running with:

juju exec --unit microk8s/0 -- sudo microk8s kubectl get pod,pvc -o wide

The output should look similar to this:

NAME        READY   STATUS    RESTARTS   AGE   IP            NODE                     NOMINATED NODE   READINESS GATES
pod/nginx   1/1     Running   0          43s   10.1.204.71   juju-9c4cd0-microk8s-0   <none>           <none>

NAME                           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE   VOLUMEMODE
persistentvolumeclaim/my-pvc   Bound    pvc-79aa802d-1864-4c28-9f97-5c34cf4cfcbc   1Gi        RWO            ceph-ext4      43s   Filesystem