Kubeflow

This guide describes how you can get started with Charmed Kubeflow (CKF), from deploying to accessing it.

CKF provides a simple, out-of-the-box way to deploy Kubeflow with a single command. It sets default configurations, while still providing flexibility to configure it as you like.

Requirements:

• Ubuntu 20.04 (Focal) or later.
• A host machine with at least a 4-core CPU processor, 32GB RAM and 50GB of disk space available.

Install and configure MicroK8s

Every Kubeflow instance needs a Kubernetes (K8s) cluster to run on. For this tutorial, you will be running a Kubeflow instance on MicroK8s, which provides a light way to spin up a K8s cluster.

MicroK8s is installed from a snap package with strict confinement. First, install MicroK8s as follows:

sudo snap install microk8s --channel=1.29-strict/stable

More info: Install MicroK8s

• The published snap maintains different channels for different releases of K8s. See Supported versions for compatibility between K8s and CKF.

• If you plan to use GPU with Charmed Kubeflow, then the Microk8s snap must be installed with classic confinement.

After MicroK8s is installed, it automatically starts running in the background. You need sufficient permissions to access MicroK8s. You can get those by running the following command:

sudo usermod -a -G snap_microk8s ubuntu

To refresh the newly granted permissions, restart your machine or run the following command:

newgrp snap_microk8s

MicroK8s is a light functional distribution of Kubernetes. For deploying CKF, additional Kubernetes features are needed, including a DNS service, storage, an ingress controller so that Kubeflow components can be accessed by the user and the MetalLB load balancer application.

These can be installed as MicroK8s add-ons. Run the following command to enable the required MicroK8s add-ons:

sudo microk8s enable dns hostpath-storage metallb:10.64.140.43-10.64.140.49 rbac

To confirm that all add-ons are available, run the following command:

microk8s status

Check that MicroK8s is running, and all addons are listed as enabled. If this is not the case, wait a few minutes and run the command again.

Install Juju

Juju is an Operation Lifecycle Manager (OLM) for bare metal, clouds or K8s. You will be using it to deploy and manage the components which make up Kubeflow.

To install Juju using snap, run this command:

sudo snap install juju --channel=3.4/stable

On some machines there might be a missing folder which is required for Juju to run correctly. To ensure that this folder exists, run:

mkdir -p ~/.local/share

As a next step, configure MicroK8s to work properly with Juju by running:

microk8s config | juju add-k8s my-k8s --client

The microk8s config command retrieves the client’s Kubernetes config which is then registered to Juju Kubernetes endpoints.

Next, run the following command to deploy a Juju controller to the Kubernetes that was set up with MicroK8s:

juju bootstrap my-k8s uk8sx

The controller is a Juju’s agent, running on Kubernetes, which can be used to deploy and control the Kubeflow components.

You now need to add a model for Kubeflow to the controller. The controller can work with different models, which map 1:1 to namespaces in K8s. In this case, the model name must be kubeflow. Run the following command to add the kubeflow model:

juju add-model kubeflow

Deploy CKF

MicroK8s uses inotify to interact with the file system. Kubeflow requires increasing the inotify limits. To do so, run the following commands:

sudo sysctl fs.inotify.max_user_instances=1280
sudo sysctl fs.inotify.max_user_watches=655360

If you want these commands to persist across machine restarts, add the following lines to /etc/sysctl.conf:

fs.inotify.max_user_instances=1280
fs.inotify.max_user_watches=655360

To deploy the CKF bundle with Juju, run the following command:

juju deploy kubeflow --trust --channel=1.9/stable

Once the deployment is completed, you get a message such as:

Deploy of bundle completed.

This means that all the bundle components have been kickstarted into action.

You can use the following command to check the status of all the model components:

juju status

The deployment is ready when the statuses of all the applications and the units in the bundle have an active status. You can also use this option to continuously watch the status of the model:

juju status --watch 5s

During the deployment process, some of the components statuses may momentarily change to blocked or error state. This is an expected behaviour, and these statuses should resolve by themselves as the bundle configures. However, if any component remains stuck in the same error state, check Troubleshooting.

While you’re waiting for the Kubeflow bundle to prepare, you can move to the next section of this tutorial to perform some post-install configuration tasks.

Configure dashboard access

First, run this command to check the IP address of the Istio ingress gateway load balancer, which is the entry point for the entire bundle:

microk8s kubectl -n kubeflow get svc istio-ingressgateway-workload -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

To enable authentication, run the following commands to set a username and a password for your Kubeflow deployment:

juju config dex-auth static-username=admin
juju config dex-auth static-password=admin

Verify deployment

To log in as a user, open a browser and visit the URL that was retrieved in the Configure dashboard access.

You should then see the Dex login screen. Enter the credentials you previously set up.

You should now see the Kubeflow “Welcome” page:

Click on the “Start Setup” button. On the next screen you will be asked to create a namespace. This is a way of keeping all the files and settings from one project in a single, easy-to-access place. Choose any name you like:

Once you click on the “Finish” button, the dashboard is displayed:

More information on accessing the dashboard can be found in this guide.

Troubleshooting

Crash Loop Backoff

If you see CrashLoopBackOff in the juju status output, it might mean that you forgot to update inotify as follows:

sudo sysctl fs.inotify.max_user_instances=1280
sudo sysctl fs.inotify.max_user_watches=655360

After doing that, the applications should slowly turn to active.

To confirm this is the issue, check the state of the pods in the cluster by running:

microk8s kubectl get pod -n kubeflow

Pods are expected to be in a Running state. If some pods are in CrashLoopBackOff state, you can further inspect the pod by checking the logs with:

microk8s kubectl logs -n kubeflow <name-of-the-pod>

If you see error messages like this one: “error”:“too many open files” then it’s likely inotify was the issue.

Next Steps

• To learn more about CKF components, see Explore components.
• To learn about differences between upstream Kubeflow and CKF, see Upstream vs Charmed Kubelow.

---