Apache Kafka - K8s
- Canonical
- Databases
Channel | Revision | Published | Runs on |
---|---|---|---|
latest/stable | 5 | 09 Mar 2022 | |
latest/edge | 27 | 25 Apr 2023 | |
latest/edge | 13 | 21 Oct 2022 | |
3/stable | 56 | 27 Feb 2024 | |
3/candidate | 56 | 27 Feb 2024 | |
3/beta | 56 | 27 Feb 2024 | |
3/edge | 76 | 11 Dec 2024 |
juju deploy kafka-k8s --channel 3/stable
Deploy Kubernetes operators easily with Juju, the Universal Operator Lifecycle Manager. Need a Kubernetes cluster? Install MicroK8s to create a full CNCF-certified Kubernetes system in under 60 seconds.
Platform:
How to deploy on AKS
Azure Kubernetes Service (AKS) allows you to quickly deploy a production ready Kubernetes cluster in Azure. To access the AKS Web interface, go to https://portal.azure.com/.
Summary
- Install Azure and Juju tooling
- Create AKS Cluster
- Bootstrap Juju controller on AKS
- Deploy charms
- Display deployment information
- Clean up
Install Client Environment
Client environment includes:
- Juju
- Azure CLI
Juju
Install Juju via snap:
sudo snap install juju --channel 3.5/stable
Check that the Juju version is correctly installed:
juju version
Sample output:
3.5.2-genericlinux-amd64
Azure CLI
Follow the user guide for installing the Azure CLI on Linux distributions.
Verify that it is correctly installed running the command below:
az --version
Sample output:
azure-cli 2.65.0
core 2.65.0
telemetry 1.1.0
Dependencies:
msal 1.31.0
azure-mgmt-resource 23.1.1
Python location '/opt/az/bin/python3'
Extensions directory '/home/deusebio/.azure/cliextensions'
Python (Linux) 3.11.8 (main, Sep 25 2024, 11:33:44) [GCC 11.4.0]
Legal docs and information: aka.ms/AzureCliLegal
Your CLI is up-to-date.
Create AKS cluster
Login to your Azure account:
az login
Create a new Azure Resource Group:
az group create --name <RESOURCE_GROUP> --location <LOCATION>
The placeholder <RESOURCE_GROUP>
can be a label of your choice, and it will be used to tag
the resources created on Azure. Also the following guide will use the single server AKS,
using LOCATION=eastus
- but feel free to change this for your own deployment.
Bootstrap AKS with the following command (increase nodes count/size if necessary):
az aks create -g <RESOURCE_GROUP> -n $<K8S_CLUSTER_NAME> --enable-managed-identity --node-count 1 --node-vm-size=<INSTANCE_TYPE> --generate-ssh-keys
We recommend selecting an instance type that provides at the very least 16
GB of RAM and 4
cores, e.g. Standard_A4_v4
.
You can find more information about the available instance types in the Azure documentation.
Sample output:
{
"aadProfile": null,
"addonProfiles": null,
"agentPoolProfiles": [
{
"availabilityZones": null,
"capacityReservationGroupId": null,
"count": 1,
"creationData": null,
"currentOrchestratorVersion": "1.28.9",
"enableAutoScaling": false,
"enableEncryptionAtHost": false,
"enableFips": false,
"enableNodePublicIp": false,
...
Dump newly bootstrapped AKS credentials:
az aks get-credentials --resource-group <RESOURCE_GROUP> --name <K8S_CLUSTER_NAME> --context aks
Sample output:
...
Merged "aks" as current context in ~/.kube/config
You can verify that the cluster and your client kubectl
CLI is correctly configured by running a simple command, such as:
kubectl get pod -A
which should provide the list of the pod services running.
Bootstrap Juju controller on AKS
Bootstrap Juju controller:
juju bootstrap aks <CONTROLLER_NAME>
Sample output:
Creating Juju controller "aks" on aks/eastus
Bootstrap to Kubernetes cluster identified as azure/eastus
Creating k8s resources for controller "controller-aks"
Downloading images
Starting controller pod
Bootstrap agent now started
Contacting Juju controller at 20.231.233.33 to verify accessibility...
Bootstrap complete, controller "aks" is now available in namespace "controller-aks"
Now you can run
juju add-model <model-name>
to create a new model to deploy k8s workloads.
Deploy Charms
Create a new Juju model, if needed:
juju add-model <MODEL_NAME>
(Optional) Increase the debug level if you are troubleshooting charms:
juju model-config logging-config='<root>=INFO;unit=DEBUG'
Then, Charmed Apache Kafka can be deployed as usual:
juju deploy zookeeper-k8s -n3 --channel 3/stable
juju deploy kafka-k8s -n3 --channel 3/stable
juju integrate kafka-k8s zookeeper-k8s
We also recommend to deploy a Data Integrator for creating an admin user to manage the content of the Kafka cluster:
juju deploy data-integrator admin --channel edge \
--config extra-user-roles=admin \
--config topic-name=admin-topic
And integrate it with the Kafka application:
juju integrate kafka-k8s admin
For more information on Data Integrator and how to use it, please refer to the how-to manage applications user guide.
Display deployment information
Display information about the current deployments with the following commands:
kubectl cluster-info
Sample output:
Kubernetes control plane is running at https://aks-user-aks-aaaaa-bbbbb.hcp.eastus.azmk8s.io:443
CoreDNS is running at https://aks-user-aks-aaaaa-bbbbb.hcp.eastus.azmk8s.io:443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://aks-user-aks-aaaaa-bbbbb.hcp.eastus.azmk8s.io:443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
az aks list
Sample output:
...
"count": 1,
"currentOrchestratorVersion": "1.28.9",
"enableAutoScaling": false,
...
kubectl get node
Sample output:
NAME STATUS ROLES AGE VERSION
aks-nodepool1-31246187-vmss000000 Ready agent 11m v1.28.9
Clean up
Always clean AKS resources that are no longer necessary - they could be costly!
To clean the AKS cluster, resources and juju cloud, run the following commands:
juju destroy-controller <CONTROLLER_NAME> --destroy-all-models --destroy-storage --force
List all services and then delete those that have an associated EXTERNAL-IP value (load balancers, …):
kubectl get svc --all-namespaces
kubectl delete svc <SERVICE_NAME>
Next, delete the AKS resources (source: Deleting an all Azure VMs)
az aks delete -g <RESOURCE_GROUP> -n <K8S_CLUSTER_NAME>
Finally, logout from AKS to clean the local credentials (to avoid forgetting and getting exposed to a risk of leaking credentials):
az logout