--- title: Charmhub | Deploy Apache Kafka Connect - K8s using Charmhub - The Open Operator Collection description: Deploy the latest version of Apache Kafka Connect - K8s as a Kubernetes Operator on any cloud. url: https://charmhub.io/kafka-connect-k8s/docs/h-deploy-eks --- # Apache Kafka Connect - K8s [Canonical](https://charmhub.io/publisher/data-platform "View all packages from Canonical") * [Canonical](https://charmhub.io/publisher/data-platform "View all packages from Canonical") Platform: 4/stable 18 ``` juju deploy kafka-connect-k8s --channel 4/stable ``` [Learn to deploy on juju >](https://juju.is/docs/juju/manage-applications) --- #### Relevant links * [Homepage](https://github.com/canonical/kafka-connect-k8s-operator) --- #### Contacts ##### Maintainers + [Canonical Data Platform](mailto:data-platform@lists.launchpad.net) * [Submit a bug](https://github.com/canonical/kafka-connect-k8s-operator/issues) --- Share your thoughts on this charm with the community on discourse. [Join the discussion](https://discourse.charmhub.io/) # How to deploy on EKS [Amazon Elastic Kubernetes Service](https://aws.amazon.com/eks/) (EKS) is a popular, fully automated Kubernetes service. To access the EKS Web interface, go to [console.aws.amazon.com/eks/home](https://console.aws.amazon.com/eks/home). ## [Summary](https://charmhub.io/kafka-connect-k8s/docs/h-deploy-eks#p-34318-summary) * [Install EKS and Juju tooling](https://charmhub.io/kafka-connect-k8s/docs/h-deploy-eks#install-eks-juju) * [Create a new EKS cluster](https://charmhub.io/kafka-connect-k8s/docs/h-deploy-eks#create-eks-cluster) * [Bootstrap Juju on EKS](https://charmhub.io/kafka-connect-k8s/docs/h-deploy-eks#boostrap-juju) * [Deploy charms](https://charmhub.io/kafka-connect-k8s/docs/h-deploy-eks#deploy-charms) * [Display deployment information](https://charmhub.io/kafka-connect-k8s/docs/h-deploy-eks#display-information) * [Clean up](https://charmhub.io/kafka-connect-k8s/docs/h-deploy-eks#clean-up) --- ## [Install EKS and Juju tooling](https://charmhub.io/kafka-connect-k8s/docs/h-deploy-eks#p-34318-install-eks-and-juju-tooling) Install [Juju](https://juju.is/docs/juju/install-juju) and the [`kubectl` CLI tools](https://kubernetes.io/docs/tasks/tools/) (that will be used for managing the Kubernetes cluster) via snap: ``` sudo snap install juju --channel 3.5/stable sudo snap install kubectl --classic ``` Follow the installation guides for: * [eksctl](https://eksctl.io/installation/) - the Amazon EKS CLI * [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) - the Amazon Web Services CLI To check they are all correctly installed, you can run the commands below. ``` juju version ``` Sample output: ``` 3.5.2-genericlinux-amd64 ``` ``` kubectl version --client ``` Sample output: ``` Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 ``` ``` eksctl info ``` Sample output: ``` eksctl version: 0.159.0 kubectl version: v1.28.2 ``` ``` aws --version ``` Sample output: ``` aws-cli/2.13.25 Python/3.11.5 Linux/6.2.0-33-generic exe/x86_64.ubuntu.23 prompt/off ``` ### [Authenticate](https://charmhub.io/kafka-connect-k8s/docs/h-deploy-eks#p-34318-authenticate) Create an IAM account (or use legacy access keys) and login to AWS: ``` > aws configure AWS Access Key ID [None]: SECRET_ACCESS_KEY_ID AWS Secret Access Key [None]: SECRET_ACCESS_KEY_VALUE Default region name [None]: eu-west-3 Default output format [None]: ``` Verify that the CLI tool is correctly authenticating ``` aws sts get-caller-identity ``` Sample output: ``` { "UserId": "1234567890", "Account": "1234567890", "Arn": "arn:aws:iam::1234567890:root" } ``` ## [Create a new EKS cluster](https://charmhub.io/kafka-connect-k8s/docs/h-deploy-eks#p-34318-create-a-new-eks-cluster) Export the deployment name for further use: ``` export JUJU_NAME=eks-$USER-$RANDOM ``` This following examples in this guide will use the location `eu-west-3` and K8s `v.1.27` - feel free to change this for your own deployment. Sample `cluster.yaml`: ``` apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: ${JUJU_NAME} region: eu-west-3 version: "1.27" iam: withOIDC: true addons: - name: aws-ebs-csi-driver wellKnownPolicies: ebsCSIController: true nodeGroups: - name: ng-1 minSize: 3 maxSize: 5 iam: attachPolicyARNs: - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore - arn:aws:iam::aws:policy/AmazonS3FullAccess instancesDistribution: maxPrice: 0.15 instanceTypes: ["m5.xlarge", "m5.2xlarge"] # At least two instance types should be specified onDemandBaseCapacity: 0 onDemandPercentageAboveBaseCapacity: 50 spotInstancePools: 2 ``` Bootstrap EKS cluster with the following command: ``` eksctl create cluster -f cluster.yaml ``` Sample `cluster.yaml`: ``` ... 2023-10-12 11:13:58 [ℹ] using region eu-west-3 2023-10-12 11:13:59 [ℹ] using Kubernetes version 1.27 ... 2023-10-12 11:40:00 [✔] EKS cluster "eks-taurus-27506" in "eu-west-3" region is ready ``` ## [Bootstrap Juju on EKS](https://charmhub.io/kafka-connect-k8s/docs/h-deploy-eks#p-34318-bootstrap-juju-on-eks) Add Juju K8s clouds: ``` juju add-k8s $JUJU_NAME ``` Bootstrap Juju controller: ``` juju bootstrap $JUJU_NAME ``` ## [Deploy Charms](https://charmhub.io/kafka-connect-k8s/docs/h-deploy-eks#p-34318-deploy-charms) Create a new Juju model, if needed: ``` juju add-model ``` (Optional) Increase the debug level if you are troubleshooting charms: ``` juju model-config logging-config='=INFO;unit=DEBUG' ``` Then, Charmed Apache Kafka can be deployed as usual: ``` juju deploy zookeeper-k8s -n3 --channel 3/stable --trust juju deploy kafka-k8s -n3 --channel 3/stable --trust juju integrate kafka-k8s zookeeper-k8s ``` We also recommend to deploy a [Data Integrator](https://charmhub.io/data-integrator) for creating an admin user to manage the content of the Kafka cluster: ``` juju deploy data-integrator admin --channel edge \ --config extra-user-roles=admin \ --config topic-name=admin-topic ``` And integrate it with the Kafka application: ``` juju integrate kafka-k8s admin ``` For more information on Data Integrator and how to use it, please refer to the [how-to manage applications](https://discourse.charmhub.io/t/charmed-kafka-how-to-manage-app/10285) user guide. ## [Display deployment information](https://charmhub.io/kafka-connect-k8s/docs/h-deploy-eks#p-34318-display-deployment-information) Display information about the current deployments with the following commands: ``` kubectl cluster-info ``` Sample output: ``` Kubernetes control plane is running at https://AAAAAAAAAAAAAAAAAAAAAAA.gr7.eu-west-3.eks.amazonaws.com CoreDNS is running at https://AAAAAAAAAAAAAAAAAAAAAAA.gr7.eu-west-3.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy ``` ``` eksctl get cluster -A ``` Sample output: ``` NAME REGION EKSCTL CREATED eks-marc-9587 eu-west-3 True ``` ``` kubectl get node ``` Sample output: ``` NAME STATUS ROLES AGE VERSION ip-192-168-1-168.eu-west-3.compute.internal Ready 5d22h v1.27.16-eks-a737599 ip-192-168-45-234.eu-west-3.compute.internal Ready 3h25m v1.27.16-eks-a737599 ip-192-168-85-225.eu-west-3.compute.internal Ready 5d22h v1.27.16-eks-a737599 ``` ## [Clean up](https://charmhub.io/kafka-connect-k8s/docs/h-deploy-eks#p-34318-clean-up) Always clean EKS resources that are no longer necessary - they could be costly! To clean the EKS cluster, resources and juju cloud, run the following commands: ``` juju destroy-controller $JUJU_NAME --yes --destroy-all-models --destroy-storage --force juju remove-cloud $JUJU_NAME ``` List all services and then delete those that have an associated EXTERNAL-IP value (e.g. load balancers): ``` kubectl get svc --all-namespaces kubectl delete svc ``` Next, delete the EKS cluster (As described on the [Deleting an Amazon EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/delete-cluster.html) page): ``` eksctl get cluster -A eksctl delete cluster --region eu-west-3 --force --disable-nodegroup-eviction ``` Finally, remove AWS CLI user credentials (to avoid forgetting and getting exposed to a risk of leaking credentials): ``` rm -f ~/.aws/credentials ``` ---