Charmed Spark K8s

Charmed Spark Components Overview

The Charmed Spark Solution bundles the following components:

• spark8t, which is a Python package to enhance Spark capabilities allowing to manage Spark jobs and service accounts, with hierarchical level of configuration
• Charmed Spark Rock OCI-compliant Image, that bundles Spark binaries together with Canonical tooling to be used to start your Spark workload on Kubernetes, to use Charmed Spark CLI tooling or derive your own images from secured and supported bases;
• Spark Client Snap, to simplify Apache Spark installation on edge nodes or local machines, by leveraging on confined SNAPs and exposing simple Snap commands to run and manage Spark Jobs
• Charmed Bundle to deploy, manage and operate Charmed Spark using Juju. This includes:
  • Spark History Server to expose a web UI for analysing the logs of previous Spark Jobs
  • Kyuubi to provide a JDBC/ODBC endpoint for running Hive powered by Spark engines
  • Spark Integration Hub to enable easy configuration of Spark service accounts, providing a native Juju integration with S3 Integrator and Azure Storage Integrator for enabling object-storage persistence and with the Canonical Observability Stack (COS) for enabling resource usage monitoring and alerting.

The following image shows how the different artifacts interacts with each other:

flowchart TD
    spark8t["`**spark8t** 
    (*python package*)
    exposes functionalities to create, configure and manage Spark users via a Python SDK`"]
    
    spark-rock["`**Charmed Spark Rock** 
    (*OCI Image*)
    provides a reliable Spark image to run Spark applications and Spark CLI tooling`"]

    spark-client["`**Spark Client Snap** 
    (*SNAP*)
    simplify client integration with a Spark Kubernetes cluster via a snap package to be installed in edge nodes or locally`"]

    spark-k8s-bundle["`**Charmed Spark** 
    (*Charmed Operator*)
    manages the entire lifecycle of Spark jobs`"]

    spark8t --> spark-rock
    spark8t --> spark-client
    spark-rock --> spark-client
    spark-rock --> spark-k8s-bundle

The Charmed Spark solution can be used to deploy and manage Apache Spark workloads using the provided distribution on any conformant Kubernetes (for versions 1.26 and above), like

• MicroK8s, which is the simplest production-grade conformant K8s. Lightweight and focused. Single command install on Linux, Windows and macOS. Refer to here for more information.
• Charmed Kubernetes, which is a platform independent, model-driven distribution of Kubernetes powered by juju
• AWK EKS, which is the managed Kubernetes service provided by Amazon Web Services to run Kubernetes in the AWS cloud and on-premises data centers.

Setup instructions are available in the Spark Tutorial Set up the environment chapter