Charmed Spark K8s

Spark Configuration Management

Apache Spark comes with wide range of configuration properties that can be fed into Spark using a single property file, e.g. spark.properties, or by passing configuration values on the command line, as argument to spark-submit, pyspark and spark-shell.

Charmed Spark improves on this capability by enabling a set of hierarchical layers of configurations, that are merged and overridden based on a precedence rule.

Each layer may also be linked to a particular component of the Charmed Spark solution. For more information about the different components, please refer to the component overview here.

Using the different layers appropriately allow to organize and centralize configuration definition consistently for groups, single users, single environment and session. The sections below summarize the hierarchical levels of configurations. The final configuration is resolved by merging the different layers, starting from top to bottom, overriding the latter sources on top of previous ones in case of multi-level definitions.

Group configuration

Group configurations are centrally stored as secrets in K8s, and managed by spark-integration-hub-k8s charm that takes care of managing their lifecycle from creation, modification and deletion. Please refer to this how-to guide for more information on the usage of the spark-integration-hub-k8s charm for setting up group configurations. Theese are valid across users, machines and sessions.

User configuration

User configurations are centrally stored as secrets in K8s, but they are managed by the user using the spark-client snap and/or spark8t Python library. For more information, please refer to here for the spark-client snap and here for the spark8t Python library. They are valid across machines and sessions.

Environment configuration

Environment configurations are stored in your local environment, and they can apply to multiple Spark users launched/used from the same machine. They are valid across users and sessions. These configurations may be stored in:

• static properties files specified via environment variable SPARK_CLIENT_ENV_CONF
• $SNAP_DATA/etc/spark8t/spark-defaults.conf

The file specified by the environment variable takes the precedence.

Session configuration

Session configurations are provided as CLI arguments to the spark-client command, and they are only valid for the related command/session. CLI configurations may be provided by:

• Single Property specified using parameter(s) --conf <key>=<value>
• Properties Files specified using parameter(s) --properties-file

Single Property takes the precedence.