Airbyte K8s Operator

This is the Kubernetes Python Operator for Airbyte.

Description

Airbyte is an open-source data integration platform designed to centralize and streamline the process of extracting and loading data from various sources into data warehouses, lakes, or other destinations.

This operator provides an Airbyte server, and consists of Python scripts which wraps the versions distributed by Airbyte.

Usage

Note: This operator requires the use of juju>=3.1.

The Airbyte charm relies on a number of other charms for core functionality.

Below is a set of requirements and the charms that fulfill them:

• Database: Postgresql-k8s
• Workflow Engine: Temporal-k8s
• Object Storage:
  • Minio
  • S3 integrator

Note: For object storage, the use of one of the Minio or S3 integrator charm is sufficient.

Deploying PostgreSQL Database

Airbyte uses PostgreSQL for storing metadata. The Airbyte and PostgreSQL operators can be deployed and connected to each other using the Juju command line as follows:

juju deploy airbyte-k8s --channel edge --trust
juju deploy postgresql-k8s --channel 14/edge --trust
juju relate airbyte-k8s postgresql-k8s

Deploying Minio

Airbyte uses Minio for storing state and relevant logs. The Airbyte and Minio operators can be deployed and connected to each other using the Juju command line as follows:

juju deploy minio --channel edge
juju relate airbyte-k8s minio

Deploying Temporal

Airbyte uses Temporal as a workflow engine for durable execution of sync jobs. The Temporal operators can be deployed and connected to each other using the Juju command line as follows:

juju deploy temporal-k8s
juju deploy temporal-admin-k8s
juju relate temporal-k8s:db postgresql-k8s:database
juju relate temporal-k8s:visibility postgresql-k8s:database
juju relate temporal-k8s:admin temporal-admin-k8s:admin

Once the units have settled, the following command can be run to create the default namespace:

juju run temporal-admin-k8s/0 tctl args="--ns default namespace register -rd 3"

The Airbyte charm is configured by default to connect to the Temporal charm at temporal-k8s:7233, so no further action is needed here.

Deploying Airbyte UI

To configure connectors through the web UI, the Airbyte operator requires integration with the Airbyte UI operator.

Once the Airbyte UI operator is deployed, it can be connected to the Airbyte operator using the Juju command line as follows:

juju deploy airbyte-ui-k8s
juju relate airbyte-k8s airbyte-ui-k8s

You should now be able to access the web UI at <airbyte_ui_unit_ip>:8080.

Verifying

To verify that the setup is running correctly, run juju status --relations --watch 2s and ensure that all pods are active and all required integrations exist.

Contributing

This charm is still in active development. Please see the Juju SDK docs for guidelines on enhancements to this charm following best practice guidelines, and CONTRIBUTING.md for developer guidance.