Data Platform Libs
- By Canonical Data Platform
- Databases
| Channel | Revision | Published | Runs on |
|---|---|---|---|
| latest/stable | 26 | 26 Jun 2023 | |
| latest/candidate | 26 | 26 Jun 2023 | |
| latest/edge | 46 | 21 Sep 2023 |
juju deploy data-platform-libs
You will need Juju 2.9 to be able to run this command. Learn how to upgrade to Juju 2.9.
Deploy universal operators easily with Juju, the Universal Operator Lifecycle Manager.
Platform:
charms.data_platform_libs.v0.data_interfaces
-
- Last updated 21 Sep 2023
- Revision Library version 0.18
Library to manage the relation for the data-platform products.
This library contains the Requires and Provides classes for handling the relation between an application and multiple managed application supported by the data-team: MySQL, Postgresql, MongoDB, Redis, and Kafka.
Database (MySQL, Postgresql, MongoDB, and Redis)
Requires Charm
This library is a uniform interface to a selection of common database metadata, with added custom events that add convenience to database management, and methods to consume the application related data.
Following an example of using the DatabaseCreatedEvent, in the context of the application charm code:
from charms.data_platform_libs.v0.data_interfaces import (
DatabaseCreatedEvent,
DatabaseRequires,
)
class ApplicationCharm(CharmBase):
# Application charm that connects to database charms.
def __init__(self, *args):
super().__init__(*args)
# Charm events defined in the database requires charm library.
self.database = DatabaseRequires(self, relation_name="database", database_name="database")
self.framework.observe(self.database.on.database_created, self._on_database_created)
def _on_database_created(self, event: DatabaseCreatedEvent) -> None:
# Handle the created database
# Create configuration file for app
config_file = self._render_app_config_file(
event.username,
event.password,
event.endpoints,
)
# Start application with rendered configuration
self._start_application(config_file)
# Set active status
self.unit.status = ActiveStatus("received database credentials")
As shown above, the library provides some custom events to handle specific situations, which are listed below:
- database_created: event emitted when the requested database is created.
- endpoints_changed: event emitted when the read/write endpoints of the database have changed.
- read_only_endpoints_changed: event emitted when the read-only endpoints of the database have changed. Event is not triggered if read/write endpoints changed too.
If it is needed to connect multiple database clusters to the same relation endpoint the application charm can implement the same code as if it would connect to only one database cluster (like the above code example).
To differentiate multiple clusters connected to the same relation endpoint the application charm can use the name of the remote application:
def _on_database_created(self, event: DatabaseCreatedEvent) -> None:
# Get the remote app name of the cluster that triggered this event
cluster = event.relation.app.name
It is also possible to provide an alias for each different database cluster/relation.
So, it is possible to differentiate the clusters in two ways.
The first is to use the remote application name, i.e., event.relation.app.name, as above.
The second way is to use different event handlers to handle each cluster events. The implementation would be something like the following code:
from charms.data_platform_libs.v0.data_interfaces import (
DatabaseCreatedEvent,
DatabaseRequires,
)
class ApplicationCharm(CharmBase):
# Application charm that connects to database charms.
def __init__(self, *args):
super().__init__(*args)
# Define the cluster aliases and one handler for each cluster database created event.
self.database = DatabaseRequires(
self,
relation_name="database",
database_name="database",
relations_aliases = ["cluster1", "cluster2"],
)
self.framework.observe(
self.database.on.cluster1_database_created, self._on_cluster1_database_created
)
self.framework.observe(
self.database.on.cluster2_database_created, self._on_cluster2_database_created
)
def _on_cluster1_database_created(self, event: DatabaseCreatedEvent) -> None:
# Handle the created database on the cluster named cluster1
# Create configuration file for app
config_file = self._render_app_config_file(
event.username,
event.password,
event.endpoints,
)
...
def _on_cluster2_database_created(self, event: DatabaseCreatedEvent) -> None:
# Handle the created database on the cluster named cluster2
# Create configuration file for app
config_file = self._render_app_config_file(
event.username,
event.password,
event.endpoints,
)
...
When it's needed to check whether a plugin (extension) is enabled on the PostgreSQL charm, you can use the is_postgresql_plugin_enabled method. To use that, you need to add the following dependency to your charmcraft.yaml file:
parts:
charm:
charm-binary-python-packages:
- psycopg[binary]
Provider Charm
Following an example of using the DatabaseRequestedEvent, in the context of the database charm code:
from charms.data_platform_libs.v0.data_interfaces import DatabaseProvides
class SampleCharm(CharmBase):
def __init__(self, *args):
super().__init__(*args)
# Charm events defined in the database provides charm library.
self.provided_database = DatabaseProvides(self, relation_name="database")
self.framework.observe(self.provided_database.on.database_requested,
self._on_database_requested)
# Database generic helper
self.database = DatabaseHelper()
def _on_database_requested(self, event: DatabaseRequestedEvent) -> None:
# Handle the event triggered by a new database requested in the relation
# Retrieve the database name using the charm library.
db_name = event.database
# generate a new user credential
username = self.database.generate_user()
password = self.database.generate_password()
# set the credentials for the relation
self.provided_database.set_credentials(event.relation.id, username, password)
# set other variables for the relation event.set_tls("False")
As shown above, the library provides a custom event (database_requested) to handle the situation when an application charm requests a new database to be created. It's preferred to subscribe to this event instead of relation changed event to avoid creating a new database when other information other than a database name is exchanged in the relation databag.
Kafka
This library is the interface to use and interact with the Kafka charm. This library contains custom events that add convenience to manage Kafka, and provides methods to consume the application related data.
Requirer Charm
from charms.data_platform_libs.v0.data_interfaces import (
BootstrapServerChangedEvent,
KafkaRequires,
TopicCreatedEvent,
)
class ApplicationCharm(CharmBase):
def __init__(self, *args):
super().__init__(*args)
self.kafka = KafkaRequires(self, "kafka_client", "test-topic")
self.framework.observe(
self.kafka.on.bootstrap_server_changed, self._on_kafka_bootstrap_server_changed
)
self.framework.observe(
self.kafka.on.topic_created, self._on_kafka_topic_created
)
def _on_kafka_bootstrap_server_changed(self, event: BootstrapServerChangedEvent):
# Event triggered when a bootstrap server was changed for this application
new_bootstrap_server = event.bootstrap_server
...
def _on_kafka_topic_created(self, event: TopicCreatedEvent):
# Event triggered when a topic was created for this application
username = event.username
password = event.password
tls = event.tls
tls_ca= event.tls_ca
bootstrap_server event.bootstrap_server
consumer_group_prefic = event.consumer_group_prefix
zookeeper_uris = event.zookeeper_uris
...
As shown above, the library provides some custom events to handle specific situations, which are listed below:
- topic_created: event emitted when the requested topic is created.
- bootstrap_server_changed: event emitted when the bootstrap server have changed.
- credential_changed: event emitted when the credentials of Kafka changed.
Provider Charm
Following the previous example, this is an example of the provider charm.
class SampleCharm(CharmBase):
from charms.data_platform_libs.v0.data_interfaces import (
KafkaProvides,
TopicRequestedEvent,
)
def __init__(self, *args):
super().__init__(*args)
# Default charm events.
self.framework.observe(self.on.start, self._on_start)
# Charm events defined in the Kafka Provides charm library.
self.kafka_provider = KafkaProvides(self, relation_name="kafka_client")
self.framework.observe(self.kafka_provider.on.topic_requested, self._on_topic_requested)
# Kafka generic helper
self.kafka = KafkaHelper()
def _on_topic_requested(self, event: TopicRequestedEvent):
# Handle the on_topic_requested event.
topic = event.topic
relation_id = event.relation.id
# set connection info in the databag relation
self.kafka_provider.set_bootstrap_server(relation_id, self.kafka.get_bootstrap_server())
self.kafka_provider.set_credentials(relation_id, username=username, password=password)
self.kafka_provider.set_consumer_group_prefix(relation_id, ...)
self.kafka_provider.set_tls(relation_id, "False")
self.kafka_provider.set_zookeeper_uris(relation_id, ...)
As shown above, the library provides a custom event (topic_requested) to handle the situation when an application charm requests a new topic to be created. It is preferred to subscribe to this event instead of relation changed event to avoid creating a new topic when other information other than a topic name is exchanged in the relation databag.
Index
class SecretGroup
Description
Secret groups as constants. None
class DataInterfacesError
Description
Common ancestor for DataInterfaces related exceptions. None
class SecretError
Description
Common ancestor for Secrets related exceptions. None
class SecretAlreadyExistsError
Description
A secret that was to be added already exists. None
class SecretsUnavailableError
Description
Secrets aren't yet available for Juju version used. None
class SecretsIllegalUpdateError
Description
Secrets aren't yet available for Juju version used. None
def
get_encoded_field(
relation,
member,
field
)
Description
Retrieve and decode an encoded field from relation data. None
def
set_encoded_field(
relation,
member,
field,
value
)
Description
Set an encoded field from relation data. None
def
diff(
event: RelationChangedEvent,
bucket
)
Retrieves the diff of the data in the relation changed databag.
Arguments
relation changed event.
bucket of the databag (app or unit)
Returns
a Diff instance containing the added, deleted and changed keys from the event relation databag.
def leader_only(f)
Description
Decorator to ensure that only leader can perform given operation. None
Methods
leader_only. wrapper( self )
def juju_secrets_only(f)
Description
Decorator to ensure that certain operations would be only executed on Juju3. None
Methods
juju_secrets_only. wrapper( self )
class Scope
Description
Peer relations scope. None
class CachedSecret
Locally cache a secret.
Description
The data structure is precisely re-using/simulating as in the actual Secret Storage
Methods
CachedSecret. __init__( self , charm: CharmBase , label: str , secret_uri )
CachedSecret. add_secret( self , content , relation: Relation )
Description
Create a new secret. None
CachedSecret. meta( self )
Description
Getting cached secret meta-information. None
CachedSecret. get_content( self )
Description
Getting cached secret content. None
CachedSecret. set_content( self , content )
Description
Setting cached secret content. None
CachedSecret. get_info( self )
Description
Wrapper function to apply the corresponding call on the Secret object within CachedSecret if any. None
class SecretCache
Description
A data structure storing CachedSecret objects. None
Methods
SecretCache. __init__( self , charm )
SecretCache. get( self , label: str , uri )
Description
Getting a secret from Juju Secret store or cache. None
SecretCache. add( self , label: str , content , relation: Relation )
Description
Adding a secret to Juju Secret. None
class DataRelation
Description
Base relation data mainpulation (abstract) class. None
Methods
DataRelation. __init__( self , charm: CharmBase , relation_name: str )
DataRelation. relations( self )
Description
The list of Relation instances associated with this relation_name. None
DataRelation. secrets_enabled( self )
Description
Is this Juju version allowing for Secrets usage? None
DataRelation. get_relation( self , relation_name , relation_id )
Description
Safe way of retrieving a relation. None
DataRelation. fetch_relation_data( self , relation_ids , fields , relation_name )
Retrieves data from relation.
Returns
a dict of the values stored in the relation data bag for all relation instances (indexed by the relation ID).
Description
This function can be used to retrieve data from a relation
in the charm code when outside an event callback.
Function cannot be used in *-relation-broken events and will raise an exception.
DataRelation. update_relation_data( self , relation_id: int , data: dict )
Description
Update the data within the relation. None
class DataProvides
Description
Base provides-side of the data products relation. None
Methods
DataProvides. __init__( self , charm: CharmBase , relation_name: str )
DataProvides. update_relation_data( self , relation_id: int , fields )
Description
Set values for fields not caring whether it's a secret or not. None
DataProvides. set_credentials( self , relation_id: int , username: str , password: str )
Set credentials.
Arguments
the identifier for a particular relation.
user that was created.
password of the created user.
Description
This function writes in the application data bag, therefore, only the leader unit can call it.
DataProvides. set_tls( self , relation_id: int , tls: str )
Set whether TLS is enabled.
Arguments
the identifier for a particular relation.
whether tls is enabled (True or False).
DataProvides. set_tls_ca( self , relation_id: int , tls_ca: str )
Set the TLS CA in the application relation databag.
Arguments
the identifier for a particular relation.
TLS certification authority.
class DataRequires
Description
Requires-side of the relation. None
Methods
DataRequires. __init__( self , charm , relation_name: str , extra_user_roles , additional_secret_fields )
Description
Manager of base client relations. None
DataRequires. secret_fields( self )
Description
Local access to secrets field, in case they are being used. None
DataRequires. is_resource_created( self , relation_id )
Check if the resource has been created.
Arguments
When provided the check is done only for the relation id provided, otherwise the check is done for all relations
Returns
True or False
Description
This function can be used to check if the Provider answered with data in the charm code when outside an event callback.
DataRequires. update_relation_data( self , relation_id: int , data: dict )
Updates a set of key-value pairs in the relation.
Arguments
the identifier for a particular relation.
dict containing the key-value pairs that should be updated in the relation.
Description
This function writes in the application data bag, therefore, only the leader unit can call it.
DataRequires. fetch_relation_field( self , relation_id: int , field: str , relation_name )
Description
Get a single field from the relation data. None
class ExtraRoleEvent
Description
Base class for data events. None
Methods
ExtraRoleEvent. extra_user_roles( self )
Description
Returns the extra user roles that were requested. None
class AuthenticationEvent
Base class for authentication fields for events.
Description
The amount of logic added here is not ideal -- but this was the only way to preserve the interface when moving to Juju Secrets
Methods
AuthenticationEvent. secrets_enabled( self )
Description
Is this Juju version allowing for Secrets usage? None
AuthenticationEvent. username( self )
Description
Returns the created username. None
AuthenticationEvent. password( self )
Description
Returns the password for the created user. None
AuthenticationEvent. tls( self )
Description
Returns whether TLS is configured. None
AuthenticationEvent. tls_ca( self )
Description
Returns TLS CA. None
class DatabaseProvidesEvent
Description
Base class for database events. None
Methods
DatabaseProvidesEvent. database( self )
Description
Returns the database that was requested. None
class DatabaseRequestedEvent
Description
Event emitted when a new database is requested for use on this relation. None
class DatabaseProvidesEvents
Database events.
Description
This class defines the events that the database can emit.
class DatabaseRequiresEvent
Description
Base class for database events. None
Methods
DatabaseRequiresEvent. database( self )
Description
Returns the database name. None
DatabaseRequiresEvent. endpoints( self )
Returns a comma separated list of read/write endpoints.
Description
In VM charms, this is the primary's address. In kubernetes charms, this is the service to the primary pod.
DatabaseRequiresEvent. read_only_endpoints( self )
Returns a comma separated list of read only endpoints.
Description
In VM charms, this is the address of all the secondary instances. In kubernetes charms, this is the service to all replica pod instances.
DatabaseRequiresEvent. replset( self )
Returns the replicaset name.
Description
MongoDB only.
DatabaseRequiresEvent. uris( self )
Returns the connection URIs.
Description
MongoDB, Redis, OpenSearch.
DatabaseRequiresEvent. version( self )
Returns the version of the database.
Description
Version as informed by the database daemon.
class DatabaseCreatedEvent
Description
Event emitted when a new database is created for use on this relation. None
class DatabaseEndpointsChangedEvent
Description
Event emitted when the read/write endpoints are changed. None
class DatabaseReadOnlyEndpointsChangedEvent
Description
Event emitted when the read only endpoints are changed. None
class DatabaseRequiresEvents
Database events.
Description
This class defines the events that the database can emit.
class DatabaseProvides
Description
Provider-side of the database relations. None
Methods
DatabaseProvides. __init__( self , charm: CharmBase , relation_name: str )
DatabaseProvides. set_database( self , relation_id: int , database_name: str )
Set database name.
Arguments
the identifier for a particular relation.
database name.
Description
This function writes in the application data bag, therefore, only the leader unit can call it.
DatabaseProvides. set_endpoints( self , relation_id: int , connection_strings: str )
Set database primary connections.
Arguments
the identifier for a particular relation.
database hosts and ports comma separated list.
Description
This function writes in the application data bag, therefore, only the leader unit can call it.
In VM charms, only the primary's address should be passed as an endpoint. In kubernetes charms, the service endpoint to the primary pod should be passed as an endpoint.
DatabaseProvides. set_read_only_endpoints( self , relation_id: int , connection_strings: str )
Set database replicas connection strings.
Arguments
the identifier for a particular relation.
database hosts and ports comma separated list.
Description
This function writes in the application data bag, therefore, only the leader unit can call it.
DatabaseProvides. set_replset( self , relation_id: int , replset: str )
Set replica set name in the application relation databag.
Arguments
the identifier for a particular relation.
replica set name.
Description
MongoDB only.
DatabaseProvides. set_uris( self , relation_id: int , uris: str )
Set the database connection URIs in the application relation databag.
Arguments
the identifier for a particular relation.
connection URIs.
Description
MongoDB, Redis, and OpenSearch only.
DatabaseProvides. set_version( self , relation_id: int , version: str )
Set the database version in the application relation databag.
Arguments
the identifier for a particular relation.
database version.
class DatabaseRequires
Description
Requires-side of the database relation. None
Methods
DatabaseRequires. __init__( self , charm , relation_name: str , database_name: str , extra_user_roles , relations_aliases , additional_secret_fields )
Description
Manager of database client relations. None
DatabaseRequires. is_postgresql_plugin_enabled( self , plugin: str , relation_index: int )
Returns whether a plugin is enabled in the database.
Arguments
name of the plugin to check.
optional relation index to check the database (default: 0 - first relation).
class KafkaProvidesEvent
Description
Base class for Kafka events. None
Methods
KafkaProvidesEvent. topic( self )
Description
Returns the topic that was requested. None
KafkaProvidesEvent. consumer_group_prefix( self )
Description
Returns the consumer-group-prefix that was requested. None
class TopicRequestedEvent
Description
Event emitted when a new topic is requested for use on this relation. None
class KafkaProvidesEvents
Kafka events.
Description
This class defines the events that the Kafka can emit.
class KafkaRequiresEvent
Description
Base class for Kafka events. None
Methods
KafkaRequiresEvent. topic( self )
Description
Returns the topic. None
KafkaRequiresEvent. bootstrap_server( self )
Description
Returns a comma-separated list of broker uris. None
KafkaRequiresEvent. consumer_group_prefix( self )
Description
Returns the consumer-group-prefix. None
KafkaRequiresEvent. zookeeper_uris( self )
Description
Returns a comma separated list of Zookeeper uris. None
class TopicCreatedEvent
Description
Event emitted when a new topic is created for use on this relation. None
class BootstrapServerChangedEvent
Description
Event emitted when the bootstrap server is changed. None
class KafkaRequiresEvents
Kafka events.
Description
This class defines the events that the Kafka can emit.
class KafkaProvides
Description
Provider-side of the Kafka relation. None
Methods
KafkaProvides. __init__( self , charm: CharmBase , relation_name: str )
KafkaProvides. set_topic( self , relation_id: int , topic: str )
Set topic name in the application relation databag.
Arguments
the identifier for a particular relation.
the topic name.
KafkaProvides. set_bootstrap_server( self , relation_id: int , bootstrap_server: str )
Set the bootstrap server in the application relation databag.
Arguments
the identifier for a particular relation.
the bootstrap server address.
KafkaProvides. set_consumer_group_prefix( self , relation_id: int , consumer_group_prefix: str )
Set the consumer group prefix in the application relation databag.
Arguments
the identifier for a particular relation.
the consumer group prefix string.
KafkaProvides. set_zookeeper_uris( self , relation_id: int , zookeeper_uris: str )
Set the zookeeper uris in the application relation databag.
Arguments
the identifier for a particular relation.
comma-separated list of ZooKeeper server uris.
class KafkaRequires
Description
Requires-side of the Kafka relation. None
Methods
KafkaRequires. __init__( self , charm , relation_name: str , topic: str , extra_user_roles , consumer_group_prefix , additional_secret_fields )
Description
Manager of Kafka client relations. None
KafkaRequires. topic( self )
Description
Topic to use in Kafka. None
KafkaRequires. topic( self , value )
class OpenSearchProvidesEvent
Description
Base class for OpenSearch events. None
Methods
OpenSearchProvidesEvent. index( self )
Description
Returns the index that was requested. None
class IndexRequestedEvent
Description
Event emitted when a new index is requested for use on this relation. None
class OpenSearchProvidesEvents
OpenSearch events.
Description
This class defines the events that OpenSearch can emit.
class OpenSearchRequiresEvent
Description
Base class for OpenSearch requirer events. None
class IndexCreatedEvent
Description
Event emitted when a new index is created for use on this relation. None
class OpenSearchRequiresEvents
OpenSearch events.
Description
This class defines the events that the opensearch requirer can emit.
class OpenSearchProvides
Description
Provider-side of the OpenSearch relation. None
Methods
OpenSearchProvides. __init__( self , charm: CharmBase , relation_name: str )
OpenSearchProvides. set_index( self , relation_id: int , index: str )
Set the index in the application relation databag.
Arguments
the identifier for a particular relation.
the index as it is created on the provider charm. This needn't match the requested index, and can be used to present a different index name if, for example, the requested index is invalid.
OpenSearchProvides. set_endpoints( self , relation_id: int , endpoints: str )
Set the endpoints in the application relation databag.
Arguments
the identifier for a particular relation.
the endpoint addresses for opensearch nodes.
OpenSearchProvides. set_version( self , relation_id: int , version: str )
Set the opensearch version in the application relation databag.
Arguments
the identifier for a particular relation.
database version.
class OpenSearchRequires
Description
Requires-side of the OpenSearch relation. None
Methods
OpenSearchRequires. __init__( self , charm , relation_name: str , index: str , extra_user_roles , additional_secret_fields )
Description
Manager of OpenSearch client relations. None