Data Platform Libs

Channel Revision Published Runs on
latest/stable 54 01 Nov 2023
Ubuntu 22.04 Ubuntu 20.04
latest/stable 2 21 Jul 2022
Ubuntu 22.04 Ubuntu 20.04
latest/candidate 54 01 Nov 2023
Ubuntu 22.04
latest/edge 76 13 Jun 2024
Ubuntu 22.04 Ubuntu 20.04
latest/edge 9 21 Oct 2022
Ubuntu 22.04 Ubuntu 20.04
juju deploy data-platform-libs
Show information

Platform:

Ubuntu
22.04 20.04

charms.data_platform_libs.v0.data_models

Library to provide simple API for promoting typed, validated and structured dataclass in charms.

Dict-like data structure are often used in charms. They are used for config, action parameters and databag. This library aims at providing simple API for using pydantic BaseModel-derived class in charms, in order to enhance:

  • Validation, by embedding custom business logic to validate single parameters or even have validators that acts across different fields
  • Parsing, by loading data into pydantic object we can both allow for other types (e.g. float) to be used in configuration/parameters as well as specify even nested complex objects for databags
  • Static typing checks, by moving from dict-like object to classes with typed-annotated properties, that can be statically checked using mypy to ensure that the code is correct.

Pydantic models can be used on:

  • Charm Configuration (as defined in config.yaml)
  • Actions parameters (as defined in actions.yaml)
  • Application/Unit Databag Information (thus making it more structured and encoded)
Creating models

Any data-structure can be modeled using dataclasses instead of dict-like objects (e.g. storing config, action parameters and databags). Within pydantic, we can define dataclasses that provides also parsing and validation on standard dataclass implementation:


from charms.data_platform_libs.v0.data_models import BaseConfigModel

class MyConfig(BaseConfigModel):

    my_key: int

    @validator("my_key")
     def is_lower_than_100(cls, v: int):
         if v > 100:
             raise ValueError("Too high")

This should allow to collapse both parsing and validation as the dataclass object is parsed and created:

dataclass = MyConfig(my_key="1")

dataclass.my_key # this returns 1 (int)
dataclass["my_key"] # this returns 1 (int)

dataclass = MyConfig(my_key="102") # this returns a ValueError("Too High")
Charm Configuration Model

Using the class above, we can implement parsing and validation of configuration by simply extending our charms using the TypedCharmBase class, as shown below.

class MyCharm(TypedCharmBase[MyConfig]):
    config_type = MyConfig

     # everywhere in the code you will have config property already parsed and validate
     def my_method(self):
         self.config: MyConfig
Action parameters

In order to parse action parameters, we can use a decorator to be applied to action event callbacks, as shown below.

@validate_params(PullActionModel)
def _pull_site_action(
    self, event: ActionEvent,
    params: Optional[Union[PullActionModel, ValidationError]] = None
):
    if isinstance(params, ValidationError):
        # handle errors
    else:
        # do stuff

Note that this changes the signature of the callbacks by adding an extra parameter with the parsed counterpart of the event.params dict-like field. If validation fails, we return (not throw!) the exception, to be handled (or raised) in the callback.

Databag

In order to parse databag fields, we define a decorator to be applied to base relation event callbacks.

@parse_relation_data(app_model=AppDataModel, unit_model=UnitDataModel)
def _on_cluster_relation_joined(
        self, event: RelationEvent,
        app_data: Optional[Union[AppDataModel, ValidationError]] = None,
        unit_data: Optional[Union[UnitDataModel, ValidationError]] = None
) -> None:
    ...

The parameters app_data and unit_data refers to the databag of the entity which fired the RelationEvent.

When we want to access to a relation databag outsides of an action, it can be useful also to compact multiple databags into a single object (if there are no conflicting fields), e.g.


class ProviderDataBag(BaseClass):
    provider_key: str

class RequirerDataBag(BaseClass):
    requirer_key: str

class MergedDataBag(ProviderDataBag, RequirerDataBag):
    pass

merged_data = get_relation_data_as(
    MergedDataBag, relation.data[self.app], relation.data[relation.app]
)

merged_data.requirer_key
merged_data.provider_key

The above code can be generalized to other kinds of merged objects, e.g. application and unit, and it can be extended to multiple sources beyond 2:

merged_data = get_relation_data_as(
    MergedDataBag, relation.data[self.app], relation.data[relation.app], ...
)

class BaseConfigModel

Description

Class to be used for defining the structured configuration options. None

Methods

BaseConfigModel. __getitem__( self , x )

Description

Return the item using the notation instance[key]. None

class TypedCharmBase

Description

Class to be used for extending config-typed charms. None

Methods

TypedCharmBase. config( self )

Description

Return a config instance validated and parsed using the provided pydantic class. None

def validate_params(cls)

Return a decorator to allow pydantic parsing of action parameters.

Arguments

cls

Pydantic class representing the model to be used for parsing the content of the action parameter

Methods

validate_params. decorator( f )

def write(
    relation_data: RelationDataContent,
    model: BaseModel
)

Write the data contained in a domain object to the relation databag.

Arguments

relation_data

pointer to the relation databag

model

instance of pydantic model to be written

def read(
    relation_data,
    obj
)

Read data from a relation databag and parse it into a domain object.

Arguments

relation_data

pointer to the relation databag

obj

pydantic class representing the model to be used for parsing

def parse_relation_data(
    app_model,
    unit_model
)

Return a decorator to allow pydantic parsing of the app and unit databags.

Arguments

app_model

Pydantic class representing the model to be used for parsing the content of the app databag. None if no parsing ought to be done.

unit_model

Pydantic class representing the model to be used for parsing the content of the unit databag. None if no parsing ought to be done.

Methods

parse_relation_data. decorator( f )

class RelationDataModel

Description

Base class to be used for creating data models to be used for relation databags. None

Methods

RelationDataModel. write( self , relation_data: RelationDataContent )

Write data to a relation databag.

Arguments

relation_data

pointer to the relation databag

RelationDataModel. read( cls , relation_data: RelationDataContent )

Read data from a relation databag and parse it as an instance of the pydantic class.

Arguments

relation_data

pointer to the relation databag

def get_relation_data_as(model_type)

Return a merged representation of the provider and requirer databag into a single object.

Arguments

model_type

pydantic class representing the merged databag

relation_data

list of RelationDataContent of provider/requirer/unit sides