Charmed OpenSearch

Relations

Relations, or what Juju documentation describes as Integrations, are the easiest way to connect to Charmed OpenSearch. Relations automatically create a username, password, and database for the desired user/application, as well as defining access permissions.

Data Integrator Charm

The best way to create a user and password for manual use (i.e. connecting to opensearch directly using curl, which is what we’ll be doing later) is to add a relation between Charmed Opensearch and the Data Integrator Charm. This is a bare-bones charm that allows for central management of database users, providing support for different kinds of data platform products (e.g. MongoDB, MySQL, PostgreSQL, Kafka, etc) with a consistent, opinionated and robust user experience. In order to deploy the Data Integrator Charm we can use the command juju deploy as follows:

juju deploy data-integrator --channel=edge --config index-name=test-index --config extra-user-roles=admin

The expected output:

Located charm "data-integrator" in charm-hub...
Deploying "data-integrator" from charm-hub charm "data-integrator"...

Wait for watch -c juju status --color to show:

Model     Controller       Cloud/Region         Version  SLA          Timestamp
tutorial  opensearch-demo  localhost/localhost  2.9.42   unsupported  15:38:21Z

App                        Version  Status   Scale  Charm                      Channel  Rev  Exposed  Message
data-integrator                     blocked      1  data-integrator            edge      11  no       Please relate the data-integrator with the desired product
opensearch                          active       1  opensearch                 edge      22  no
tls-certificates-operator           active       1  tls-certificates-operator  stable    22  no

Unit                          Workload  Agent  Machine  Public address  Ports  Message
data-integrator/0*            blocked   idle   2        10.180.162.96          Please relate the data-integrator with the desired product
opensearch/0*                 active    idle   0        10.180.162.97
tls-certificates-operator/0*  active    idle   1        10.180.162.44

Machine  State    Address        Inst id        Series  AZ  Message
0        started  10.180.162.97  juju-3305a8-0  jammy       Running
1        started  10.180.162.44  juju-3305a8-1  jammy       Running
2        started  10.180.162.96  juju-3305a8-2  jammy       Running

Relate to OpenSearch

Now that the Database Integrator Charm has been set up, we can relate it to Charmed OpenSearch. This will automatically create a username, password, and CA certificate for the Database Integrator Charm. Relate the two applications with:

juju relate data-integrator opensearch

Wait for watch -c juju status --color to show:

Model     Controller       Cloud/Region         Version  SLA          Timestamp
tutorial  opensearch-demo  localhost/localhost  2.9.42   unsupported  15:40:22Z

App                        Version  Status  Scale  Charm                      Channel  Rev  Exposed  Message
data-integrator                     active      1  data-integrator            edge      11  no
opensearch                          active      1  opensearch                 edge      22  no
tls-certificates-operator           active      1  tls-certificates-operator  stable    22  no

Unit                          Workload  Agent  Machine  Public address  Ports  Message
data-integrator/0*            active    idle   2        10.180.162.96
opensearch/0*                 active    idle   0        10.180.162.97
tls-certificates-operator/0*  active    idle   1        10.180.162.44

Machine  State    Address        Inst id        Series  AZ  Message
0        started  10.180.162.97  juju-3305a8-0  jammy       Running
1        started  10.180.162.44  juju-3305a8-1  jammy       Running
2        started  10.180.162.96  juju-3305a8-2  jammy       Running

Relation provider                       Requirer                               Interface                 Type     Message
data-integrator:data-integrator-peers   data-integrator:data-integrator-peers  data-integrator-peers     peer
opensearch:opensearch-client            data-integrator:opensearch             opensearch_client         regular
opensearch:opensearch-peers             opensearch:opensearch-peers            opensearch_peers          peer
opensearch:service                      opensearch:service                     rolling_op                peer
tls-certificates-operator:certificates  opensearch:certificates                tls-certificates          regular
tls-certificates-operator:replicas      tls-certificates-operator:replicas     tls-certificates-replica  peer

To retrieve information such as the username, password, and database. Enter:

juju run-action data-integrator/leader get-credentials --wait

This should output something like:

unit-data-integrator-0:
  UnitId: data-integrator/0
  id: "2"
  results:
    ok: "True"
    opensearch:
      endpoints: 10.180.162.97:9200
      index: test-index
      password: wPe02Gl7OiJPBTKh21DKvC0X9bzb9ZjQ
      tls-ca: |-
        -----BEGIN CERTIFICATE-----
        MIIC6jCCAdKgAwIBAgIULtS8XNzJj5N8...
        -----END CERTIFICATE-----
      username: opensearch-client_5
      version: 2.6.0
  status: completed
  timing:
    completed: 2023-04-05 15:41:11 +0000 UTC
    enqueued: 2023-04-05 15:41:09 +0000 UTC
    started: 2023-04-05 15:41:10 +0000 UTC

Save the CA certificate (value of tls-ca in the previous response), username, and password, because you’ll need them in the next section.

Create and Access OpenSearch Indices

Before connecting to OpenSearch, it is mandatory that you enable TLS on this cluster, following the previous step in the tutorial.

You can access the opensearch REST API any way you prefer, but in this tutorial we’re going to use curl. Get the IP of an opensearch node from the output of juju status (any of the nodes should work fine), and store the CA certificate in a local file. Run the following command, swapping the values where necessary:

curl --cacert demo-ca.pem -XGET https://username:password@opensearch_node_ip:9200/

Sending a GET request to this / endpoint should return some basic information about your opensearch deployment, which should look something like this:

{
  "name" : "opensearch-0",
  "cluster_name" : "opensearch-tutorial",
  "cluster_uuid" : "4Oe44nUzQOavu71_ifHn0w",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.6.0",
    "build_type" : "tar",
    "build_hash" : "7203a5af21a8a009aece1474446b437a3c674db6",
    "build_date" : "2023-02-24T18:57:04.388618985Z",
    "build_snapshot" : false,
    "lucene_version" : "9.5.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

If this command fails, ensure the opensearch units are all in an active-idle state.

The command we just ran used the --cacert flag to pass in the CA chain generated by the TLS operator, ensuring secure and encrypted HTTP communications between our local host and the opensearch node. To recap, the CA chain is generated by the TLS operator, and is passed over to the opensearch charm, which provides this cert in its databag to any application that relates to it using the opensearch-client relation interface. When developing charms that relate to the opensearch operator, ensure you use this cert along the user credentials (username/password pair) to authenticate communications.

To index some data, run the following command:

curl --cacert demo-ca.pem \
  -XPOST https://username:password@opensearch_node_ip:9200/albums/_doc/1?refresh=true \
  -d '{"artist": "Vulfpeck", "genre": ["Funk", "Jazz"], "title": "Thrill of the Arts"}' \
  -H 'Content-Type: application/json' -H 'Accept: application/json'

This command uses the same certificate and user credentials to send a POST request to the same node as before, but it sends a specific JSON payload to a specific document address. The output should look like the following:

{
  "_index": "albums",
  "_id": "1",
  "_version": 1,
  "result": "created",
  "forced_refresh": true,
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 0,
  "_primary_term": 1
}

Note from the response that our request was successful and the document indexed.

Use the following command to retrieve the previous document:

curl --cacert demo-ca.pem -XGET https://username:password@opensearch_node_ip:9200/albums/_doc/1

This call should output something like the following:

{
  "_index":"albums",
  "_id":"1",
  "_version":1,
  "_seq_no":0,
  "_primary_term":1,
  "found":true,
  "_source":{
    "artist": "Vulfpeck",
    "genre": ["Funk", "Jazz"],
    "title": "Thrill of the Arts"
  }
}

To add data in bulk using the OpenSearch Bulk API, copy and paste the following data into a file called bulk-albums.json, ensuring that you keep the newline at the end of the file:

{ "index" : { "_index": "albums", "_id" : "2" } }
{"artist": "Herbie Hancock", "genre": ["Jazz"],  "title": "Head Hunters"}
{ "index" : { "_index": "albums", "_id" : "3" } }
{"artist": "Lydian Collective", "genre": ["Jazz"],  "title": "Adventure"}
{ "index" : { "_index": "albums", "_id" : "4" } }
{"artist": "Rush", "genre": ["Prog"],  "title": "Moving Pictures"}

We should receive a rather long response. What is of interest to us is the portion of this response that indicates whether an error occurred or not, shown below:

{
  "took": 16,
  "errors": false,
  "items": [ ... ]
}

Then, to send this data to the bulk endpoint, run the following command:

curl --cacert demo-ca.pem -XPOST https://username:password@opensearch_node_ip:9200/_bulk --data-binary @bulk-albums.json  -H 'Content-Type: application/json'

To view the previously indexed documents, we can run a search query for the Jazz keyword in our albums index, using the following command:

curl --cacert demo-ca.pem -XGET https://username:password@opensearch_node_ip:9200/albums/_search?q=Jazz

This should return a JSON response with all the Jazz albums in the index:

{
  "took": 35,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 0.4121628,
    "hits": [{
      "_index": "albums",
      "_id": "1",
      "_score": 0.4121628,
      "_source": {
        "artist": "Vulfpeck",
        "genre": ["Funk", "Jazz"],
        "title": "Thrill of the Arts"
      }
    }, {
      "_index": "albums",
      "_id": "2",
      "_score": 0.4121628,
      "_source": {
        "artist": "Herbie Hancock",
        "genre": ["Jazz"],
        "title": "Head Hunters"
      }
    }, {
      "_index": "albums",
      "_id": "3",
      "_score": 0.4121628,
      "_source": {
        "artist": "Lydian Collective",
        "genre": ["Jazz"],
        "title": "Adventure"
      }
    }]
  }
}

Remove the user

In order to remove the user used in the previous calls, remove the relation. Removing the relation automatically removes the user that was created when the relation was created. Run the following to remove the relation:

juju remove-relation opensearch data-integrator

Now try again to connect in the same way as the previous section

curl --cacert demo-ca.pem -XGET https://username:password@opensearch_node_ip:9200/

This should output something like the following error:

Unauthorized

If you wanted to recreate this user all you would need to do is relate the two applications and run the same action on data-integrator to get the new credentials:

juju relate data-integrator opensearch
juju run-action data-integrator/leader get-credentials --wait

You can now connect to the database with this new username and password:

curl --cacert demo-ca.pem -XGET https://new_username:new_password@opensearch_node_ip:9200/albums/_search?q=Jazz

Note that the data in our index has not changed.

Also, note that the certificate does not change across relations. To create a new certificate, remove the relation between opensearch and the tls-certificates operator, wait for opensearch to enter a blocked status, then recreate the relation. Run the get-credentials action on the data-integrator charm again to get the new credentials, and test them again with the above search request.

Next Steps

The next stage in this tutorial is about managing user credentials through Juju Actions, and can be found here.