Loki
- By Canonical Observability
Channel | Revision | Published | Runs on |
---|---|---|---|
latest/stable | 151 | 01 Aug 2024 | |
latest/candidate | 160 | 01 Aug 2024 | |
latest/beta | 161 | 01 Aug 2024 | |
latest/edge | 163 | 09 Aug 2024 | |
1.0/stable | 104 | 12 Dec 2023 | |
1.0/candidate | 104 | 22 Nov 2023 | |
1.0/beta | 104 | 22 Nov 2023 | |
1.0/edge | 104 | 22 Nov 2023 |
juju deploy loki-k8s --channel candidate
Deploy Kubernetes operators easily with Juju, the Universal Operator Lifecycle Manager. Need a Kubernetes cluster? Install MicroK8s to create a full CNCF-certified Kubernetes system in under 60 seconds.
Platform:
After relating loki to other charms, you may encounter situations where log lines appear to be missing.
Checklist
- Source of log files is related to loki.
- Loki url (from grafana-agent or promtail config files) is reachable from the source container.
Check status
You can curl
the loki unit IP for status.
In the sample output below, the ingester isn’t ready yet.
❯ curl 10.1.166.94:3100/ready
Ingester not ready: waiting for 15s after being ready
❯ curl 10.1.166.94:3100/services
querier => Running
query-frontend-tripperware => Running
ring => Running
query-scheduler => Running
query-frontend => Running
ingester-querier => Running
compactor => Running
ruler => Running
ingester => Running
distributor => Running
server => Running
memberlist-kv => Running
analytics => Running
store => Running
cache-generation-loader => Running
query-scheduler-ring => Running
Confirm if Loki received anything at all
You can curl
the loki unit IP for labels and alerts.
❯ curl 10.1.166.94:3100/loki/api/v1/labels
{"status":"success"}
❯ curl 10.1.166.94:3100/loki/api/v1/labels
{"status":"success","data":["filename","job","juju_application","juju_charm","juju_model","juju_model_uuid","juju_unit"]}
❯ curl 10.1.166.94:3100/loki/api/v1/label/juju_unit/values
{"status":"success","data":["pg/0"]}
❯ curl 10.1.166.94:3100/loki/api/v1/rules
no rule groups found
Now that you know which labels exist, you can retrieve some logs:
❯ curl -sG 10.1.166.94:3100/loki/api/v1/query_range --data-urlencode 'query={juju_unit="pg/0"}' | jq '.data.result[0]'
You can query for the average logging rate. In the sample below, it is 0.1 log lines per second (6 log lines per minute).
❯ curl -sG 10.1.166.94:3100/loki/api/v1/query --data-urlencode 'query=rate({job=~".+"}[10m])' | jq '.data.result'
[
{
"metric": {
"filename": "/var/log/postgresql/patroni.log",
"job": "juju_test-bundle-iwfn_f427ffe2_pg",
"juju_application": "pg",
"juju_charm": "postgresql-k8s",
"juju_model": "test-bundle-iwfn",
"juju_model_uuid": "f427ffe2-9d96-482c-80c4-f200a20eb1bd",
"juju_unit": "pg/0"
},
"value": [
1715247333.466,
"0.1"
]
}
]
Query for particular log lines
If only a subset of logs is missing, you can confirm their existence in Loki by filtering labels and/or content. In the sample below loki is queried for log lines that contain “leader”.
❯ curl -sG 10.1.166.94:3100/loki/api/v1/query --data-urlencode 'query=({job=~".+"} |= "leader")' | jq '.data.result'
[
{
"stream": {
"filename": "/var/log/postgresql/patroni.log",
"job": "juju_test-bundle-iwfn_f427ffe2_pg",
"juju_application": "pg",
"juju_charm": "postgresql-k8s",
"juju_model": "test-bundle-iwfn",
"juju_model_uuid": "f427ffe2-9d96-482c-80c4-f200a20eb1bd",
"juju_unit": "pg/0"
},
"values": [
[
"1715258886211320804",
"2024-05-09 12:48:06 UTC [15]: INFO: no action. I am (pg-0), the leader with the lock "
],
[
"1715258876184953745",
"2024-05-09 12:47:56 UTC [15]: INFO: no action. I am (pg-0), the leader with the lock "
],
[
"1715258866412113833",
"2024-05-09 12:47:46 UTC [15]: INFO: no action. I am (pg-0), the leader with the lock "
]
]
}
]
List active loggers
To obtain a list of all sources that logged something recently,
❯ curl -sG 10.1.166.94:3100/loki/api/v1/query --data-urlencode 'query=count_over_time({filename=~".+"}[1m]) > 2' | jq '.data.result'
[
{
"metric": {
"filename": "/var/log/postgresql/patroni.log",
"job": "juju_test-bundle-iwfn_f427ffe2_pg",
"juju_application": "pg",
"juju_charm": "postgresql-k8s",
"juju_model": "test-bundle-iwfn",
"juju_model_uuid": "f427ffe2-9d96-482c-80c4-f200a20eb1bd",
"juju_unit": "pg/0"
},
"value": [
1715249068.007,
"6"
]
}
]
Logs pushed by grafana-agent or promtail
Confirm that logs are being sent out:
# grafana-agent
juju ssh grafana-agent/0 curl localhost:12345/metrics | grep "promtail_sent_"
# promtail
juju ssh mysql-router/0 curl localhost:9080/metrics | grep -E "promtail_read_|promtail_sent_"
If the values are zero (or constant for quite some time), make sure the monitored log files exist and are not empty.
References
- https://grafana.com/docs/loki/v2.9.x/query/
- https://grafana.com/docs/loki/v2.9.x/reference/api/#query-loki
- https://megamorf.gitlab.io/cheat-sheets/loki/
- https://grafana.com/docs/loki/v2.9.x/query/log_queries/
- https://helgeklein.com/blog/logql-a-primer-on-querying-loki-from-grafana/