This guide presents an overview of the Charmed Kubeflow (CKF) charms that provide alert rules to facilitate their monitoring.
All alerts can be accessed using the Prometheus or Grafana User Interface (UI). See Prometheus alerting rules for more information.
The tables below provide the following information:
- Alert: alert name within the Prometheus dashboard.
- Description: when the alert is triggered. See Grafana alerting for more details.
- Severity: alert severity, where common values are “Warning” or “Critical”.
| Alert |
Description |
Severity |
| ArgoWorkflowWarningLoglines |
The argo-controller warning logs have increased by at least 40 lines per minute for the last four minutes. |
Warning |
| ArgoWorkflowErrorLoglines |
The argo-controller warning logs have increased by at least 10 lines per minute for the last four minutes. |
Critical |
| ArgoWorkflowsFailed |
Amount of failing Argo Workflows is increasing. |
Warning |
| ArgoWorkflowsErroring |
Amount of erroring Argo Workflows is increasing. |
Warning |
| ArgoWorkflowsPending |
Amount of pending Argo Workflows is increasing. |
Warning |
| KubeflowServiceDown |
Argo-controller service is down. |
Critical |
| KubeflowServiceIsNotStable |
Argo-controller service is not stable. |
Warning |
| Alert |
Description |
Severity |
| KubeflowServiceDown |
Dex-auth service is down. |
Critical |
| KubeflowServiceIsNotStable |
Dex-auth service is not stable. |
Warning |
| Alert |
Description |
Severity |
| KubeflowServiceDown |
Envoy service is down. |
Critical |
| KubeflowServiceIsNotStable |
Envoy service is not stable. |
Warning |
| Alert |
Description |
Severity |
| HTTP5xxRateHigh |
5xx rate too high. |
Critical |
| WorkloadLatencyP99High |
The workload request latency P99 > 160ms. |
Warning |
| IngressLatencyP99High |
The ingress latency P99 > 250ms. |
Warning |
| IngressTrafficMissing |
Ingress gateway traffic missing. |
Critical |
| IstioMetricsMissing |
Istio Metrics missing. |
Critical |
| KubeflowServiceDown |
Istio-gateway service is down. |
Critical |
| KubeflowServiceIsNotStable |
Istio-gateway service is not stable. |
Warning |
| Alert |
Description |
Severity |
| IstioPilotAvailabilityDrop |
Istio-pilot availability drops. |
Critical |
| KubeflowServiceDown |
Istio-gateway service is down. |
Critical |
| KubeflowServiceIsNotStable |
Istio-gateway service is not stable. |
Warning |
| Alert |
Description |
Severity |
| JupyterControllerRuntimeReconciliationErrorsExceedThreshold |
Total number of reconciliation errors per controller. |
Critical |
| UnfinishedWorkQueueAlert |
Increase in unfinished work in the work queue. |
Critical |
| KubeflowServiceDown |
Jupyter-controller service is down. |
Critical |
| KubeflowServiceIsNotStable |
Jupyter-controller service is not stable. |
Warning |
| FileDescriptorsExhausted |
File descriptors at 98% of maximum. |
Critical |
| FileDescriptorsSoonToBeExhausted |
File descriptors expected to reach maximum in one hour. |
Warning |
| Alert |
Description |
Severity |
| KubeflowServiceDown |
Katib-controller service is down. |
Critical |
| KubeflowServiceIsNotStable |
Katib-controller service is not stable. |
Warning |
| Alert |
Description |
Severity |
| KubeflowServiceDown |
Kfp-api service is down. |
Critical |
| KubeflowServiceIsNotStable |
Kfp-api service is not stable. |
Warning |
| Alert |
Description |
Severity |
| KubeflowServiceDown |
Knative-operator service is down. |
Critical |
| KubeflowServiceIsNotStable |
Knative-operator service is not stable. |
Warning |
| Alert |
Description |
Severity |
| KubeflowServiceDown |
Kserve-controller service is down. |
Critical |
| KubeflowServiceIsNotStable |
Kserve-controller service is not stable. |
Warning |
| Alert |
Description |
Severity |
| KfamDown |
Kubeflow-kfam service is down. |
Critical |
| ProfilesDown |
Kubeflow-profiles service is down. |
Critical |
| KubeflowServiceDown |
Kubeflow-profiles service is down. |
Critical |
| KubeflowServiceIsNotStable |
Kubeflow-profiles service is not stable. |
Warning |
| Alert |
Description |
Severity |
| KubeflowServiceDown |
Metacontroller-operator service is down. |
Critical |
| KubeflowServiceIsNotStable |
Metacontroller-operator service is not stable. |
Warning |
| Alert |
Description |
Severity |
| KubeflowServiceDown |
MinIO service is down. |
Critical |
| KubeflowServiceIsNotStable |
MinIO service is not stable. |
Warning |
| Alert |
Description |
Severity |
| MLFlowRequestDurationTooLong |
MLflow-server requests taking longer than expected. |
Critical |
| KubeflowServiceDown |
MLflow-server service is down. |
Critical |
| KubeflowServiceIsNotStable |
MLflow-server service is not stable. |
Warning |
| Alert |
Description |
Severity |
| KubeflowServiceDown |
Pvcviewer-operator service is down. |
Critical |
| KubeflowServiceIsNotStable |
Pvcviewer-operator service is not stable. |
Warning |
| Alert |
Description |
Severity |
| SeldonWorkqueueTooManyRetries |
Seldon workqueue retries increasing for kubeflow/seldon-core/0. |
Critical |
| SeldonHTTPError |
Seldon HTTP error in kubeflow/seldon-core/0. |
Critical |
| SeldonReconcileError |
Seldon reconciliation kubeflow/seldon-core/0 failed. |
Critical |
| SeldonUnfinishedWorkIncrease |
Seldon unfinished work for kubeflow/seldon-core/0 is increasing. |
Critical |
| SeldonWebhookError |
Seldon webhook failed for kubeflow/seldon-core/0. |
Critical |
| KubeflowServiceDown |
Seldon-core service is down. |
Critical |
| KubeflowServiceIsNotStable |
Seldon-core service is not stable. |
Warning |
| Alert |
Description |
Severity |
| KubeflowServiceDown |
Tensorboard-controller service is down. |
Critical |
| KubeflowServiceIsNotStable |
Tensorboard-controller service is not stable. |
Warning |
| Alert |
Description |
Severity |
| KubeflowServiceDown |
Training-operator service is down. |
Critical |
| KubeflowServiceIsNotStable |
Training-operator service is not stable. |
Warning |
