Error: Internal error occurred: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://rancher-monitoring-operator.cattle-monitoring-system.svc:443/admission-prometheusrules/mutate?timeout=10s": service "rancher-monitoring-operator" not found
This error can be caused by a number of different issues. But first we need to understand what this error means.
The TLDR is that the webhook is not available. The webhook is a service that is used to validate and mutate resources. In this case, the webhook is used to validate and mutate PrometheusRule resources.
This error can be caused by a number of different issues.
If you are using Rancher Monitoring v2, then you should be using the
rancher-monitoring-crd chart. This chart is used to install the CRDs for the monitoring stack. If this chart is in a failed state, then the webhook might not be available. To fix this, you can try to upgrade the chart. If that doesn’t work, then you can try to uninstall and reinstall the chart.
If you were using Rancher monitoring v2, then uninstall it. This should remove the CRDs and the webhook but it might not. So we need to manually remove the webhook using the following command:
kubectl delete validatingwebhookconfiguration.admissionregistration.k8s.io rancher-monitoring-admission