I have deployed grafana in a namespace dashoard. Similarly, I have installed victoria-metrics in a different namespace victoria-metrics.
There is istio in the kubernetes cluster with sidecar enabled for both namespaces. I have also configured peerauthentication in both namespaces with mtls.mode: STRICT.
Now when I try to configure the victoria-metrics datasource in grafana, I get the following error:
Post "https://victoria-metrics-victoria-metrics-cluster-vmselect.victoria-metrics.svc.cluster.local:8481/select/multitenant/prometheus": tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match victoria-metrics-victoria-metrics-cluster-vmselect.victoria-metrics.svc.cluster.local - There was an error returned querying the Prometheus API.
However if I make the setting mtls.mode: PERMISSIVE in the victoria-metrics namespace, and change the datasource URL in grafana to HTTP (http://victoria-metrics-victoria-metrics-cluster-vmselect.victoria-metrics.svc.cluster.local:8481/select/multitenant/prometheus)
it works without any errors.
So mTLS auth isn't working or getting set.
How to properly configure datasources such as in this scenario where mTLS from the service mesh is enabled? How would we configure the certificates for mTLS to work?
