【help】How to implement the failover of the same application Pod between different clusters in istio non-flat network multi-control plane environment

Question

How do I implement such a requirement

1. From nanjing cluster test-source-869888dfdc-9k6bt Pod a request on http://helloworld.sample.svc.cluster.local:5000/hello
2. Dispatch to different subsets based on the weight defined by the VirtualService configuration
3. When dispatching to to-nanjing-local-subsets subsets, it communicates with the local helloworld of Nanjing cluster according to the DR Configuration rules (not mTLS). When the local helloworld of Nanjing cluster fails, Failover and access the helloworld of the Beijing cluster through the East-West gateway
4. When dispatching to to-beijing-eastwestgateway-subsets, it communicates with East-West gateways (mTLS) according to DR Configuration rules, and accesses helloworld of beijing cluster. When Beijing cluster fails, failure migration accesses helloworld of Nanjing cluster

Here's the configuration I'm currently running:

cat <<EOF | kubectl apply -f -
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: helloworld-vs
  namespace: sample
spec:
  gateways:
    - mesh
    - istio-system/cross-network-gateway
  hosts:
    - "helloworld.sample.svc.cluster.local"
  http:
    - match:
        - port: 5000
      route:
        - destination:
            host: "helloworld.sample.svc.cluster.local"
            subset: to-nanjing-local-subsets
          weight: 50
        - destination:
            host: "eastwestgateway.remote.cluster.global"
            subset: to-beijing-eastwestgateway-subsets
            port:
              number: 15443
          weight: 50
EOF
cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: helloworld-dr
  namespace: sample
spec:
  host: "helloworld.sample.svc.cluster.local"
  subsets:
    - name: to-nanjing-local-subsets
      labels:
        app: "helloworld"
        version: "v1"
        topology.istio.io/network: "nj-k8s-cluster-network-01"
      trafficPolicy:
        portLevelSettings:
          - port:
              number: 5000
            tls:
              mode: DISABLE
        loadBalancer:
          simple: ROUND_ROBIN

          localityLbSetting:
            enabled: true
            failoverPriority:
              - region
            failover:
              - from: "china-beijing"
                to: "china-jiangsu"
    - name: to-beijing-eastwestgateway-subsets
      labels:
        app-gateway: "eastwestgateway"
        region: "china-beijing"
        topology.istio.io/network: "bj-k8s-cluster-network-01"
      trafficPolicy:
        portLevelSettings:
          - port:
              number: 5000
            tls:
              mode: ISTIO_MUTUAL
              sni: helloworld.sample.svc.cluster.local
        loadBalancer:
          simple: ROUND_ROBIN

          localityLbSetting:
            enabled: true
            failoverPriority:
              - region
            failover:
                - from: "china-jiangsu"
                  to: "china-beijing"
  trafficPolicy:
    outlierDetection:
      consecutiveGatewayErrors: 3
      consecutive5xxErrors: 3
      interval: 10s
      baseEjectionTime: 3600s
      maxEjectionPercent: 100
EOF
cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: to-bj-eastwest-gateway-se
  namespace: sample
spec:
  hosts:
    - "eastwestgateway.remote.cluster.global"
  ports:
    - number: 5000
      name: https-5000
      protocol: HTTPS
  resolution: STATIC
  location: MESH_EXTERNAL
  endpoints:
    - address: 192.168.110.230
      ports:
        https-5000: 35443
      locality: "china-beijing"
      labels:
        app-gateway: "eastwestgateway"
        region: "china-beijing"
        topology.istio.io/network: "bj-k8s-cluster-network-01"
EOF

With the above configuration, the results of my test on the Nanjing cluster are as follows:

while true; do   kubectl exec "$(kubectl get pods -n sample -l app=test-source -o jsonpath='{.items[0].metadata.name}')" -n sample -c test-source -- curl -s http://helloworld.sample.svc.cluster.local:5000/hello;   sleep 1; done
Hello version: v1, instance: helloworld-v1-86f57ccb45-5v9dw
Hello version: v1, instance: helloworld-v1-86f57ccb45-5v9dw
Hello version: v1, instance: helloworld-v1-86f57ccb45-5v9dw
Hello version: v1, instance: helloworld-v1-86f57ccb45-5v9dw
Hello version: v1, instance: helloworld-v1-86f57ccb45-5v9dw
Hello version: v1, instance: helloworld-v1-86f57ccb45-5v9dw

The response of Nanjing helloworld can be returned normally

The Nanjing helloworld fault is simulated：

kubectl scale deployment helloworld-v1 -n sample --replicas=0
while true; do   kubectl exec "$(kubectl get pods -n sample -l app=test-source -o jsonpath='{.items[0].metadata.name}')" -n sample -c test-source -- curl -s http://helloworld.sample.svc.cluster.local:5000/hello;   sleep 1; done
That's all you get back
no healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstream

Given the above configuration, how do I implement my requirements? Or is failover not supported in non-flat networks with multiple control planes?

Some additional queries：

kubectl get service,ep -n sample

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/curl         ClusterIP   10.96.18.117   <none>        80/TCP     35d
service/helloworld   ClusterIP   10.108.70.77   <none>        5000/TCP   25h
NAME                   ENDPOINTS            AGE
endpoints/curl         <none>               35d
endpoints/helloworld   10.244.134.53:5000   10h

istioctl proxy-config diagnostics：

istioctl proxy-config listener test-source-869888dfdc-9k6bt -n sample --port 5000

ADDRESSES PORT MATCH                                      DESTINATION
0.0.0.0   5000 Trans: raw_buffer; App: http/1.1,h2c       Route: 5000
0.0.0.0   5000 ALL                                        PassthroughCluster
0.0.0.0   5000 SNI: eastwestgateway.remote.cluster.global Cluster: outbound|5000||eastwestgateway.remote.cluster.global

istioctl proxy-config route test-source-869888dfdc-9k6bt -n sample --name 5000

NAME     VHOST NAME                                   DOMAINS                                       MATCH     VIRTUAL SERVICE
5000     helloworld.sample.svc.cluster.local:5000     helloworld, helloworld.sample + 1 more...     /*        helloworld-vs.sample

istioctl proxy-config cluster test-source-869888dfdc-9k6bt -n sample --fqdn "outbound|5000|to-nanjing-local-subsets|helloworld.sample.svc.cluster.local"

SERVICE FQDN                            PORT     SUBSET                       DIRECTION     TYPE     DESTINATION RULE
helloworld.sample.svc.cluster.local     5000     to-nanjing-local-subsets     outbound      EDS      helloworld-dr.sample

istioctl proxy-config endpoints test-source-869888dfdc-9k6bt -n sample --cluster "outbound|5000|to-nanjing-local-subsets|helloworld.sample.svc.cluster.local"

ENDPOINT               STATUS      OUTLIER CHECK     CLUSTER
10.244.134.53:5000     HEALTHY     OK                outbound|5000|to-nanjing-local-subsets|helloworld.sample.svc.cluster.local

istioctl proxy-config cluster test-source-869888dfdc-9k6bt -n sample --fqdn "outbound|15443|to-beijing-eastwestgateway-subsets|eastwestgateway.remote.cluster.global"

SERVICE FQDN     PORT     SUBSET     DIRECTION     TYPE     DESTINATION RULE
No content output

kubectl get pods -n sample --show-labels

NAME                             READY   STATUS    RESTARTS   AGE    LABELS
helloworld-v1-86f57ccb45-ds76z   2/2     Running   0          41m    app=helloworld,pod-template-hash=86f57ccb45,security.istio.io/tlsMode=istio,service.istio.io/canonical-name=helloworld,service.istio.io/canonical-revision=v1,topology.istio.io/network=nj-k8s-cluster-network-01,topology.istio.io/subzone=qinhuai,topology.kubernetes.io/region=china-jiangsu,topology.kubernetes.io/zone=nanjing,version=v1
test-source-869888dfdc-9k6bt     2/2     Running   0          2d6h   app=test-source,pod-template-hash=869888dfdc,security.istio.io/tlsMode=istio,service.istio.io/canonical-name=test-source,service.istio.io/canonical-revision=latest,topology.istio.io/network=nj-k8s-cluster-network-01,topology.istio.io/subzone=qinhuai,topology.kubernetes.io/region=china-jiangsu,topology.kubernetes.io/zone=nanjing