I deployed Redis replication cluster on 2 data centers (ping time between 2 DCs is around 1ms).
I used Master (Node1 on DC 1) - Slave (Node2 in DC2) replication model and Sentinel to control promoting or demoting. I faced some scenarios with issues of network and VMs (all Redis nodes using virtual machine)
- Node 1 (Master) has network interface issue, all nodes cannot reach Node1, Sentinel promote Node2 to Master. After 1-2 minutes, Node1 back normally, what happens in this case? (Node1 Master and Node 2 Master also)
- Node 1 (Master) down, Sentinel promote Node2 to Master. After 2 hours, Node1 back, when Node1 synchronizes data from Node2, suddenly Node2 down. What happens with Node1? (assume node2 cannot back totally)
- Node 1 (Master) and Node 2 (Standby) in turn down after 5 minutes. Ex: at 10:00 am, Node1 down, at 10:05 am Node2 down, node1 back. At 10:10am, Node1 down, node2 back and so on... What happens?
Case 1 and 2 are reality. Case 3 is my assumption when node1 and node2 unstable often. My solution for 3 cases is using script manual promote or demote and does not use Sentinel because I don't believe it can process complex cases.
Could you give me advice on my cases?
