I have 1 MariaDB Galera cluster composed of 5 nodes in 2 different datacenters: 3 in DC1, 2 in DC2, On DC1, one of the nodes has a pc.weight=2. Bootstrap was set on Node 1 from DC1. Data between the 5 nodes is correctly synced and system works perfectly fine. I created a table, with column ID (PK int auto increment) and column Data (varchar) where the DC's id would be inserted, such that I know from where the data was generated. Also a 3rd column InsertDate as Timestamp. During a test case, i use the following procedure:
- run in both DCs a script that inserts 1 row every second, during 10 seconds. We endup with 20 rows inserted and synced in all 5 nodes. This is OK.
- disconnect network connectivity between the 2 DCs
- the 3 nodes on DC 1 are still available and ready for query
- the 2 nodes on DC 2 are unavailable for query. However, by executing:
SET GLOBAL wsrep_provider_options='pc.bootstrap=YES';
the 2 nodes present on DC2 start accepting queries. We have now 2 independent clusters, 1 in each DC.
- still while the network is down between DC's, on DC2 I run once again the script during 10 seconds, inserting 10 new rows on DC2. And I do the same for DC1.
- currently we have 30 rows on DC1 (20 from previous run, and 10 of the last run) and on DC2 we also have 30 rows.
- I reconnect network communications between DCs
- the 3 nodes in DC1 is still on, accepting queries, but DC2 nodes are still not available to DC1, because the did not join DC1 nodes yet.
- so I reboot nodes from DC2 and the have now joined DC1 and forme da sinclge cluster with 5 nodes where DC1 has the majority (2+1+1)
Checking data, I find that the whole cluster still has 30 rows, when there should be 40 (20 from fist run, 10 from second run from DC1 + 10 from DC2) so it seems that rows from DC2 were discarded. What I need is when nodes from DC2 join the cluster, the new rows found on DC2 should be replicated on DC1 nodes, and the same from DC1 to DC2: nodes from DC1 should replicated to DC2. I noticed that some IDs (PK autoincrement) from DC1 would be found on DC2 but the column Data would not be the correct one, as well as InsertedDate that would differ between DC1 and DC2. I suspect that the fact that during the test and while data was being inserted when both DCs were not communicating, the PKs matched and once the connectivity was UP again, Galera just discarded values that would conflict. Could that be the reason? How can I make sure that when connectivity is back, new data found on DC2 and DC1 are synced between nodes, without dicarding any? I thought about auto_increment_increment and auto_increment_offset, but that did work, Galera overrides those settings and manages the auto increment PK. Any idea?
