1

I have 2 servers (One for Database Connections and one or more for client connections) running in a machine. The job of the database connections server is to fetch data from MySQL database (Aurora) and give this to the other servers on request.

Right now the data transfer (Between the 2 servers) happens via Json (json_spirit)(Don't know why I designed it this way.).

I am coming up to a stage where the data that is loaded from the MySQL DB is huge when the server starts up and every 1 minute. And 1000s of smaller queries in between.

I can see the impact Json is having since I have to Parse MYSQL_RES to Json and transmit from the DB Connection server to Client connection server and then parse the Json to a data set. I am looking to serialize my data or do something other than parsing since the overhead is slowing down the client connection server since it is waiting on the response from DB Connection Server.

What would you suggest to serialize MYSQL_RES struct? I have read about Protobuf, Flatbuffer and serialization. But simply cannot make a decision.

Prasanth Madhavan
  • 111
  • 3

2 Answers2

1

Redesign.

The clients should access the database directly. This is much lighter weight than going through a "server".

No, don't argue that you have to keep the current architecture. You will simply be making more work for yourself now and in the future.

Rick James
  • 80,479
  • 5
  • 52
  • 119
1

loaded from the MySQL DB is huge when the server starts up and every 1 minute.

If that is the main problem, then the solution is to use replication to keep the recipient server up to date without having to do a full copy.

Writes to the Master are 'replicated', as they happen, to the Replica. If the replicas become overworked, you can add more replicas, thereby spreading the load out.

Effectively, replication can provide a virtually infinite read capacity. (Write capacity is not helped.)

To discuss this, and other options, please provide more details on the flow of data, including frequency and other metrics. (For example, it is trivial to poll 10K devices evenly spread out across a day, but "impossible" to poll 10k devices at exactly midnight.)

Rick James
  • 80,479
  • 5
  • 52
  • 119