5

I need to process 2 million messages per second (perhaps in a scale out configuration) and route each message to a N delegates or multicast delegates.

Question

How should I structure my application in C# so that I can achieve the best performance when receiving the message, and routing it to the correct delegate?

Additional Details

Each inbound message has the following properties in the format of a JSON array:

  • Customer
  • Category
  • TopicID
  • DateTime
  • Data[]

Each message will be processed by a delagate function that is interested one or more of those properties.

Simple examples of delegate processing needed:

  • A customer counter function may count the quantity of messages per customer,
  • A category counter may count the quantity of messages that counter
  • A customer-counter will count messages unique to that customer.
Community
  • 1
makerofthings7
  • 6,080
  • 4
  • 42
  • 78

1 Answers1

5

200Gb of data per second?

To do this you're not going to get too much help with C#, all the High Performance Computing applications are written in C/C++, so for a start, you should be looking to go this direction simply to leverage their expertise, libraries and frameworks.

That said, a lot of telecoms work is done using functional languages, and they deliver input streams of a magnitude similar to what you're expecting, so try Haskel too.

In any case, your first task is to split that input stream to several servers, then pass them to more servers for processing. In these cases, you don't need delegates, you need a full message passing architecture that is stateless and memory efficient. Try OpenMPI for some hints.

gbjbaanb
  • 48,749
  • 7
  • 106
  • 173