Relational or non-relational when dealing with highly relational, big data

Question

I'm working on a project (in Java) that will be processing a large event stream (GBs every minute). This data is heavily processed to find relationships between events. This is done through a normalization and correlation process. Right now, we're looking at around 10 different data types (in other words, 10 tables) that will eventually make their way into a database, as well as Kafka for downstream processing. All of these data types will relate to one another, in some fashion.

The system is based on a microservice architecture and all of the data is processed in real-time using Kafka. This means, there are a ton of writes to the database with very little to no reads (with the exception of our middleware / frontend).

I've done quite a bit of reading on NoSQL (specifically MongoDB), and I'm a bit torn. Most articles break down to choosing NoSQL over relational database to the following:

1. Dealing with large amounts of data and scalability is a concern
2. Schema changes are regular with each iteration
3. The data you are dealing with is non-relational

My use case conflicts with these three points:

1. Scaling is always a concern, considering that we will be storing several terabytes of data on a daily basis. A lot of this data is repetitive and could probably be consolidated.
2. Schema changes will be frequent during the initial iterations of the development phase but I imagine would slow down once the project reaches a GA release. After which point, schema changes will likely be minimal but very likely to happen.
3. The purpose of this software is to identify relationships in the data, where there will be several one-to-many relationships.

If I go down the route of using SQL, I could very well reduce the amount of stored data by using lookup / mapping tables and consolidating repetitive data. But at the end of the day, scaling is my #1 concern. Once we've committed to using relational database, it will be very difficult migrating away.

If I chose NoSQL, I could ease my mind on the scaling aspect. It would also allow for more frictionless development as our application evolves heavily over the beginning of the development phase.

Answer 1

Unfortunately a multitude of the articles on this subject tend to generalize away the specific key facts that actually matter when making a decision on this matter, which has led to a lot of misinformation and confusion.

First and foremost, there is nothing that makes a NoSQL database system faster when it comes to reading and writing data, no matter how much data one is considering storing. Both are equally capable of reading and writing data at (roughly) the same speeds, on the same hardware, all things equal. A terabyte of data is a terabyte of data, whether it's stored using a NoSQL database system or a RDBMS. This is one of the biggest points of confusion people stumble on when considering when to use a NoSQL system. It doesn't sound like you're specifically confused by this, but I just want to make it clear to future readers.

That being said, the three points you mentioned are pretty much the primary things to consider, so let's briefly break each one down:

1. Dealing with large amounts of data and scalability is a concern: NoSQL was initially incepted to solve the the problem of infrastructure scalability by allowing more flexibility with horizontal scaling across multiple servers back when companies were starting to reach the vertical limitations of the hardware in their servers. NoSQL achieved this by trading off some of the ACID principles a traditional relational database management system (RDBMS) would have to adhere to, such as immediate consistency for eventual consistency, which improved it's flexibility for scaling.

In modern computing, hardware and infrastructure as a whole (e.g. with cloud services as an option), and even RDBMS themselves have evolved enough such that the original scalability problem that NoSQL was invented for has now become a moot point, mostly. That's not to say NoSQL has no use cases, for example mobile development tends to be a fitting place one might find the natural horizontal scalability of NoSQL a fitting choice for their applications (because of the eventual consistency needed when distributing the data across multiple devices that need to support an offline first design). But scaling vertically, whether physically, virtually, or in the cloud, is now just as easy as horizontal scaling is, and RDBMS can even easily scale horizontally nowadays as well.

The only reason this point might matter to a developer today, is if your infrastructure team truly can't maintain the vertical scaling of a RDBMS and for some unique and interesting reason finds it much easier and quicker to scale the infrastructure horizontally.

2. Schema changes are regular with each iteration: While a NoSQL database lessens the concern for schema changes at the database layer, it doesn't remove the fact that the schema does still need to be maintained by the consumer which will be built on top of the data. So while there may be a little more work upfront maintaining schema changes in a RDBMS, it is not something foreign or should prohibit one's choice from using such a system, as typically no matter how one chooses to store their data, the idea of a schema to that data will always exist at some point of its lifecycle. And making schema changes in a RDBMS generally isn't that hard to maintain (unless you're again trying to synchronize those changes across a distributed set of clients such as mobile devices with offline first in mind, then a little extra thought is needed).

3. The data you are dealing with is non-relational: This is one of the most defining characteristics in my opinion when making the decision between different types of database systems. If your data has a strongly defined relational schema to it, then it usually makes sense to store it in a RDBMS. Doing otherwise can dumb down the architecture of your data's design and make it more difficult to query for more complicated use cases later on. If the data isn't relational, then a NoSQL system might prove to be a simpler home for your data, if you never intend to query it in more complex ways.

Based on the information you've provided, unless your infrastructure team has a significant bottleneck with scaling the hardware of your servers vertically, I'd recommend using a RDBMS since your data is strongly relational already.

For more information, please see this DBA.StackExchange answer I've written on the subject in the past that talks a little bit about the differences of the two types of database systems and touches on the history of NoSQL.