How to write highly changeable, highly complex software?

Question

I know questions like this has been asked before. But none of them truly answered me.

How to keep a big and complex software product maintainable over the years?
How do you organize highly customized software?
...

We're working on a software that has a lot of rules and conditions. Also, those rules change frequently.

So far we've used many designs and techniques to cope with the requirements and implement them. To be more specific, let us focus on the following examples:

• Using configurable options, so that values can be changed easily by business people, like the amount of debt a customer can undergo.
  Problem: Soon there are like hundreds of configurable options, and even naming them becomes a nightmare, let alone managing them, changing them, tracking them in code, etc.

• Using small satellite project files as additives to the main engine (something like extensible workflow), so that new rules can be added hot, and also unwanted rules can be removed from the process pipelines.
  Problem: Soon the number of project files become so huge that the file-system management of the projects becomes a pain in the ass. What if for a short period of time business team needs a special customization that only lasts for two months, and then it should be removed from the system. Something like prepaid registration to track user-behavior for making the real decisions. Or a rule as simple as banning customers with suspicious activity that should be launched really soon.

It seems that our team is going the wrong path altogether. So how a team can handle creating highly-complex, highly-changeable software when it encounters the specific problems mentioned? What points should that team consider? Are there specific architectures or designs that will help us creating such software?

Answer 1

I will focus on the two core problems described in your question:

Soon there are like hundreds of configurable options, and even naming them becomes a nightmare, let alone managing them, changing them, tracking them in code

The naming problem can be solved by organizing them in hierarchies. For example, look at the thousands of options you will find in the about:config page of the Firefox web browser - they are well-structured and organized in groups of multiple levels.

Tracking in code should follow the same principle: create a hierarchy of objects or data structures (actually, naming and code hierarchies should match, and one should be generated from the other). Of course, that alone is not enough, for hundreds of options you will need to maintain some formal documentation in a disciplined manner (again, check how the Firefox community solves this). You need to establish some quality gates like reviews, and you should to clean up things from time to time. In the end, this is a purely organizational problem, but it is in fact in no way different from managing something like a large database schema.

Using small satellite project files as additives ... Soon the number of project files become so huge that the file-system management of the projects becomes a pain in the ass

That is actually not different from the first case: hierarchies, formal documentation, quality assurance and some discipline will help you here, too. Moreover, assumed your additives are code, there should be no real difference to code which is not "hot-pluggable" - the latter makes only a difference for the user of the system, but not for the maintenance by developers. So you have to care for the same things you always have when writing bigger software: dependencies, clean architecture, SOLID code etc.

As you mentioned "a business team": another option to keep "additives" more manageable is to provide an extension mechanism like a script or query language, so your business team can write such additives on their own. That delegates the burden of managing the extensions away from the development team to the users of the system on the "business side". Though that sounds just like shifting the problem around, it can actually help you, because you create a strict boundary between your core system and those extensions, with in total more people involved for doing the management tasks that give you a headache.

And maybe the people "inventing" all these new requirements will be thinking twice what they do, if they have to write these scripts on their own.

One final thing to add: I heavily recommend to keep new options as "orthogonal as possible". In other words: each new option should have as few dependencies as possible to other existing options, ideally none. That typically implies to split up new requirements for "one option" into several smaller options, but I can tell you from experience, that really a key factor to keep such a system manageable.

Answer 2

People who work in software product lines frequently wrestle with the same problems - how to write software that can vary depending on the deployment environment. There are several methods that have been identified to manage variability - build-time configuration, plug-in architecture models, run-time configuration of properties, configuration files, extending or modifying the source code for each variant. However, like you've found, there's no perfect solution. There are just trade-offs between the options.

To answer your question - yes, there are processes and methods, tools, architectures, and built software systems that achieve what you're trying to achieve. However, without a much deeper understanding of your software system and your organization, it's not possible to provide you with guidance on what, exactly to do.

The field of software product lines is very big. It ranges from requirements development through software maintenance and includes organizational structure, project and program management, and more. The books Software Product Lines: Practices and Patterns and Software Product Lines in Action: The Best Industrial Practice in Product Line Engineering can give you a good introduction to product line engineering, which addresses variability and reuse. More focused on software reuse is Software Reuse: Architecture, Process and Organization for Business Success.