How to prevent in house frameworks and libraries from becoming technical debt

Question

My company develops many relatively small projects that a lot of times do the same things and have a similar structure. (e.g. read/write to a databse, data pre processing, building a query etc.). This creates a natural urge to create libraries and frameworks that either implement some of the repetitive logic and/or force a certain uniform program structure.

These libraries are often developed by developers without a large enough sample size of projects to be generic and are later not maintained properly because the team that developed them has different priorities than to improve their framework, especially for use cases of other teams.

Working with these libraries has caused me to automatically be repulsed by the idea of in house libraries and frameworks, and I rather copy the code from project to project, which will make it a hell lot easier to change than having to deal with releasing a new library version, even though I hate inconsistency and code duplication.

I would like advice on how to properly deal with this situation.

Answer 1

There is no general solution to this problem. Welcome to approximately every organization that develops software. In many ways this is precisely the problem that the open source community has, as well.

Some person writes something useful. A few other people use it, too. Then a few more. Soon bugs are discovered and enhancements are requested, but the original maintainer is off doing something else. Copy and paste was your only recourse, until GitHub came along.

IMSoP definitely alludes to something important in their comment on your question. The obvious problem is managing dependencies, but this is only part of the solution. The organization needs a large amount of automation as well, otherwise distributing and testing these libraries becomes prohibitively labor-intensive.

At a high level, these libraries need:

• Excellent automated test coverage.
• An automated build system.
• A system to run automated tests.
• A system that empowers collaboration, so the original maintainers can retain veto power over changes, but are not solely responsible for implementing those changes (think: code reviews or pull requests).
• A centralized package management system.
• The ability to push new versions of packages to the package repository.
• To use Semantic Versioning, which allows you to communicate bug fixes, versus feature enhancements, versus breaking changes using version numbers.
• A hosting solution for developer documentation that integrates with existing continuous integration/continuous delivery pipelines (CI/CD), so documentation changes can be deployed automatically.
• A good distributed version control system that allows people to fork code repositories, and collaborate on writing code.
  • This allows for customizations that the original maintainers reject, but your project finds useful.
• Some tool that allows the organization to identify and change the maintainers of these libraries.

And all of this will likely need to exist in a walled garden, of sorts, that is only available for authorized developers with your organization. Perhaps even within your organization's internal network.

You basically need to replicate the open source community inside your organization (along with similar collaboration tools) — an internal developer ecosystem. Implementing this is no small feat.

If your organization already uses a tool like GitHub, GitLab, Jira, Azure DevOps, etc, look at the features and components it supports. You might already have all the tools necessary to implement this, and your organization only needs a champion for the cause.

After this is all put together, maintainers spend time reviewing bug fixes and feature enhancements instead of writing them. You spend time fixing bugs and adding enhancements to the libraries as part of your regularly scheduled work (you just work in multiple code repositories), so management might not even care that you are spending time doing that. The CI pipeline performs all necessary testing. The CD pipeline packages things up and pushes them to the central package repository (including documentation changes). And when maintainers do not want to incorporate your change, you can fork their repository and implement the change yourself, and create your own package. Breaking changes can be introduced more safely when using a proper package manager and Semantic Versioning.

Answer 2

Easy: make someone owner of that library and assign serious time to let him/her take care of it.

If you do scrum however there may be a problem... This is a good example of where scrum fails miserably. No product owner will put an issue that deals with this sort of maintenance in the sprint. It takes a strong team to incorporate this kind of work into their routine (it is basically the team's responsibility, they need to take this up with any issue that depends on it and estimate accordingly).

Answer 3

Though I agree with @GregBurghardt's answer in general that for library development in larger organizations one has to scale the organizational infrastructure with all the measures he mentioned, to my experience, there is a middle ground for smaller teams, which does not require to "replicate the open source community inside ones organization", at least not in full . In case

• the group of people accessing a common library is small (5 to 6 persons at most)

• you have shared code ownership for the team (so anyone can fix bugs on the libs)

• you have tools for reverse-lookup where in the whole code base a certain library and/or library function is used (so anyone who starts to change something in a lib can make an impact analysis beforehand)

then a certain amount of library/framework development is possible and manageable without causing too much hassle. Of course, you have to bring up some discipline to follow a few strategic and tactical rules:

• "Small is beautiful"! Don't put things into common libraries "just in case" - only put code there which was used at least two or three times somewhere.

• Avoid developing large "frameworks", which too easily get interlocked with the applications depending on them. They often suffer from exactly the issues you mentioned in your queestion.

• Ideally, put only stuff into a lib where the API can be kept highly backwards compatible, and avoid incompatible changes to function signatures or function semantics

• If there is code which brings in 3rd party dependencies, make sure will be kept in separate libraries, so noone will be forced to introduce a dependency they don't need.

• More general: don't be tempted to throw too much stuff into a "commonlibrary" dumpster. Keep your libs clearly separated by topics and responsibilities.

Some of the points on Greg's list most teams should already have in place today, regardless whether they develop common libs or not, so it is not necessarily extra effort. The most obvious thing is state-of-the art version control. Moreveover, automated builds and a CI system are pretty standard today, to make sure noone accidentally breaks the build (and in case this happens, each team member should be able to fix that, regardless if the bug is in a lib or elsewhere). Automated tests are also helpful, but usually it is sufficient to use the tests your team has already written for the software based on the common libs, so this also might not cause too much extra effort.

However, the larger the number of people working with these inhouse libs gets, the more management and communication effort will be needed. At a point where you split one team into two, to allow them more individual development, you have to decide whether each of the teams will have their own "common library", or if the extended communication and maintenance effort is still balanced by the time and effort saved through reusage.

Answer 4

If the library in question is not yet developed for your use case, it is causing you more work than expanding the library would cost, and developers have "other priorities" which prevent them from expanding the library, then this is a priority problem.

You already know the solution, the company just doesn't allot you the time needed to fix it.

To be clear here, if adapting the library takes more time and only applies to your use case and there are no expected future reusability gains, then you should not be foisting it into the library.

Talk to your manager and explain the amount of delays encountered by using the library as is. Either provide an estimate on how long it would take to expand the library, or get your manager to ask the team that develops the library for one (provide a detailed analysis of the change that needs to be made).

If you are correct and management listens to reason, you'll get your change. If you're wrong, then this is not an issue that should be addressed. If they won't listen to reason, then the company chooses to perpetuate the issue and you can't change that.

Answer 5

The worst kind of technical debt is having multiple copies of a library, each slightly different in incompatible ways.

So your organisation should make that absolutely forbidden. If anyone needs added behaviour, or changed behaviour, the original creator or maintainer of the library should make the changes, either in a way that is backward compatible or configurable at build time or at run time. (Not at build time obviously if you distribute binaries, unless you can distribute different binaries).