Why prefer a package manager over a library folder?

Question

When I think about the pros and cons of a static library folder and a package manager I feel like the library folder is a better approach.

Pros I see with a library folder:

1. No need for an external tool to manage packages.
2. No internet connection required to build.
3. Faster build (no package checking).
4. Simpler environment (less knowledge required).

Pros I see with a package manager:

1. Helps with complex dependency trees (and that can be managed downloading a dependency together with all its dependencies).
2. Helps checking if there is a new version available.

It seems the industry has decided to follow the package manager path for almost everything built today. So, what am I missing?

Answer 1

An important point missing from the other answers:

Using a package manager means having a configuration that indicates which library versions you are using and makes sure that config information is actually correct.

Knowing which libraries you use, and which version, is very important if you:

• need to update a library due to a critical bug / security hole;
• or just need to check whether an announced security hole affects you.

In addition, when you actually do update, the package manager (usually) makes sure any transitive dependencies are updated as required.

Whereas with a lib folder, you just have a bunch of (possibly binary, and possibly modified) files, and you'll have to guess where they came from and what version they are (or trust some README, which may or may not be correct).

---

To address your other points:

No need of external tool to manage packages.

True, but a) as a software developer you need to install loads of tools anyway, so one more does not usually matter, and b) usually there are only one or a few package managers in any given field (Maven/Gradle for Java, npm for JS/TypeScript, etc), so it's not like you need to install dozens of them.

No internet connection required to build.

All packages managers I know work off-line, once they have downloaded the required dependencies (which can happen right after downloading the project itself).

Faster build (no package checking).

Probably true, but it seems unlikely the offline package checking will take a significant amount of time (it's just comparing some version numbers). An online check may take a while, but that can be turned off if desired (if it is even on by default - Maven for example never checks for updates for release versions).

Simpler environments (less knowledge required).

True, but as explained above, a lib folder also requires knowledge. Also, as explained above, you'll probably only work with a handful differen package managers, which you'll know already.

Answer 2

Pros of lib folder disappear quickly after you move from small scale development to bigger work.

For example the "benefit" of not requiring an external tool is trumped by the work required to manually manage your dependencies, so the tool will be you (in more than one sense of the word).

You don't need an internet connection for a package manager. You can use local repositories.

Faster build might be true, but it's hardly something that should determine whether to use a package manager or not. We're not talking about magnitudes of difference after all, and this too depends on your configuration. You can easily make a slow build using a package manager, but that's basically sabotage.

Simpler environments (less knowledge required)? Again, in small scale development definitely a possibility. You might be able to hold the project in your head completely, down to each of the few libraries being used. Add a simple makefile / other build script and you've got yourself a complete package.

But it doesn't make environments simpler, it only works in simple environments. In larger scale development you'll be happy that you're using standard tooling instead of custom solutions. After all, you only have to learn it once (and when the package manager du jour is replaced by the new cool thing, you have to learn that too).

Answer 3

You're missing many of the advantages of package managers.

1. Package managers allow you to avoid checking large (several megabyte or larger) binaries into source control. Doing so is anathema to many source control tools, the pervasive git being one of them. We had a repository hit Bit Bucket's size limits a couple months ago because developers were checking in CocoaPods. Another project was already halfway there when we migrated from SVN because we had been checking in all our lib binaries (and hadn't been using NuGet). Since package managers will download packages on the fly for each developer, they eliminate the need to check these binaries in.
2. They prevent mixing incompatible files/libraries. Folders can contain mixed versions of the library's files if someone doesn't clean it out properly during an upgrade. I've seen a case where half the binaries in the folder were upgraded, resulting in very weird bugs. (It didn't even crash!) It took us literally months (not man hours, just overall time) to figure out the problem. By letting the package manager control the entire folder, you can't get mixed versions; you ensure consistency. They also make it much harder to use incompatible dependencies, by automatically updating everything together, installing different versions where needed, or even just throwing warnings or errors when attempting to use incompatible versions of libraries.
3. They establish a shared convention for managing libraries. When new developers come onto your project, team, company, etc., they're likely to know the conventions of the package manager. This means they don't have to waste time figuring out the fine details of how libraries are managed in your code base.
4. They give you a standard way of versioning and distributing your own dependencies and files that don't belong in your repository. I have even personally leveraged them for some large static data files my application required, so it works well for versioning things besides binary code.
5. Some package managers provide additional features during installation. NuGet will add dependencies and content files to your csproj file and can even add configuration elements to the config file.
6. Their package list files document the versions of your libraries in a single, centralized place. I don't have to right click a DLL and look at the version number to figure out what version of the library I'm using. In Python, the library author may not have even included the version number in the py files, so I might not even be able to tell the library version from them.
7. They discourage machine wide installation of dependencies. Package managers provide a conventional way of installing dependencies without a global installer. When your options are lib folder and global installation, many library developers will choose to offer their libraries primary as global installations rather than as downloadable binaries that you have to set up yourself. (MS history demonstrates this. It's also the case for many libraries in Linux.) This actually makes managing multiple projects more difficult, since you may have projects with conflicting versions and some developers will certainly choose the seemingly simpler global install over having their own lib dir.
8. They tend to centralize hosting and distribution. You no longer have to depend on that random library's web site. If they go out of business, a successful package manager's site still has every version ever uploaded. Developers also don't have to hunt down many web sites just to download new libraries; they have a go-to place to look first and even browse for different options. It's also easier to mirror packages organized in a standard way than to manually host copies of everything from ad-hoc websites.

You're also overstating the value of your "benefits."

1. No need of external tool to manage packages.

   "External" to what? I check the NuGet executable into my repositories. It's the only binary I feel okay checking in, since it's small and means I don't need to check any other binaries in.

   pip doesn't pose problems on this front since it's bundled with Python by default now and breaking and backwards incompatible changes are exceedingly rare. You're not going to develop Python code without having Python installed externally to your project, anyway.

   By the time they reach widespread adoption, package managers tend to be very stable. You can't get away without some kind of globally installed tooling for most projects, and a single package manager is a pretty light weight requirement. It's usually not much more cumbersome than having the language runtime installed.

2. No internet connection required to build.

   I can't connect to my database without a network connection. If the database is Amazon hosted, I need a full internet connection anyway. I need an Internet connection to push and pull changes through source control; a build server can't check out code to build without some kind of network connection either. You can't send or receive e-mail without one. You can't download libraries to put them in your lib folder without one! Permanently developing without an internet connection is virtually unheard of. In some rare cases where it's necessary, you can deal with this by downloading the packages to a location that the package manager can consume. (I know NuGet and pip are quite happy to pull from a simple folder or network drive; I suspect most others can as well.)

3. Faster build (no package checking).

   30 seconds during the automated build and 5 seconds during local dev builds are a good trade off for the benefits I outlined above. These are trivial time frames that are usually not even worth considering in comparison to the problems the benefits solve.

4. Simpler environments (less knowledge required).

   One tool for package management against nothing for library management isn't really a fair comparison, anyway. Without the tool, you have to learn whatever custom process the project is using to manage its libraries. This means you're never sure whether your existing knowledge applies to any new project you approach. You'll have to deal with whatever hodgepodge approach someone came up with, or make up your own. That might be a directory containing all the libraries, or it might be something much weirder. Maybe to avoid checking the libraries in, someone put them all out on a network drive and the only version indicator is the folder name. How is that or a global install really any better? By comparison, a package manager gives you a clean convention that will apply across most projects you encounter.

The common theme is that they provide consistency, documentation, and features not only within projects, but even across them. This simplifies everyone's life.

Answer 4

Having recently converted our product from using manually downloaded libraries to automatic package management with Nuget, I can say that using a package manager has massive benefits.

Our product is implemented across 27 C# projects, which is relatively small by today's standards. Some of our third party dependencies have dozens of assemblies.

Prior to Nuget, if I wanted to update all of our dependencies to the latest version I would have to:

1. Track down where I could get all of the updated libraries
2. Download them and unzip/install
3. Add the new versions to source control
4. Manually look through all references in our projects and update them to point to the new assemblies

With 27 projects and dozens of dependency assemblies, this process was very error prone and could take hours.

Now that we've updated to using Nuget, it's all done for me with a single command.

Answer 5

No need of external tool to manage packages

That's kind of a non-point is it? If I use a package manager, I don't need to have a lib folder. I also don't have to manage the packages myself.

No internet connection required to build

Aside from that not having an internet connection today while developing is somewhat rare (maybe with the exception of being in transit), a decent package manager shouldn't require you to have to have the latest version to build your application. It might complain, but there's no reason to not build with the version that it already installed

Faster build (no package checking)

That's a pretty marginal speedboost, but you can arguably make a point for that.

Simpler environments (less knowledge required)

Most package managers are so simple these days that it's hardly worth trying to get around them by doing these. There's even visual clients if you want to. They actually hide a lot of the croft that is going on.

Package managers also allow you to share these packages between different projects. If 5 of my projects use the same version of Boost, there's no need to duplicate this for each project. This is especially true for the complex dependency trees you speak of.

With a lib folder you manage packages only for that project, while a package manager allows you to do this for your entire development environment with a single tool.

Answer 6

It is the difference between just using libraries (lib directory), and using them, maintaining meta-information (package manager). Such meta-information concerns version numbers, (transitive) dependencies between libraries and such.

The discussions of DLL hell, library compatibility, java module system, OSGi and such should at least be sufficient convincing of some worth of having some form of dependency management.

• Library version and dependency problems can be a wast of time.

Also there is the benefit of a shared (local) repository, so several project do not need to maintain copies of imported libs. If one has a project with 20 submodules, some of those modules having 40 odd dependencies.

• More structure
• More trimming of libraries
• No ad-hoc human decisions about libraries

Answer 7

There are some cases where a lib folder might be necessary, for example when dealing with obsolete libraries (a version of it no longer maintained/available), a locally modified version of a library, ...

But for everything else, it's like the developer assuming the role of the package manager:

• The developer will have to download the libraries (internet required)
• The developer will have to check for newer releases manually
• ...

And IMHO, it's less knowledge required, because you have to learn about the usage of the external tool, but less about the libraries (i.e. dependencies).

Answer 8

There is another problem not covered by other questions: sharing deps.

Let's say, you have two packages building same library. In best case, there will not be any coflicts, but same HDD/SSD space used twice. In worst one, there will be varios conflicts, like versions.

If you use package manager, it will install library only once (per version) and already provide path to it.

P.S.: of course, you need dynamic linkage (or similar function in your language) to get this pro.

Answer 9

One major reason why shared libraries were considered an item of progress in 90's era Unix and Windows systems was how they could lower RAM usage when multiple programs using the same set of libraries were loaded. Code space only needs to be allocated ONCE per exact library and version of that library, the only per-instance memory usage remaining is for static variables.

Many operating systems implement shared libraries in a way that depends on mechanisms like the unix mmap() api - which implies that a library will not only need to be the exact same version but actually the very same FILE. This is simply impossible to take full advantage of for a program that ships its own set of libraries.

Given that memory is far cheaper, and library versions needed more diverse, than in the 90s, this argument doesn't carry as much weight today.