33

I am developing code mainly using Bash, C, Python and Fortran and recently also HTML/CSS+JavaScript. My OS is Ubuntu.

Maybe I am exaggerating, but I figured that I kind of spend more time getting software (Debian and Python packages mainly, sometimes also from source) to be installed properly than actually developing code. And I am not talking about coding vs. debugging, debugging is part of coding for me.

It happens so often to me that I update my Linux packages and then my Python packages and my software does not work anymore, because some .so files have another name now, and Python does not find them anymore. Or I setup a totally clean Ubuntu VM, install a package with pip and get two screens of error message, because some debian package was not installed. I am not a system administrator, I enjoy developing software. But this just annoys me. I do not want to inform myself on all the 157 Python packages and thousands of Debian packages I have on my system and know what their dependancies are. I want to write code and implement new functionality into my code.

What am I doing wrong?

Max H. Balsmeier
  • 457

4 Answers4

46

What am I doing wrong?

You're trying to develop in an environment where you're also the sysadmin, devops and the local technical product owner for every pip package you use - and you're assuming that the sysadmin, devops and TPO roles should be no effort just because they're not what you're interested in.

Those are paid full-time jobs (ok, maybe not TPO) that people have because they are not trivial.

Maintaining up-to-date development environments can be a lot of work. The usual approaches are

  1. to work for a large enough organization that it's someone else's job, or
  2. to somehow automate it (which is why things like conda and docker exist - although this is still a non-trivial amount of work you'd prefer the person from #1 to do instead)
  3. to just update infrequently

Specifically, you have two different package managers (apt and pip) that don't know much about each other and aren't co-ordinated.

I'd recommend you:

  • get a working initial development environment
  • choose some way to be able to clone that environment when you want a new VM (or docker or other) container starting at a working baseline
  • don't update it at all unless there's a specific feature or security update you want
  • don't update it when you actually want to be developing, because you'll get frustrated whenever it doesn't work instantly
  • ideally perform updates in a clone, so you can give up and get back to developing in a working environment if it is more broken than you can face fixing right away
Useless
  • 12,823
3

I avoid updating & upgrading my OS and IDE, unless I know there is something in new versions that will solve my problem.

lennon310
  • 3,242
Ehsan Abidi
  • 146
  • 2
2

My approach is:

  • update the Linux distro often (daily) - most (even Debian testing) are very stable
  • update python rarely. In my experience it often breaks.
  • as a corollary, install any python packages available from your Ubuntu repos from there - not through pip.
Vorac
  • 7,169
1

In terms of practical solutions, I'd encourage you to have a look at Nix. It's a package manager which makes it possible to isolate the environment for each of your projects from each other and from the base system. In practice this means you can set up a small file like this in your project, listing the development dependencies, and simply run nix-shell in that directory to build and enable all those tools for the remainder of that session. There are also tools which wrap common project-level package managers to install those packages as part of your isolated environment. Basically, you can replace all of pip, npm, apt, docker, make etc. with a declarative setup.

As with all such things, there's an initial learning investment to get started, but after using frankly inferior solutions like Ansible and Puppet for years it's well worth it.

If that sounds interesting you might also be interested in NixOS, which takes it to the next level by configuring the entire system like this. I recently migrated to using Nix on NixOS instead of Puppet, ending up with this configuration. That might look like a lot, but it replaced over 130 files and nine third party modules with 350 lines which do a lot more than the original Puppet.

l0b0
  • 11,547