11

When I (re-)build large systems on a desktop/laptop computer, I tell make to use more than one thread to speed up the compilation speed, like this:

$ make -j$[ $K * $C ]

Where $C is supposed to indicate the number of cores (which we can assume to be a number with one digit) the machine has, while $K is something I vary from 2 to 4, depending on my mood.

So, for example, I might say make -j12 if I have 4 cores, indicating to make to use up to 12 threads.

My rationale is, that if I only use $C threads, cores will be idle while processes are busy fetching data from the drives. But if I do not limit the number of threads (i.e. make -j) I run the risk to waste time switching contexts, run out of memory, or worse. Let's assume the machine has $M gigs of memory (where $M is in the order of 10).

So I was wondering if there is an established strategy to choose the most efficient number of threads to run.

MattDavey
  • 7,176
bitmask
  • 879

2 Answers2

16

I ran a series of tests, building llvm (in Debug+Asserts mode) on a machine with two cores and 8 GB of RAM:

compiling llvm time depending on number of jobs

Oddly enough, it seems to climb until 10 and then suddenly drops below the time it takes to build with two jobs (one jobs takes about the double time, not included in the graph).

The minimum seems to be 7*$cores in this case.

bitmask
  • 879
3

I'm running Gentoo Linux (source-based distribution) and from my experience i can say that (with more or less recent hardware) n*2 + x is the best value. Let me explain this:

  • n*2: Even slower CPU's have enough power to run 2 tasks at a time. most compile tasks are completed very fast.
  • +x this number depends on your system (mainly memory and disk). If you have enough RAM and a fast disk, set x=n. However, this depends on the source code (Open Office, i'm looking at you!) and the used language (compiling C/C++ is very memory intensive).

However, you have to run some tests with some -j values to get the best number. Also, try to parallelize other steps of the build process: unpacking, running configure and so on.

ercpe
  • 151