18

Is there any software available in linux which compiles a source code containing large number of files parallely on either multicore or distributed systems. Libraries like gcc or xserver takes very time for compilation on unicore/dual machine and most of the times it is frustrating when you need lot of recompilation. Is there any technique for compiling such source code parallely ?

atv
  • 1,950
  • 4
  • 21
  • 26
  • related: http://stackoverflow.com/questions/414714/compiling-with-g-using-multiple-cores – miku Dec 27 '09 at 17:33

3 Answers3

31

On distributed-memory systems, you can use distcc to farm out compile jobs to other machines. This takes a little bit of setup, but it can really speed up your build if you happen to have some extra machines around.

On shared-memory multicore systems, you can just use make -j, which will try to spawn build jobs based on the dependencies in your makefiles. You can run like this:

$ make -j

which will impose no limit on the number of jobs spawned, or you can run with an integer parameter:

$ make -j8

which will limit the number of concurrent build jobs. Here, the limit is 8 concurrent jobs. Usually you want this to be something close to the number of cores on your system.

Todd Gamblin
  • 58,354
  • 15
  • 89
  • 96
  • 1
    AFAIK there are two schools on the "number of jobs". One being n+1 and the other 2n, where n is the number of cores. The rationale of both is that there is always a thread waiting once one gets completed... Personally, I have not seen real difference whether I put -j5 or -j8 on a quadcore – Kimvais Dec 27 '09 at 18:58
  • I also haven't seen much benefit from either n+1 or 2n on multicores, but then I haven't seen much of a decrease in performance from those either. I'm suspicious that you don't get all the benefit you could out of those options b/c make doesn't set affinity for cores or sockets, but I haven't really tested this (I haven't even checked if make sets affinity, but I bet it doesn't). – Todd Gamblin Dec 27 '09 at 19:29
  • 1
    Building is often IO bound. Lots of small files to work on means moving a lot of data to and from the disk... Plus forks are inexpensive on linux, but they *do* cost something, which contributes to the non-scaling. Finally, recursive make and other poor makefile practices can steal *a lot*. – dmckee --- ex-moderator kitten Dec 28 '09 at 02:16
4

ccache can be used too to speed up compilation process

dimba
  • 26,717
  • 34
  • 141
  • 196
  • ccache only really benefits when you often need to rebuild identical artifacts, and if you do need to do that, your build system is defect to begin with. – JesperE Dec 27 '09 at 18:46
3

GNU make (the standard make installed on Linux systems) supports parallel command execution - take a look at http://www.gnu.org/software/make/manual/make.html#Parallel