5

I have a makefile which wraps the real build in a single recursive call, in order to grab and release a license around the build, regardless of whether the build succeeds. Example:

.PHONY main_target
main_target:
    @license_grab &
    @sleep 2
    -@$(MAKE) real_target
    @license_release

This works great, until I want to run the make in parallel. On some systems it works fine, but on other systems I run into the documented problem with the -j flag when passing options to a sub-make:

If your operating system doesn’t support the above communication, then ‘-j 1’ is always put into MAKEFLAGS instead of the value you specified. This is because if the ‘-j’ option were passed down to sub-makes, you would get many more jobs running in parallel than you asked for.

Since I only ever have one recursive call, I'd really like to pass the -j flag untouched to the sub-make. I don't want to hard-code the -j flag (with a number) into the makefile, because the makefile runs on multiple machines, with differing numbers of processors. I don't want to use -j with no limit, because that launches a bunch of processes right away rather than waiting for jobs to finish. I tried using the -l flag when I build, but I found that the limit doesn't apply right away, probably because limits don't apply until make can start sampling.

Is there a way to force a sub-make to use multiple jobs? Or, a way to make the -l flag actually accomplish something?

I think I could do it using a makefile modification like using -@$(MAKE) $(JOBS) real_target, and invoking make like make JOBS="-j4" main_target.

But, I'd prefer to just use standard make parameters, not adding extra dependencies on variables. I'd also prefer to modify the makefile as little as possible.

Community
  • 1
  • 1
Ben
  • 8,725
  • 1
  • 30
  • 48
  • 1
    Oops, I just realized the same [workaround to pass the -f flag](http://stackoverflow.com/a/28888418/1390430) on my old system will also work for the -j flag. E.g. `make MAKE="make -j4" main_target`. So unless there is a better solution, I think I'll use that. – Ben Mar 11 '15 at 18:41

2 Answers2

4

There is no way to change this behavior on systems which don't support jobserver capabilities. Something like your JOBS variable is the only way I can think of.

I'll point out a few things: first, I assume when you say other systems you mean Windows systems: that's the only commonly-used platform I'm aware of which doesn't support the jobserver. If so, note that as of GNU make version 4.0, jobserver support is implemented on Windows. So another option is to upgrade to a newer version of GNU make.

Second, the issues with the -l option were at least partly solved in GNU make version 3.81, where an extra algorithm was introduced to artificially adjust the load average based on the number of jobs make invokes. You shouldn't see this issue any longer with versions of make after that.

MadScientist
  • 92,819
  • 9
  • 109
  • 136
  • Sadly "other systems" refers to an old Solaris build server which for some reason is ignoring my -j flag and passing -j1 to the sub make. And I'm stuck using a fork of GNU make 3.79.1. Oddly, parallel builds seem to work fine on Windows with a cygwin-based version of the build tools. – Ben Mar 11 '15 at 18:53
  • There is no reason that jobserver support shouldn't work from day 1 on even the most ancient UNIX system: it was purposefully written to require only the most basic of UNIX features. If it isn't enabled on this Solaris system then it's an error at configure time, where configure can't tell that Solaris has the necessary support and so disables jobserver. Or possibly your fork has broken things. However, if you want to use such an old version of GNU make and not upgrade, then you're pretty much SOL, sorry... – MadScientist Mar 11 '15 at 19:09
  • 1
    Also, Cygwin provides a POSIX API, so jobserver works there; the new support in 4.0 allows jobserver to work with the native Windows API (for example, in the MinGW port of GNU make). – MadScientist Mar 11 '15 at 19:09
  • That's good to know. I've confirmed there is definitely only one job running at a time without using some trick or another, so they must have intentionally disabled or accidentally broken the job server when they forked to certify the compiler tools way back whenever they did it. – Ben Mar 11 '15 at 19:25
2

The problem is that for whatever reason my make does not support the job server, thus it does not pass the -j flag on in the $(MAKEFLAGS) variable. Since upgrading make to a version that does pass the flag is not an option, the solution is to pass the -j flag elsewhere. Just like the $(MAKE) variable can be abused to pass the -f flag, it can be used in the same way to pass the -j flag:

make MAKE="make -j4" main_target

This will start the main_target build with one job, but invoke make with 4 jobs on the sub-make process. Obviously, if you need a special make tool (the normal purpose of $(MAKE) then you'll need to specify it in the MAKE string as well as in the command.

Community
  • 1
  • 1
Ben
  • 8,725
  • 1
  • 30
  • 48