0

I want to move to make 4.2 so I can have the ability to do parallel jobs on Windows using --jobs=N.

make 4.2 is not available in binary form from SourceForge, only 3.81

So I had to build it from source - they included a VS studio project - and I was up and running.

Problem is, it appears to have a bug though it could be a bug with my makefile.

I ran make4.2 all --print-data-base --debug=vjm on my project and I got ERRORS. What's unfortunate is that the error is NOT related to syntax of rules as this project builds without error on make 3.81 using the same options.

Here is the command it fails on.

COPY_DIR      = cp -f -R
MKDIR         = mkdir -p

deploy_marketProperties: deploy_themeMappingConfig
    @echo Copying application data... && $(MKDIR) "C:/Users/User1/Desktop/TargetDir" && $(COPY_DIR) $(wildcard C:/Users/User1/Desktop/Source/*) "C:/Users/User1/Desktop/TargetDir"
  1. Creates a directory
  2. Copies bunch of files from one directory to that new directory

Used the same make call for both versions of make: make all --print-data-base --debug=vjm

However, I get the following error when using make4.2:

Copying application data...
cp: target `C' is not a directory
makev42[3]: *** [Makefile:383: deploy_marketProperties]

The command it tried running was:

echo Copying application data... && mkdir -p "C:/Users/User1/Desktop/TargetDir" && cp -f -R C:/Users/User1/Desktop/Source/im.json "C:/Users/User1/Desktop/TargetDir"

There's actually 264 .json files but I only show one for readability.

Again, this runs fine on make v3.81 using exact same command-line options.

It appears make 4.2 has a problem parsing that huge list of files maybe??

The question

Is there something I can change with my makefiles to make it run on make v4.2?

We are highly motivated to make it work with a 4.0+ version of make. Otherwise we have to use a build tool that will cost $$.

UPDATE 1

Per the suggestion of @MadScientist, I have executed both versions of make and present the outputs.

C:\release>make381 deploy_marketProperties --debug=vjm > 381.err 2>&1

C:\release>make42 deploy_marketProperties --debug=vjm > 42.err 2>&1

The target deploy_marketProperties has the following rule:

deploy_marketProperties: 
    echo Copying application data... && $(MKDIR) "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && $(COPY_DIR) $(wildcard C:/Users/User1/Desktop/A/Project/src/dir1/marketProperties/*) "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"

The output of make v 3.81. NOTE: there's actually 264 .json files and total number of characters that make up the list of .json files, including ONE white space per file, is 18251 (in case that's relevant):

GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for Windows32
find_and_set_shell path search set default_shell = C:/QNX650/host/win32/x86/usr/bin/sh.exe
Reading makefiles...
Reading makefile `Makefile'...
Updating goal targets....
Considering target file `deploy_marketProperties'.
 File `deploy_marketProperties' does not exist.
 Finished prerequisites of target file `deploy_marketProperties'.
Must remake target `deploy_marketProperties'.
echo Copying application data... && mkdir -p "C:/Users/User/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && cp -f -R C:/Users/User/Desktop/A/Project/src/fordhmi/marketProperties/2blank.json C:/Users/User/Desktop/A/Project/src/fordhmi/marketProperties/3blank.json  "C:/Users/User/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"
Main thread handle = 0x000000ac
Copying application data...
Successfully remade target file `deploy_marketProperties'.

The output of make v 4.2. NOTE: there's actually 264 .json files and total number of characters that make up the list of .json files, including ONE white space per file, is 18251 (in case that's relevant):

GNU Make 4.2
Built for Windows32
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
find_and_set_shell() path search set default_shell = C:/QNX650/host/win32/x86/usr/bin/sh.exe
Reading makefiles...
Reading makefile 'Makefile'...
Updating makefiles....
Updating goal targets....
Considering target file 'deploy_marketProperties'.
 File 'deploy_marketProperties' does not exist.
 Finished prerequisites of target file 'deploy_marketProperties'.
Must remake target 'deploy_marketProperties'.
echo Copying application data... && mkdir -p "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && cp -f -R C:/Users/User1/Desktop/A/Project/src/SubProj/marketProperties/im.json C:/Users/User1/Desktop/A/Project/src/SubProj/marketProperties/da.json  "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"
Main thread handle = 00000088
Copying application data...
cp: target `C' is not a directory
make_msvc.net2003: *** [Makefile:384: deploy_marketProperties] Error 1

Note that both make versions use the same shell:

C:\Users\User1\Desktop\A\Project\bld\armle-v7\release\subProj>where sh
C:\QNX650\host\win32\x86\usr\bin\sh.exe

C:\Users\User1\Desktop\A\Project\bld\armle-v7\release\subProj>sh --help
GNU bash, version 3.1.17(1)-release-(i686-pc-msys)
Usage:  sh [GNU long option] [option] ...
        sh [GNU long option] [option] script-file ...
GNU long options:
        --debug
        --debugger
        --dump-po-strings
        --dump-strings
        --help
        --init-file
        --login
        --noediting
        --noprofile
        --norc
        --posix
        --protected
        --rcfile
        --restricted
        --verbose
        --version
        --wordexp
Shell options:
        -irsD or -c command or -O shopt_option          (invocation only)
        -abefhkmnptuvxBCHP or -o option
Type `sh -c "help set"' for more information about shell options.
Type `sh -c help' for more information about shell builtin commands.
Use the `bashbug' command to report bugs.

NOTE THE msys @MadScientist; is that relevant?

UPDATE 2

I re-ran make 4.2 but I deleted all but ONE .json file in the directory and - surprise! - it works.

This is the ENTIRE output from make42 deploy_marketProperties --debug=vjm:

GNU Make 4.2
Built for Windows32
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
find_and_set_shell() path search set default_shell = C:/QNX650/host/win32/x86/usr/bin/sh.exe
Reading makefiles...
Reading makefile 'Makefile'...
Updating makefiles....
Updating goal targets....
Considering target file 'deploy_marketProperties'.
 File 'deploy_marketProperties' does not exist.
 Finished prerequisites of target file 'deploy_marketProperties'.
Must remake target 'deploy_marketProperties'.
echo Copying application data... && mkdir -p "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && cp -f -R C:/Users/User1/Desktop/A/Project/src/subProject/marketProperties/2blank.json "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"
Main thread handle = 000000B4
Copying application data...
Successfully remade target file 'deploy_marketProperties'.

UPDATE 3

I tried it with make 4.2.1 and I got the exact same behavior as on make 4.2: fails when there are hundreds of .json files but succeeds if there less than some magic number of characters.

I got this message during one of my trials:

cp: target `C:/Users/User1/Des' is not a directory

Obviously it's getting cut-off but why? Same shell being used and only thing different is version of make.

Bob
  • 4,576
  • 7
  • 39
  • 107
  • Where did you get your previous version of make from? – user657267 Jun 21 '16 at 03:52
  • @user657267 I got the 3.81 from an SDK that we use from QNX. And I built 4.2 from source from the links I posted. – Bob Jun 21 '16 at 03:54
  • Also what environment are you running make under? Cygwin? MSYS? – user657267 Jun 21 '16 at 06:01
  • @user657267 just regular Windows batch. I guess that's MSYS? – Bob Jun 21 '16 at 14:26
  • The Windows shell environment is not even remotely close to the emulated UNIX environment for windows (MSYS / Cygwin), you're mixing up quite a few things here. If you actually **need** the UNIX env ... well ... you might run into weird problems - at least thats my experience. MSYS, Cygwin and similar solutions tend to be of .... questionable stability - they might use excellent sources but the WINAPI adaptations are buggy at best. Sorry but it s just like that – specializt Jun 21 '16 at 15:45
  • @specializt I'm using the code from the authors of `make` using their visual studio solution file to build it. This means they meant for it to be compiled for Windows and for it to actually run on windows. – Bob Jun 21 '16 at 15:47
  • un-hunh ... thats nice. Btw : Visual studio is a complex *(and sometimes horrible)* beast - a lot of people get frustrated and start inventing their own build system which circumvents MSBUILD entirely or in parts. Thats where the problems start, sometimes. – specializt Jun 21 '16 at 15:48
  • @specializt I don't need the Unix environment. I need to know why `make 4.2` chokes while `make 3.81` works fine. – Bob Jun 21 '16 at 15:49
  • http://stackoverflow.com/a/1745964/351861 – specializt Jun 21 '16 at 15:51
  • @specializt there is nothing to debug. It's a command that creates a directory then copies a bunch of files to that directory. Has nothing to do with make rules. I've already run `make 4.2` with `--trace --print-data-base -d` and I've pasted what it says. It has a parsing issue that I'm pretty sure is a bug. – Bob Jun 21 '16 at 15:53
  • Make sure you configure with `--host=` set to the correct value for your environment. There are a *lot* of triples that run on the Windows kernel. – o11c Jun 21 '16 at 21:49
  • @o11c is `--host=` a `make` thing? Please provide a link. Thanks. – Bob Jun 21 '16 at 21:52
  • @Adrian It's standard for all GNU software, and a lot of other software too. See https://www.gnu.org/software/autoconf/manual/html_node/Specifying-Target-Triplets.html#Specifying-Target-Triplets – o11c Jun 21 '16 at 23:08
  • @o11c I'm running proprietary compilers not gcc. What does this have to do with my specific issue? See the latest update: make 4.2 fails because it has a character limit. – Bob Jun 21 '16 at 23:12
  • Autoonf exists *because* of proprietary compilers. If GCC could be relied on when building softare, everything would be *much* simpler. – o11c Jun 21 '16 at 23:13
  • @o11c my project doesn't use autoconf. – Bob Jun 21 '16 at 23:16
  • @Adrian But GNU make does, and that's what has been built incorrectly. – o11c Jun 21 '16 at 23:52
  • @o11c make 3.81 works while make 4.2 or 4.2.1 doesn't work. I'm not using autoconf at all. I don't see what autoconf has to do with this. – Bob Jun 21 '16 at 23:58
  • @Adrian In order to *build* GNU make, you are using autoconf. That's the problem here! – o11c Jun 22 '16 at 01:41
  • @o11c I'm using the visual studio solution file that comes with the gzip file from the GNU Make site. Since they're the ones who created the `.sln` project file specifically for Windows, I'm assuming they know how to do it correctly. Once I've built make 4.2, I use that to build my actual project. – Bob Jun 22 '16 at 01:43
  • @o11c whether autoconf is used as part of the `.sln` configuration is irrelevant to me: I extract their gzipped folder, build using visual studio and the included `.sln` and then I have a `make 4.2.1` executable. – Bob Jun 22 '16 at 01:48
  • @o11c in any case, if you read the updates to my post, you'll see that it works as long as I don't have too many JSON files in the line of the recipe whereas `make 3.81` can handle it fine. THAT is the issue. – Bob Jun 22 '16 at 01:51
  • you seem to have identified your exact problem cause ... so by now you know that you need to debug and you dont need any more help, good luck. – specializt Jun 22 '16 at 07:47

2 Answers2

0

You should use 4.2.1, it has some important fixes (I doubt it will make a difference for this problem).

As discussed in the comments, using POSIX tools on Windows requires a specialized environment. The normal "shell" on Windows is command.com, which doesn't accept POSIX sh syntax, such as &&. The normal mkdir command on Windows doesn't accept the -p flag; that's a POSIX flag. There is no cp command on Windows.

So it's clear that your makefile is expecting to be run with some sort of POSIX environment.

Based on what you've described above, it's not clear how the problem is related to GNU make directly. GNU make is parsing the makefile and it's invoking the shell and passing along the command line properly: this can be inferred because the shell is running the echo command and running the mkdir command, and even running the cp command.

It's the cp command which is printing the message, not make. I believe you that it works with 3.81, but I suspect a non-obvious difference between the way 3.81 was compiled and the way you're building 4.2. It's even possible that there have been custom patches applied to the 3.81 version you're using, although I don't know.

To debug further I recommend the following:

  • First remove the @ prefix from the recipe line: that's always the number one thing to do; if you can't see how make is invoking the command how can you debug?
  • Then run the same makefile with the two different versions of make, from the same shell/prompt (so the path to make is the only difference). Is there any difference in the printed output? If so what is it?
  • If not, then clearly there's something more mysterious going on. Try to see if maybe the QNX version of make is modifying the environment somehow: you can run the env command in your recipe to see it.

Maybe it's related to the length of the command line? Try temporarily moving some of the json files out of that directory so the command line is shorter.

MadScientist
  • 92,819
  • 9
  • 109
  • 136
  • The `mkdir` and `cp` commands are part of the QNX SDK. `C:\Users\User\Desktop\A\test_compile>where cp C:\QNX650\host\win32\x86\usr\bin\cp.exe C:\Users\User\Desktop\A\test_compile>where mkdir C:\QNX650\host\win32\x86\usr\bin\mkdir.exe` – Bob Jun 21 '16 at 19:03
  • Looks like you have a MinGW32 setup that comes with QNX. the question is which shell is being invoked by make. This is a complex question; see the README files. Make wants to use a UNIX shell, if it can find one, so it goes looking. It's possible that the QNX make finds a different one. Anyway I recommend following my debug steps above and updating your question with the new information (putting it in comments is not useful since there's no good formatting). – MadScientist Jun 21 '16 at 19:13
  • I already pasted the command it tried running; see `The command it tried running was:` – Bob Jun 21 '16 at 19:21
  • I asked you to try 3 things: your paste may have covered the first one, at most. – MadScientist Jun 21 '16 at 19:53
  • Looks like it's related to length of command-line. How do I fix it? – Bob Jun 21 '16 at 21:37
0

I asked the make-w32@gnu.org mailing list and I have the answer.

To fix it, I had to uncomment the macro BATCH_MODE_ONLY_SHELL in config.h of the make source code.

/*
 * If you have a shell that does not grok 'sh -c quoted-command-line'
 * correctly, you need this setting. Please see below for specific
 * shell support.
 */
#define BATCH_MODE_ONLY_SHELL 1

Taking quote from reply from mailing list:

What that flag means (as I understand it: I'm not that familiar with this aspect of Windows support) is that make will never try to invoke the shell directly passing the recipe to be run on the command line.

Instead it is being forced to always write the recipe to a temporary file ("batch file") on your disk and invoke the shell such that it runs the recipe in the temporary file.

Apparently your shell is able to read and execute very long command lines from a file, which it cannot successfully process when passed in as arguments, even though these command line arguments are not exceeding Windows limits.

Bob
  • 4,576
  • 7
  • 39
  • 107