node-gyp: run binding.gyp in all subdirectories

Question

I'm developing a big node.js project which also includes several native libraries.
To use these libraries in JavaScript I'm compiling them to node addons (.node) using node-gyp.

I'd like to run node-gyp once from the root directory to compile all the available binding.gyp recursively (in all the subdirectories).

Is there any way to do that?

score 5 · Accepted Answer · edited Aug 16 '16 at 08:54

5

GYP allows to set a list of dependencies for a target. You can create a target of type: none in the top-level bindings.gyp and list there dependencies from subdirectories:

{
    'targets': [
        {
            'target_name': 'build_all',
            'type': 'none',
            'dependencies': ['subdir1/bindings.gyp:*', 'subdir/subdir2/bindings.gyp:*'],
            # or generate dependencies list with a command expansion
            'dependencies': ['<!@(find -mindepth 2 -name binding.gyp | sed -e s/$/:*/)'],
        }
    ]
}

This will compile all the dependencies and put them into build/ directory in the root.
For putting each addon in its corresponding directory, add a postbuild target inside the addon's binding.gyp:

{
  "targets": [
    {
      "target_name": "my-target",
      "sources": [ "example.cpp" ]      
    },      
    {
      "target_name": "action_after_build",
      "type": "none",
      "dependencies": [ "my-target" ],
      "copies": [
        {
          "files": [ "<(PRODUCT_DIR)/my-target.node" ],
          "destination": "."
        }
      ]
    }
  ]
}

edited Aug 16 '16 at 08:54

JeB

11,653
10
58
87

answered Aug 03 '16 at 12:36

pmed

1,536
8
13

This way every time I add a new native module I'll have to update this bindings.gyp index. – JeB Aug 03 '16 at 14:22
GYP also can expand a command, so you can combine approaches from both answers. But you will need to re-generate build files with `node-gyp rebuild` after adding new native module. – pmed Aug 03 '16 at 15:23
I intend to run `node-gyp rebuild` anyways (in fact you can see that in my solution). But I'm not sure I got your point... What do you mean by GYP also can expand a command? – JeB Aug 04 '16 at 14:27
See (Command Expansions)[https://gyp.gsrc.io/docs/InputFormatReference.md#Command-Expansions] in GYP reference. GYP will execute a shell command on `!(...)` or `!@(..)` variable expansion. But GYP is running only on configure stage, i.e. `node-gyp configure` or `node-gyp rebuild` – pmed Aug 04 '16 at 14:28
I see... so you propose put `"find ./app/* -name binding.gyp"` inside a root `gyp` file instead of `package.json` scripts? Why do you think it is a better approach? – JeB Aug 04 '16 at 14:31
I don't know which is the better approach, it's up to you. I just suggest another possible way. – pmed Aug 04 '16 at 14:35
I understand. I think I'll accept your answer, as it integrates in a more seamless way into the build process. In fact, `node-gyp rebuild` is a [default command in install script](https://docs.npmjs.com/misc/scripts) (given there is a `bindings.gyp` file in the root directory). And it is [mentioned](https://docs.npmjs.com/misc/scripts) in best practices to avoid the use of `install` and use `gyp` file instead. – JeB Aug 04 '16 at 14:45
For some reason the expansion won't work when there are several `binding.gyp` files in the sub directories... It works fine when there is only one, but when there are two or more I get this error `gyp: native\node-addon\binding.gyp \native\node-addon-Copy\binding.gyp not found` – JeB Aug 07 '16 at 13:41
Figured out what is the problem: when you use list expansion, something like `<@(var)blah`, the `blah` thing will be only attached to the last entry in the list. Therefore the solution you proposed won't work. – JeB Aug 08 '16 at 09:25
1

Yes, I have discovered the same. I always thought that expansion is being applied to each list item. Anyway there is a workaround: to generate list items with required suffix `:*`, like `find -mindepth 2 -name binding.gyp | sed -e s/$/:*/` I also added `-mindepth 2` to exclude the top-level `binding.gyp` from the result list – pmed Aug 08 '16 at 09:32
Yeah, I've been looking into the `sed` too. Solves the problem. – JeB Aug 08 '16 at 09:42
Obviously this won't work on Windows. A kind of `find` analogue is `dir /b/s` there, so to append `:*` suffix such one-line can be used: `for /f %i in ('dir /s/b binding.gyp') do @echo %i:*` but I haven't tested it with GYP – pmed Aug 08 '16 at 09:46
Correct. Although it won't be an issue if one uses bash on Windows (gitbash or something). There's still another issue though... If you build a common target (top level `binding.gyp`) it puts all the binaries inside the top level build `directory`. I'd like each of the binaries to be placed in its corresponding directory (alongside the corresponding `binding.gyp`) – JeB Aug 08 '16 at 10:00
Meanwhile the only solution I found for this one is adding a copy step in each child `binding.gyp`. I'll suggest and edit for your answer. – JeB Aug 08 '16 at 13:35

score 2 · Answer 2 · answered Aug 01 '16 at 09:04

I didn't find any option to do this with just node-gyp, but one of the possible solutions is doing this in a script.
For example, adding the following to the package.json in the root folder:

 "scripts": {
    "install": "find ./app/* -name binding.gyp -execdir node-gyp rebuild ;"
 }

This will cause all the native addons to compile when running npm install in the root folder.

score 0 · Answer 3 · answered Mar 31 '20 at 10:08

An alternative to the other answers which seems to work so far (without ever having to update binding.gyp):

{
  "targets": [
    {
      "target_name": "addon",
      "sources": [ 
            "<!@(node -p \"var fs=require('fs'),path=require('path'),walk=function(r){let t,e=[],n=null;try{t=fs.readdirSync(r)}catch(r){n=r.toString()}if(n)return n;var a=0;return function n(){var i=t[a++];if(!i)return e;let u=path.resolve(r,i);i=r+'/'+i;let c=fs.statSync(u);if(c&&c.isDirectory()){let r=walk(i);return e=e.concat(r),n()}return e.push(i),n()}()};walk('./sources').join(' ');\")"
        ]
    }
  ]
}

(from https://stackoverflow.com/a/60947528/2016831)

node-gyp: run binding.gyp in all subdirectories

3 Answers3