Starting application using node works but not yarn or npm

Question

There is probably an obvious answer to someone who is more familiar with NodeJS than me but:

When running an application which is using a node-gyp package, it runs without problems using node ./src/index.js

However running it using a very simple package.json start-script:

"scripts": {
    "start": "node ./src/index.js"
  }

in npm:

npm start

npm info it worked if it ends with ok
npm info using npm@6.9.0
npm info using node@v11.10.1
npm info lifecycle rosbag2@0.1.0~prestart: rosbag2@0.1.0
npm info lifecycle rosbag2@0.1.0~start: rosbag2@0.1.0

> rosbag2@0.1.0 start /Users/andreasklintberg/personal/xviz/examples/converters/rosbag2
> node ./src/index.js "./src/index.js"

internal/modules/cjs/loader.js:779
  return process.dlopen(module, path.toNamespacedPath(filename));
                 ^

Error: dlopen(/Users/andreasklintberg/personal/xviz/examples/converters/rosbag2/node_modules/rosbags2_nodejs/build/Release/rosbags2_nodejs.node, 1): Library not loaded: @rpath/librosidl_typesupport_cpp.dylib
  Referenced from: /Users/andreasklintberg/personal/xviz/examples/converters/rosbag2/node_modules/rosbags2_nodejs/build/Release/rosbags2_nodejs.node
  Reason: image not found

And similarly in yarn yarn start

yarn run v1.13.0
warning package.json: No license field
$ node ./src/index.js ./src/index.js -d ../../../data/rosbag2/rosbag2_2019_03_09-21_54_44 -o ../../xviz-data/rosbag2/rosbag2_2019_03_09-21_54_44/
internal/modules/cjs/loader.js:779
  return process.dlopen(module, path.toNamespacedPath(filename));
                 ^

Error: dlopen(/Users/andreasklintberg/personal/xviz/examples/converters/rosbag2/node_modules/rosbags2_nodejs/build/Release/rosbags2_nodejs.node, 1): Library not loaded: @rpath/librosidl_typesupport_cpp.dylib
  Referenced from: /Users/andreasklintberg/personal/xviz/examples/converters/rosbag2/node_modules/rosbags2_nodejs/build/Release/rosbags2_nodejs.node
  Reason: image not found

I know the error is because a dynamic link/@rpath is messed up, and I was thinking it was because npm/yarn messes with the env variables or something, but running yarn env it seems everything is correct.

So I guess i'm wondering if anyone know what the difference is running it in native node and using npm/yarnscript to wrap the start? And why this difference messes up the @rpath/dynamic linking ?

Edit:

My index.js file is very simple, just imports the built node-gyp package:

let RosbagDeserializer = require('../build/Release/rosbags2_nodejs.node');


const deserializer = new RosbagDeserializer.Rosbag2Wrapper();

This is the project in question, https://github.com/klintan/rosbag2_nodejs

Could you try deleting node_modules folder and run `npm update`. I think this problem could be due to build from a certain version of node, you might need to rebuild certain packages from source. — PrivateOmega, Apr 02 '19 at 03:28
Also could you try building https://github.com/cruise-automation/rosbag.js in your computer? This project might require some special settings to build on OSX I guess. Which I think is your OS, please correct me if I am wrong. — PrivateOmega, Apr 02 '19 at 03:50
I'll try that, this is the project in question (which is Ros2 not Ros1 as the cruise repo): https://github.com/klintan/rosbag2_nodejs needs Ros2 env to be sourced, but then you should be able to run the example. (That is the one I'm trying to run, which works by running "node examples/deserialize.js" but not using "npm start" Yes on MacOS. node v11.12.0 — Andreas Klintberg, Apr 02 '19 at 04:00
Did you face any issues while building the project by steps mentioned under Installation Section? I assumed it was the one by cruise-automation because it was found first on a quick google search. — PrivateOmega, Apr 02 '19 at 04:06
The `node ./src/index.js "./src/index.js"` is interesting. It seems to me that the `npm` runs your file in a weird way. Could you please provide us with some more details in your package.json? Like `main` filed for example — Sebastian Kaczmarek, Apr 05 '19 at 10:09
@SebastianKaczmarek added some info, it's very simple, it basically just tries to import/require the built node-gyp file. In one I defined main to be index.js but I'll try that again. — Andreas Klintberg, Apr 07 '19 at 05:28
@AndreasKlintberg dynamic linking may be affected by environment variables - I believe that you may be observing an effect of this. `npm start` will start the app with some environment cleanup (like `LD_LIBRARY_PATH` for instance). Try setting `scripts.start` to something like this: `node -e 'console.log(process.env)` and see the difference with running this from command line. — Michał Karpacki, Apr 08 '19 at 12:01
You can go through this thread, I am not so well versed in Mac but I guess this will help. https://stackoverflow.com/questions/17703510/dyld-library-not-loaded-reason-image-not-found — Avik, Apr 09 '19 at 02:32

Andreas Klintberg · Answer 1 · 2019-04-09T15:50:23.267

I would love a more comprehensive answer but at least I solved it with the help of a lot of wonderful commenters:

The @rpath is not set on Mac, but with DYLD_LIBRARY_PATH it works, because that is one of the runtime search paths, it seems (https://superuser.com/questions/282450/where-do-i-set-dyld-library-path-on-mac-os-x-and-is-it-a-good-idea). However for some reason NPM removes this, I'm guessing it has to do with SIP on Mac, or maybe some other security measure that removes this env variable.

DYLD_LIBRARY_PATH is available if you run node index.js but NOT if you are running it using npm start (node index.js start script).

Both @Michal Kapracki and @Avik was hinting at this (I missed the missing env variables when comparing the first time)

What I did to solve it was similar to the dyld: Library not loaded ... Reason: Image not found question as @Avik linked, I had to add a bunch of libraries which were dependent (that was automatically resolved before) and a post_install.sh script which uses the install_name_tool to change the @rpaths to the correct path.

It's a pretty ugly solution https://github.com/klintan/rosbag2_nodejs/blob/master/post_install.sh but it works. Only for Mac though, so needs to be generalized.

So in summary what I think is the issue:

NPM for some reason can't use or removes the env variable DYLD_LIBRARY_PATH which is used to search for dynamically linked libraries during runtime. This makes it necessary to link these manually using the mac tool install_name_tool.

Starting application using node works but not yarn or npm

1 Answers1