349

I have a question concerning best practice for including node_modules into a HTML website.

Imagine I have Bootstrap inside my node_modules folder. Now for the production version of the website, how would I include the Bootstrap script and CSS files located inside the node_modules folder? Does it make sense to leave Bootstrap inside that folder and do something like the following?

<script src="./node_modules/bootstrap/dist/bootstrap.min.js"></script>

Or would I have to add rules to my gulp file which then copy those files into my dist folder? Or would it be best to let gulp somehow completely remove the local bootstrap from my HTML file and replace it with the CDN version?

Buldo
  • 99
  • 7
Chris
  • 6,093
  • 11
  • 42
  • 55
  • 2
    related topic http://stackoverflow.com/questions/24397379/script-path-for-expressjs-in-node-modules. SO heres the questino @Palak Bhansali Assuming one need only, does it justify implementing bower for this sole purpose on say for example a gulp express app isntalling bootstrap directly from npm, needing access to the files now in gulp, which are in node_modules. Does this justify a single use case for needing bower now for this sole purpose? Thats the problem im running into. I already use composer, npm, gulp, grunt, dont want bower personally, and dont want grunt on this app either. – blamb Oct 12 '15 at 10:27
  • Excellent question that needs an intuitive solution! – Sydwell Apr 25 '17 at 11:07
  • Realistically speaking, this is a vital issue that is left untouched. There are so many JS/CSS libraries that are not compatible with popular framework. This is a huge gap when developers are moving always from bower and JSPM. – vichsu Oct 22 '20 at 22:09

12 Answers12

362

Usually, you don't want to expose any of your internal paths for how your server is structured to the outside world. What you can is make a /scripts static route in your server that fetches its files from whatever directory they happen to reside in. So, if your files are in "./node_modules/bootstrap/dist/". Then, the script tag in your pages just looks like this:

<script src="/scripts/bootstrap.min.js"></script>

If you were using express with nodejs, a static route is as simple as this:

app.use('/scripts', express.static(__dirname + '/node_modules/bootstrap/dist/'));

Then, any browser requests from /scripts/xxx.js will automatically be fetched from your dist directory at __dirname + /node_modules/bootstrap/dist/xxx.js.

Note: Newer versions of NPM put more things at the top level, not nested so deep so if you are using a newer version of NPM, then the path names will be different than indicated in the OP's question and in the current answer. But, the concept is still the same. You find out where the files are physically located on your server drive and you make an app.use() with express.static() to make a pseudo-path to those files so you aren't exposing the actual server file system organization to the client.

If you don't want to make a static route like this, then you're probably better off just copying the public scripts to a path that your web server does treat as /scripts or whatever top level designation you want to use. Usually, you can make this copying part of your build/deployment process.

If you want to make just one particular file public in a directory and not everything found in that directory with it, then you can manually create individual routes for each file rather than use express.static() such as:

<script src="/bootstrap.min.js"></script>

And the code to create a route for that

app.get('/bootstrap.min.js', function(req, res) {
    res.sendFile(__dirname + '/node_modules/bootstrap/dist/bootstrap.min.js');
});

Or, if you want to still delineate routes for scripts with /scripts, you could do this:

<script src="/scripts/bootstrap.min.js"></script>

And the code to create a route for that

app.get('/scripts/bootstrap.min.js', function(req, res) {
    res.sendFile(__dirname + '/node_modules/bootstrap/dist/bootstrap.min.js');
});
jfriend00
  • 683,504
  • 96
  • 985
  • 979
  • 2
    Do I need to copy those manually or is there a Grunt way to do that? I think somehow the best would be to completely copy the bootstrap folder to `/scripts` because that way any dependencies inside the module would be maintained. – Chris Dec 13 '14 at 22:28
  • @phpheini - perhaps this will help: http://stackoverflow.com/questions/18966485/copy-all-files-from-directory-to-another-with-grunt-js-copy and this https://www.npmjs.com/package/grunt-copy – jfriend00 Dec 13 '14 at 22:31
  • Thank you, that helps. However I still have a question: imagine I have a package `Foo` inside my `node_modules` folder and in it there is another folder `node_modules` with all the dependencies `Foo` has. However there is no `dist` folder in the `Foo` folder. So I would basically need to upload everything (including the `node_modules` folder inside `Foo`) which is as you stated not a good idea because we want to hide it. In the end we might have 20 packages each one with a `node_modules` folder in it with all the dependencies the package needs (some of them will probably even be duplicates) – Chris Dec 14 '14 at 01:48
  • Do you really have twenty packages that all have client JS libraries? If so, then just put 20 lines in your grunt script to copy the public JS libraries to one common location. If you really have 20 client-side JS libraries, then you should probably be combining and minimizing them anyway. – jfriend00 Dec 14 '14 at 03:08
  • Just as an example: I am using the package `BigVideo.js` which includes a folder `lib` (with one JS file in it), a folder `CSS` and a folder `node_modules` with about ten dependencies. I would now as you said copy the JS file into my public folder. What happens with all the dependencies though? Do I really need to copy all the node_modules from each package into a public `vendor` folder? – Chris Dec 14 '14 at 11:24
  • This is wrong! In npm 3.8+, dependencies and your own module are all positions in the same directory, where the app.use may be: ```app.use('/scripts', express.static(__dirname + '../node_modules/bootstrap/dist/'));``` – Yin Oct 27 '15 at 08:54
  • @Jake - This is not a wrong answer to the specific question that was asked. The changes to NPM may have changed where NPM puts things now, but that isn't what the original question here was about. The question specifically asked how to get access to things inside the node_modules sub-directory. This is not a generic question about how to expose bootstrap files. The question is how to get access to `./node_modules/bootstrap/dist/bootstrap.min.js` which this answer provides a correct answer to. I will make a notation in the answer to that effect. – jfriend00 May 17 '16 at 22:08
  • Added notes to indicate that newer versions of NPM will have different path names for bootstrap files than indicated in the OP's question and in the answers here, but the concepts of how to solve the problem are the same - you just use the updated path locations. – jfriend00 May 17 '16 at 22:13
  • Something that might be useful to add `__dirname` for me seemed to be too low. However `process.cwd()` gave me one level up in the directory structure which got me to where I wanted. It might be good to include the `process.cwd()` as an alternative. – Ian May 25 '16 at 15:55
  • @Ian - It depends upon where your source files are. You HAVE to know that in order to access them. Once you know that, you can decide how best to reference that location (with an absolute path or with a path relative to some known starting point). I'd be careful about `process.cwd()` because the cwd can be changed by code. `__dirname` is used to reference files that are part of a module and are in the directory the module files are located in or something relative to that. That's what its for. Of course, if your files are located elsewhere, you have to build a path to that other location. – jfriend00 May 25 '16 at 16:13
  • So I've got `\board` which is where I'm launching node from. Within there I've 2 directories `\board\api` and `\board\src`. My express app is `\board\api\app.js` - but `__dirname` there is always the `\board\api` directory. Is there a more reliable way to get the root `\board`? – Ian May 25 '16 at 16:16
  • @Ian - This isn't really the right place to problem solve on your specific issue. If you want help on a specific issue, then you probably should ask your own question and describe your specific question. I don't know enough about your context to know if you should use `path.join(__dirname, "..", "src")` or just use `\board\src` since you know the root directory or use something else. – jfriend00 May 25 '16 at 16:21
  • indeed - that's fine. I thought it might be making this answer more thorough - but I think you're right. – Ian May 25 '16 at 16:30
  • @jfriend00 but what if i don't use express? what if I have a src folder with and index.html that needs to reference these files from the nodel_modules folder. Using gulp – Gavin Wood Apr 25 '17 at 09:55
  • Presumably, though, in a production site I'd want to include scripts from multiple node modules. As I understand this solution, I'd have to set up multiple static routes, one for each module's `dist` directory. Is that correct? – Jacob Ford Jun 30 '17 at 15:23
  • @jfriend00 Could you possibly share a little detail on why "you don't want to expose any of your internal paths for how your server is structured to the outside world"? Is this for security reasons, to avoid sharing information that could make a hacker's task easier? – mpavey Dec 01 '17 at 16:01
  • A couple of other thoughts: 1) Is this compatible with the common server config of looking directly on the filesystem for anything that looks static (ending in `.js`, `.css`, etc.)? I would have thought the request would never hit the application layer (NodeJS, PHP, etc.). 2) Assuming you do configure the server to pass the request to the application in order to find the file in the correct location, what are the performance considerations? Presumably this leads to more load on the server than simply letting the server find the file itself (e.g. by copying it somewhere else). – mpavey Dec 01 '17 at 16:15
  • @mpavey - All this does is tell the server where to look for a group of .js files that you intend to make public. It is your responsibility to be smart and not point it to a place that also includes things you do not intend to make public. If you would rather not serve the file from it's natural location (for whatever reasons you have), then yes you can copy it (and all other resources it might depend upon) to another location and serve if from there. This is meant for client-side .js files that are part of a distributed module and are packaged in a way to do it this way. – jfriend00 Dec 01 '17 at 16:57
  • @jfriend00 Thanks, you prompted me to learn more about Node.js. I had thought it was a runtime engine for server-side JS applications similar to PHP, which sat on top of a separate web server (nginx or Apache). I now see that while that's possible, it can also perform the functions of the server too, and presumably that's what's happening in your example? FWIW, I'm still curious about the reasons why one shouldn't make the `node_modules` folder public. – mpavey Dec 01 '17 at 17:15
  • @mpavey - You would NOT make a whole folder public if it contains things that should not be public. In that case, you would either make only a single file or specific set of files in that folder public by referencing them by name in your route definition or you would copy only the things you intend to be public to a directory where everything is intended to be public and make that directory public. – jfriend00 Dec 01 '17 at 17:18
  • @mpavey - This example is often used for a case where a server-side module that will naturally be installed in the node_modules hierarchy also contains one or more client-side files that are intended to be served to the client (often .JS files, but could also be .HTML or .CSS) and, in doing so, they will organize them safely within their module so they can be served from a single directory. You are correct to be concerned about doing this safely, but done properly it can be done safely from within. – jfriend00 Dec 01 '17 at 17:21
  • @jfriend00 Got it, thanks for the explanation! I think much of my confusion may stem from the fact that I am currently using almost NPM solely for client-side JS, to include third-party libraries. In most cases I bundle it (using Webpack or Browserify) and put the bundle into a separate directory before serving it, but for now at least there are one or two libraries I am including directly, hence my original interest in this question. But in any case, I am guessing the reason for being concerned about exposing the entire `node_modules` directory is that it may contain proprietary code too. – mpavey Dec 01 '17 at 17:27
  • @jfriend00 Does Express check to see if there are multiple files of the same name across the static paths? If not, is there a function that will do that or is there a "best practice" technique for checking-for/avoiding this problem? It seems like it could lead to many hard to diagnose problems (multiple files with the same name). – Robert Oschler Apr 09 '18 at 22:29
  • 1
    @RobertOschler - No, it does not check for dups. `express.static()` sends the first match it finds. Since each `express.static()` statement only creates one possible matching path, there should not be duplicates from one `express.static()` statement. If you have multiple `express.static()` statements that each could have a match, then the first one in your code is the one whose match will be used. Also, you really shouldn't have multiple files with the same name and relative path reachable by `express.static()` statements. Best practice is to not do that. – jfriend00 Apr 09 '18 at 22:49
  • @jfriend00 Thanks. Unfortunately I don't have control of all of that. There are several NPM packages that have "/dist" directories in the "node_modules" directory tree and if there happens to be a file overlap, it will be a problem. I know I could copy selected files to a new directory and even make that part of the build process, but I'm hoping to avoid that as long as I can since it creates other headaches. – Robert Oschler Apr 10 '18 at 14:10
  • 1
    @RobertOschler - You have to be very, very careful about what you provide unregulated access to. In general, you should only point `express.static()` at a directory that ONLY contains public files (include sub-directories). So, I can't really imagine a project where there were lots of `dist` directories being published. That seems very unusual to me. Perhaps you want to make a specific route to a specific file or to 2-3 files. In that case, you are responsible for disambiguating those files. It's not going to do you any good to have four `script.js` files anyway. – jfriend00 Apr 10 '18 at 20:36
  • 1
    @RobertOschler - If you don't make a unique path be required in front of them in order to match, there's just no way for any code to know which one to serve when `/script.js` is requested by the browser. So, the abstract answer is that you should not allow the situation of duplicate filenames to exist because it's not a solvable problem unless you require a unique prefix for each directory of files. Then, duplicate root names are not a problem anyway. For a more specific answer with a detailed recommendation, you would need to post your own question with exact details. – jfriend00 Apr 10 '18 at 20:38
  • @jfriend00 hi, I hit a similar problem and I was wondering can you take a look at my problem? https://stackoverflow.com/questions/75876042/how-do-i-set-script-src-to-include-a-javascript-file-outside-express-st – Qiulang Mar 29 '23 at 11:30
21

I would use the path npm module and then do something like this:

var path = require('path');
app.use('/scripts', express.static(path.join(__dirname, 'node_modules/bootstrap/dist')));

IMPORTANT: we use path.join to make paths joining using system agnostic way, i.e. on windows and unix we have different path separators (/ and \)

Metro Smurf
  • 37,266
  • 20
  • 108
  • 140
Alexander Egorov
  • 593
  • 10
  • 14
  • mark duplicate as above, path.join is just an added measure, but still same solution by adding a static path to the node_modules dir. – blamb Oct 12 '15 at 10:40
  • 2
    "path.join is just an added measure, but still same solution" not the same solution. Strictly speaking it uses system specific joining (keep in mind / and \ separators in windows and unix) – Alexander Egorov Feb 07 '17 at 13:09
  • 4
    OK, i see your point, yes its always good to have system agnostic, i wasnt aware that's what that was. Thanks for pointing that out. It would be wise to add that to the sentence in your answer, instead of just providing code :-), e.g. explain your code. – blamb Feb 07 '17 at 21:06
  • why don't you also use path.join on `node_modules/bootstrap/dist` and instead use `/`? – João Pimentel Ferreira Jan 21 '21 at 11:46
  • You are right but I think the framework will handle this situation :) – Alexander Egorov Jan 22 '21 at 14:25
12

If you want a quick and easy solution (and you have gulp installed).

In my gulpfile.js I run a simple copy paste task that puts any files I might need into ./public/modules/ directory.

gulp.task('modules', function() {
    sources = [
      './node_modules/prismjs/prism.js',
      './node_modules/prismjs/themes/prism-dark.css',
    ]
    gulp.src( sources ).pipe(gulp.dest('./public/modules/'));
});

gulp.task('copy-modules', ['modules']);

The downside to this is that it isn't automated. However, if all you need is a few scripts and styles copied over (and kept in a list), this should do the job.

Jesse
  • 429
  • 6
  • 12
  • 1
    One could add this in their package.json in scripts `'scripts': {'install':'yarn gulp copy-modules'}`, assuming yarn is the package manager and gulp is installed in package. – Todd Feb 27 '19 at 13:15
11

As mentioned by jfriend00 you should not expose your server structure. You could copy your project dependency files to something like public/scripts. You can do this very easily with dep-linker like this:

var DepLinker = require('dep-linker');
DepLinker.copyDependenciesTo('./public/scripts')
// Done
Marcelo Lazaroni
  • 9,819
  • 3
  • 35
  • 41
7

The directory 'node_modules' may not be in current directory, so you should resolve the path dynamically.

var bootstrap_dir = require.resolve('bootstrap')
                           .match(/.*\/node_modules\/[^/]+\//)[0];
app.use('/scripts', express.static(bootstrap_dir + 'dist/'));
Yin
  • 612
  • 7
  • 10
7

This is what I have setup on my express server:

// app.js
const path = require('path');
const express = require('express');
const expressApp  = express();
const nm_dependencies = ['bootstrap', 'jquery', 'popper.js']; // keep adding required node_modules to this array.
nm_dependencies.forEach(dep => {
  expressApp.use(`/${dep}`, express.static(path.resolve(`node_modules/${dep}`)));
});

<!-- somewhere inside head tag -->
<link rel="stylesheet" href="bootstrap/dist/css/bootstrap.css" />

<!-- somewhere near ending body tag -->
<script src="jquery/dist/jquery.js" charset="utf-8"></script>
<script src="popper.js/dist/popper.js" charset="utf-8"></script>
<script src="bootstrap/dist/js/bootstrap.js" charset="utf-8"></script>

Good Luck...

Aakash
  • 21,375
  • 7
  • 100
  • 81
5

I didn't find any clean solutions (I don't want to expose the source of all my node_modules) so I just wrote a Powershell script to copy them:

$deps = "leaflet", "leaflet-search", "material-components-web"

foreach ($dep in $deps) {
    Copy-Item "node_modules/$dep/dist" "static/$dep" -Recurse
}
Timmmm
  • 88,195
  • 71
  • 364
  • 509
3

I want to update this question with an easier solution. Create a symbolic link to node_modules.

The easiest way to grant public access to node_modules is to create a symbolic link pointing to your node_modules from within your public directory. The symlink will make it as if the files exist wherever the link is created.

For example, if the node server has code for serving static files

app.use(serveStatic(path.join(__dirname, 'dist')));

and __dirname refers to /path/to/app so that your static files are served from /path/to/app/dist

and node_modules is at /path/to/app/node_modules, then create a symlink like this on mac/linux:

ln -s /path/to/app/node_modules /path/to/app/dist/node_modules

or like this on windows:

mklink /path/to/app/node_modules /path/to/app/dist/node_modules

Now a get request for:

node_modules/some/path

will receive a response with the file at 

/path/to/app/dist/node_modules/some/path

which is really the file at 

/path/to/app/node_modules/some/path

If your directory at /path/to/app/dist is not a safe location, perhaps because of interference from a build process with gulp or grunt, then you could add a separate directory for the link and add a new serveStatic call such as:

ln -s /path/to/app/node_modules /path/to/app/newDirectoryName/node_modules

and in node add:

app.use(serveStatic(path.join(__dirname, 'newDirectoryName')));
curtwphillips
  • 5,441
  • 1
  • 20
  • 19
  • 2
    But then you have moved your app to a different path and the symlink is invalid. Or you want to run your app on a different machine (test / prod), you'll need to create it again. Any contributor to your app must do the same. It adds some maintanace over the above solutions. – mati.o Sep 19 '16 at 19:19
  • @mati.o What about relative symbolic links? I like this solution, but only with relative symlinks. – Lukáš Bednařík Oct 30 '16 at 19:50
1

I did the below changes to AUTO-INCLUDE the files in the index html. So that when you add a file in the folder it will automatically be picked up from the folder, without you having to include the file in index.html

//// THIS WORKS FOR ME 
///// in app.js or server.js

var app = express();

app.use("/", express.static(__dirname));
var fs = require("fs"),

function getFiles (dir, files_){
    files_ = files_ || [];
    var files = fs.readdirSync(dir);
    for (var i in files){
        var name = dir + '/' + files[i];
        if (fs.statSync(name).isDirectory()){
            getFiles(name, files_);
        } else {
            files_.push(name);
        }
    }
    return files_;
}
//// send the files in js folder as variable/array 
ejs = require('ejs');

res.render('index', {
    'something':'something'...........
    jsfiles: jsfiles,
});

///--------------------------------------------------

///////// in views/index.ejs --- the below code will list the files in index.ejs

<% for(var i=0; i < jsfiles.length; i++) { %>
   <script src="<%= jsfiles[i] %>"></script>
<% } %>
SuperNova
  • 25,512
  • 7
  • 93
  • 64
0

To use multiple files from node_modules in html, the best way I've found is to put them to an array and then loop on them to make them visible for web clients, for example to use filepond modules from node_modules:

const filePondModules = ['filepond-plugin-file-encode', 'filepond-plugin-image-preview', 'filepond-plugin-image-resize', 'filepond']
filePondModules.forEach(currentModule => {
    let module_dir = require.resolve(currentModule)
                           .match(/.*\/node_modules\/[^/]+\//)[0];
    app.use('/' + currentModule, express.static(module_dir + 'dist/'));
})

And then in the html (or layout) file, just call them like this :

    <link rel="stylesheet" href="/filepond/filepond.css">
    <link rel="stylesheet" href="/filepond-plugin-image-preview/filepond-plugin-image-preview.css">
...
    <script src="/filepond-plugin-image-preview/filepond-plugin-image-preview.js" ></script>
    <script src="/filepond-plugin-file-encode/filepond-plugin-file-encode.js"></script>
    <script src="/filepond-plugin-image-resize/filepond-plugin-image-resize.js"></script>
    <script src="/filepond/filepond.js"></script>
Mourad BAHLOUL
  • 1
  • 1
0

If you are linking to many files, create a whitelist, and then use sendFile():

app.get('/npm/:pkg/:file', (req, res) => {
    const ok = ['jquery','bootstrap','interactjs'];
    if (!ok.includes(req.params.pkg)) res.status(503).send("Not Permitted.");
    res.sendFile(__dirname + `/node_modules/${req.params.pkg}/dist/${req.params.file}`);
    });

For example, You can then safely link to /npm/bootstrap/bootsrap.js, /npm/bootstrap/bootsrap.css, etc.

As an aside, I would love to know if there was a way to whitelist using express.static

SamGoody
  • 13,758
  • 9
  • 81
  • 91
  • it's could be great solution. but there is not standard in libraries. each one can create the file hierarchy different then the other. so there is not option but do it separately. – ofir_aghai Aug 17 '23 at 13:47
0

for make it generic i do like this:
index.js:

app.use('/scripts/bootstrap', express.static(path.join(__dirname, '/node_modules/bootstrap/bootstrap.min.js')));
app.use('/scripts/<pkg_name>', express.static(path.join(__dirname, '/node_modules/<pkg_name>/bootstrap.min.js')));
...

in html:

<script src="/scripts/bootstrap/dist/bootstrap.min.js"></script>
ofir_aghai
  • 3,017
  • 1
  • 37
  • 43