Node: Check if URL is safe to request

Question

I am collecting URLs from visitors of my site that I then (on-demand) request from my backend. However, I have gotten stuck on the validation of the URL.

A URL is safe to request when:

It is a valid URL
It is unambiguous (in RFC 1918 terms, it is public)

What I have found so far:

The URL interface (validates URLs)
The ipaddr.js library (validates IPs)
The is-valid-domain library (Looks to be broken, since it returns true for 256.256.256.256)

However, none of these seems to do what I need it to - not even a combination of them does.

Is there anything I have overlooked?

I found a interesting Post here, hope it helps you! https://stackoverflow.com/a/30931191/10648655 — Antax, Feb 15 '19 at 09:20
@Antax Unfortunately, that doesn't check whether the URL is public or not. Also, it says that 'http://256.256.256.256' is a valid url. — Kryštof Píštěk, Feb 15 '19 at 09:26
check that link: https://www.quora.com/How-do-I-identify-when-an-IP-address-is-private-or-public — N00b, Feb 15 '19 at 10:42
@N00b I know how to identify a private IP address by hand (I even referenced RFC 1918), but the link does not contain any code. — Kryštof Píštěk, Feb 15 '19 at 10:43
@N00b I get a URL (https://123.123.123.123/api/x) and I need to check whether is is OK to give it an HTTP request. — Kryštof Píštěk, Feb 15 '19 at 11:02
Sorry cannot really help with nodejs, but if you want use also php, there is those functions to check ip and url. Change language for that if there is no reason to make code with nodejs. — N00b, Feb 15 '19 at 11:08

score 1 · Accepted Answer · answered Feb 15 '19 at 13:05

I solved my own problem. I used:

The URL interface for URL validation
The hostname-is-private NPM module

First, I validate the input with new URL(input), and then I use require('hostname-is-private').isPrivate (this is an async function) on that URL.hostname. This also works on IP addresses (even if the name does not suggest that).

The code looks something like this:

let isPrivate = require('hostname-is-private').isPrivate;

function validateURL(urlstr, cb) {
  try {
    let url = new URL(urlstr);
    isPrivate(url.hostname, (err, res) => {
      if (err) {cb(false)}
      cb(!res);
    })
  } catch (e) {
    return cb(false);
  }
}

Node: Check if URL is safe to request

1 Answers1