6

I'm trying to use Node.js to get a file from a remote URL then send it to another server (using an API provided by each of the two websites). I already managed to successfully upload a local file to the remote server using fs.createReadStream("file.png"). Remote files seem to be a different story however: I can't simply put "https://website.com/file.png" in there, I need an equivalent for createReadStream for remote files.

Obviously I could use a separate command to download the file locally and upload it using createReadStream then delete the local file, but I want my code to be efficient and not rely on manually downloading temporary files, plus this is a good learning experience. I'd thus like to know the simplest way to pipe files as streams between two different servers.

Also I would like to avoid using extra dependencies if possible, as I'm writing a simple script which I'd rather not make reliant on too many npm packages. I rely on require("https") and require("fs") primarily. I'm curious if this can be achieved through a simple https.get() call.

MirceaKitsune
  • 777
  • 1
  • 5
  • 14
  • Why do you think you _need_ an `createReadStream`? Your HTTP body should _already be a stream_ which can be used the same way as a stream created from `createReadStream`. – tkausl Jan 31 '19 at 15:50
  • I didn't quite understand how that works. So if I use `https.get` am I able to pipe in the `response.on("data")` object? I thought I need to use a special function to download and format it as stream. If it's as simple as that then it's great! I'll test this soon. – MirceaKitsune Jan 31 '19 at 16:33
  • I can't currently test it and I haven't worked with node's http(s) packages but I'm pretty sure the response object you get passed in your callback is a stream (or behaves like one). – tkausl Jan 31 '19 at 16:59
  • I found a way to use `https.get` to receive the file data into a local buffer. The problem I'm facing now is that I can't convert it to a ReadStream, which is what the API of the target expects to receive. Using console.log on the result of createReadStream with the local file, I found that the object needs to have the following format: `ReadStream { _readableState: ReadableState { ... } }` Like I said, `fs.createReadStream` generates exactly that kind of object. But it does it from a local file. What I need now is a way to do it from a buffer object or binary string in a variable. – MirceaKitsune Jan 31 '19 at 17:50
  • See this post https://stackoverflow.com/questions/32138748/pipe-a-uploaded-file-to-a-remote-server-with-node-ideally-with-same-filename/32139236 – Maayan Hope Aug 29 '21 at 14:50

0 Answers0