5

The Problem

I have a Node.js end-point that properly triggers an arbitrarily-large file download when accessed using the following:

response.setHeader('Content-disposition', 'attachment; filename=' + fileName);
response.set('Content-Type', 'text/csv');
response.status(200);
result.pipe(response);

where result is a transform stream, and response is an Express object.

This works fine when directly accessing the end-point in Chrome, Firefox, Internet Explorer, etc. The problem arises from trying to hit the end-point when token-based authentication is enabled.

From a user's perspective, when they click on a button, a file is downloaded.

How do I make the button hit this end-point with the correct authentication token in the request's header and cause a file to be downloaded?

Brainstorm of Some Possible Approaches

  1. When the user clicks on the button, it fires an action that's handled by the redux-api-middleware, which makes the GET request to the end-point (with the authentication token automatically included in the request). This middleware saves the response in a variable, which is picked up by a React component. In this React component, if the browser being used (i.e. Chrome and Opera) supports streams, response.body will exist so you can do something like the following.

    if (response.body) {
        const csvReader = response.body.getReader();
        const textDecoder = new TextDecoder();
        const processCsvRow = (csvRow) => {
            if (csvRow.done) {
                console.log('Finished downloading file.');
                return Promise.resolve();
            }
    
            // Write to new file for user to download
            console.log(textDecoder.decode(csvRow.value, {stream: true}));
    
            return csvReader.read().then(processCsvRow);
        };
    
        csvReader.read().then(processCsvRow);
    }
    

    With this approach, I'd need to investigate how to handle the data if it's being received by a non-stream-supporting browser.

  2. When the user clicks on the button, it fires an action that saves the end-point into the Redux store by the reducer, which triggers a React component to create an anchor tag that is automatically clicked.

    const link = document.createElement('a');
    document.body.appendChild(link); // Firefox requires the link to be in the body
    link.href = endPoint;
    link.target = '_blank';
    link.click();
    document.body.removeChild(link); // Remove the link when done
    

    where endPoint is the end-point that responds with the file.

    This approach works when authentication is disabled. When authentication is re-enabled, the authentication token must somehow be injected into the anchor tag's request header.

  3. Similar to #2, look into simulating an anchor-click by constructing an HTTP request with the authentication token built in.
  4. Combining elements from #1 and #2, when the user clicks on the button, an action is fired that sends a GET request to the end-point, which returns a second temporary unsecured end-point that returns the actual file response. Pass this temporary unsecured end-point to the anchor tag in #2.
  5. Redirect the response to a new tab/window somehow.
  6. Pass the authentication token as a URL parameter (this is a security concern), then use #2.
  7. Create a new end-point that generates and returns a new temporary one-time-use download-only token. Use this new end-point to get a temporary token to pass as a URL parameter into the original end-point via #2.
David
  • 167
  • 3
  • 11
  • You can use `fetch()` to send authentication header, substitute `.text()` for `.getReader()`, `.read()`. What is the `Content-Length` of the response that is offered for download? – guest271314 Mar 07 '17 at 00:14
  • Does Question have two parts? – guest271314 Mar 07 '17 at 01:18
  • The `content-length` of the response is not defined as the response is being streamed from the database through a Node.js `transform stream`. – David Mar 07 '17 at 15:50
  • Still not sure what issue is? Why do you not use single `POST` with authentication token, then offer client download after calling `.text()` and constructing file to download? You could also use `XMLHttpRequest()` with `responseType` set to `"blob"` or `"arraybuffer"`, process response, offer file for download. – guest271314 Mar 07 '17 at 21:52
  • The issue with XMLHttpRequest() is described in http://blog.bguiz.com/2014/07/03/file-download-with-http-request-header/ – David Mar 07 '17 at 22:15
  • Began composing an Answer yesterday, though not certain what issue is? Authentication? Make `POST` request with `fetch()` with authentication included, get response, call `.text()` or `.blob()`, create `` element, set `href` to `Blob URL`, call `.click()`. – guest271314 Mar 07 '17 at 22:16
  • Yeah, token-based authentication. – David Mar 07 '17 at 22:19
  • Blobs have a maximum size restriction as they need to fit in memory. I'm dealing with arbitrarily-large file sizes. – David Mar 07 '17 at 22:20
  • Have you tried using `StreamSaver.js`? For token authentication see [javascript - Using an authorization header with Fetch in React Native - Stack Overflow](http://stackoverflow.com/questions/30203044/using-an-authorization-header-with-fetch-in-react-native) at http://stackoverflow.com/a/35780539/; for arbitrarily large files see [Download large data stream (> 1Gb) using javascript](http://stackoverflow.com/questions/42614880/download-large-data-stream-1gb-using-javascript#comment72361323_42614880). Putting those two approaches together should resolve Question. – guest271314 Mar 07 '17 at 22:23
  • A typical size would be several gigabytes. I've tried StreamSaver, but haven't gotten it to work due to the error "ReferenceError: WritableStream is not defined(…)". – David Mar 07 '17 at 22:33
  • Have you filed an issue at the repo? Perhaps @Brian will post their working implementation. There is a streams polyfill available at github. What is the current working implementation of `javascript` at Question? – guest271314 Mar 07 '17 at 22:34
  • I haven't filed an issue, but it might be covered by https://github.com/jimmywarting/StreamSaver.js/issues/43. I'm currently using ES6 with babel-polyfill 6.3.14. – David Mar 07 '17 at 23:01
  • You could add your own experience to the open issue. What is current working implementation? Or are you trying to develop a working implementation? Have you pinged @Brian to ask how they were able to download 1GB+ file? – guest271314 Mar 07 '17 at 23:03
  • My current working implementation uses Blob, createObjectURL(), and creating an anchor element, which has the memory limit issue discussed above. I can download 1 GB+ files if I disable authentication and hit the end-point directly. The back-end has already been updated to work with Node.js streams. – David Mar 07 '17 at 23:06
  • _"I can download 1 GB+ files if I disable authentication and hit the end-point directly"_ Have you tried using authentication approach at linked Question to meet that portion of requirement? – guest271314 Mar 07 '17 at 23:10
  • Which linked question? – David Mar 07 '17 at 23:12
  • http://stackoverflow.com/a/35780539/ – guest271314 Mar 07 '17 at 23:12
  • I believe `fetch` returns the response in a `Promise`, which would be equivalent to what I have right now with Approach #1 above. In Approach #1, the authentication token is already included in the `GET` request, and the response is available in a variable. – David Mar 07 '17 at 23:20
  • Then what is issue? – guest271314 Mar 07 '17 at 23:21
  • For Approach #1, it's how to handle the response. Ideally, what would happen is: when we get the response, redirect the response to a new window, which causes the file-download to automatically be triggered as if we hit the end-point directly. – David Mar 07 '17 at 23:23
  • Use `response.blob()`, `URL.createObjectURL()`, though within `.then()` create another `Blob` with `response.blob()` as parameter with `type` set to `"application/octet-stream"`, then call `window.open()` with new `Blob` as parameter, see [How to download a file without using element with download attribute or a server?](http://stackoverflow.com/questions/38711803/how-to-download-a-file-without-using-a-element-with-download-attribute-or-a-se) – guest271314 Mar 07 '17 at 23:32
  • Or use `
    ` approach described by @LeoFarmer
    – guest271314 Mar 07 '17 at 23:41
  • Doesn't the form approach require a URL or complete file? If I passed window.open() my end-point, the authentication token would not be automatically included. – David Mar 07 '17 at 23:48
  • Once the response has been processed with `.getReader()`, `.read()` or `.blob()` you can generate a `Blob URL` to populate `action` property of `
    `. That is, pass authentication token, get response, process response, create `Blob URL` or `data URI`, use `window.open()` or `
    ` approach to offer download of file.
    – guest271314 Mar 07 '17 at 23:50
  • `response.blob().then( (result) => { console.log(result); } );` fails because "net::ERR_INCOMPLETE_CHUNKED_ENCODING" and "customers:1 Uncaught (in promise) TypeError: Failed to fetch at TypeError (native)" – David Mar 07 '17 at 23:59
  • Then use `.getReader()`, `.read()`, or adjust response. See also [How to solve Uncaught RangeError when download large size json](http://stackoverflow.com/questions/39959467/how-to-solve-uncaught-rangeerror-when-download-large-size-json/40017582?s=6|1.1830#40017582) – guest271314 Mar 08 '17 at 00:01
  • `.getReader()` and `.read()` -> `Blob` -> `URL.createObjectURL()` -> `window.open()`? Wouldn't that create many `Blob`s and cause many windows/tabs to open? Otherwise, if this creates one large `Blob`, we would run out of memory again. – David Mar 08 '17 at 00:08
  • No, `.read()` `.value` is `Uint8Array`, at `reader.closed` pass accumulated `Uint8Array`s to single `Blob`. – guest271314 Mar 08 '17 at 00:11
  • I'm using a single `Blob` in my current working implementation. However, it's running out of memory. – David Mar 08 '17 at 00:12
  • Became aware at linked Question that computer used affects what occurs. Initially tried at computer having low disk space and `RAM`, where total process took 4:20 and partially froze UI; tried at different computer without those limitations and download was prompt, was unable to notice difference between 189MB download and download of file having less total size. Conclusion was performance as to download of file was directly correlative to computer available disk space and `RAM`. – guest271314 Mar 08 '17 at 00:15
  • Browse the comments as last linked Answer. – guest271314 Mar 08 '17 at 00:21
  • I'll give your linked answer a try tomorrow. Thanks for your help. – David Mar 08 '17 at 00:22
  • I just read through your example (http://stackoverflow.com/questions/39959467/how-to-solve-uncaught-rangeerror-when-download-large-size-json/40017582#40017582), and it doesn't look like it would work for me because `json` would become too large to fit into memory. In addition, `Blob` is immutable, the input to `Blob` seems like it would have to exist in memory before the creation of the `Blob`. – David Mar 08 '17 at 19:02
  • I've decided to go with Approach #7 for now. – David Mar 08 '17 at 19:33

1 Answers1

1

I've decided to go with Approach #7 for now.

David
  • 167
  • 3
  • 11