3

I'm streaming data from a file upload a la javascript FileReader - parsing long file in chunks in browser javascript.

Basically its reading a file in chunks by doing:

var r = new FileReader()
var blob = file.slice(curPosition, curPosition+chunkSize); 
r.onload = function(e) {
  emit('data', e.target.result)
}
r.readAsDataURL(blob)

It does this as many times as necessary to read the whole file.

But when I do this, I get the data-url preamble each time, then a bunch of base64 data. Even when I remove the preamble, simple string concatenation doesn't work (the file ends up corrupted).

How can you concatenate two data urls? Or really, I think this question boils down to: how do you concatenate two base64 strings?

Alternatively, how do you build a data url in pure javascript?

B T
  • 57,525
  • 34
  • 189
  • 207
  • Can you please give us more details ? – Nihar Sarkar Aug 17 '16 at 06:07
  • @A.J. Accidentally submitted it guys, keep yer dern pernts ern. – B T Aug 17 '16 at 06:07
  • _"I'm streaming data from a file upload"_ What do you mean by "streaming data"? To which element? Why would there be more than single `data URI`? Are you trying to render multiple files sequentially at an `HTMLMediaElement`? _"I get its preamble"_ What is "preamble"? – guest271314 Aug 17 '16 at 06:12
  • @guest271314 Please read the top answer from the link I included in my question. The answer slices up a single file into bite sized chunks so your browser doesn't lock up on larger files. This is uploading it directly into your browser page's memory, not a server. – B T Aug 17 '16 at 06:15
  • Yes, read linked Question. Not certain how linked Question is related to concatenating `data URL`s? What do you mean by "streaming data"? Can you include `html`, `javascript` that you have tried at Question?, create a stacksnippets, jsfiddle http://jsfiddle.net or plnkr http://plnkr.co to demonstrate? – guest271314 Aug 17 '16 at 06:17
  • @guest271314 I added some more explanation. By streaming data, I mean processing a file in chunks. – B T Aug 17 '16 at 06:22
  • `javascript` at Question does not pass `file` to `r.readAsDataURL()`? What is `file`? `r.readAsDataURL()` would return `data URL` of that portion of `file`, resulting in `chunkSize` `data URL`'s – guest271314 Aug 17 '16 at 06:25
  • You could probably use `.readAsArrayBuffer()` to create "slices" or "chunks" of files which combined are single file. What are you trying to stream? Not certain it is possible to "stream" a `data URL`. Is `file` a text, audio or video file? – guest271314 Aug 17 '16 at 06:30
  • @guest271314 Sorry, typo, its supposed to pass `blob` to `readAsDataURL`. `file` is a `File` object. Shouldn't matter what's in the file. – B T Aug 17 '16 at 06:36
  • What do you mean by "stream"? Stream text, audio, video, bytes? Does a `DOM` element render the stream, or output from `FileReader`? What is purpose of using `.readAsDataURL()` instead of `.readAsArrayBuffer()`? – guest271314 Aug 17 '16 at 06:38
  • @guest271314 I already answered that.. Processing the file in chunks. Its an arbitrary file. You're looking for meaning where there is none. I'm *testing* by generating a data URL for an image and displaying that image in the dom, but that's not the only usecase. – B T Aug 17 '16 at 06:52
  • You can use `.readAsArrayBuffer()`, `load`, `loadend` , `progress` events of `FileReader()` to read, process, stream each byte of `ArrayBuffer` of `File` object returned by `.readAsArrayBuffer()` to achieve expected result – guest271314 Aug 17 '16 at 09:03

3 Answers3

2

If you want to concatenate two base64-encoded strings, try this:

var data1 = ..., data2 = ...; // base64 encoded
var bothData = atob(data1) + atob(data2); // binary string
var bothData64 = btoa(bothData); // base64 encoded

atob converts base64 into plain old (binary) strings, which can be concatenated as usual. If you want to convert it back into base64, use btoa. See the MDN article.

If you want to stream a file into a data: URL, you could also use .readAsArrayBuffer(), concatenate the results (see this), and convert the ArrayBuffer to base64 (see this).

Community
  • 1
  • 1
qxz
  • 3,814
  • 1
  • 14
  • 29
  • `Failed to execute 'atob' on 'Window': The string to be decoded is not correctly encoded` What would be result of `bothData` ? – guest271314 Aug 17 '16 at 06:26
  • 1
    `data1` and `data2` must be just the base64 part of the `data:` URL, e.g., without the `data:foo/bar;base64,` – qxz Aug 17 '16 at 06:31
  • This sounds awfully inefficient. Remember I'm processing large files (think gigabytes). – B T Aug 17 '16 at 06:34
  • 2
    Unfortunately, you have to decode the base64 to concatenate it. See [this answer](http://stackoverflow.com/a/29080108/1848578). You can save the re-encoding for the very end, or skip it if you just need the binary data. – qxz Aug 17 '16 at 06:40
  • Also the bottleneck will almost certainly be the network connection, and not some meager client processing like this. – qxz Aug 17 '16 at 06:42
  • I'm not sending the data over a network connection, just directly onto the browser page. Bottleneck or not, it would be simpler and more efficient to create a full ArrayBuffer and generate a data URL from it all at once than to do this. I'm 99% sure you can concatenate base64 strings as long as you handle the padding characters which only appear at the end. In my case, this will be at most 3 characters out of 64000. The note about the padding characters is helpful tho, thanks! – B T Aug 17 '16 at 06:59
  • 1
    You can't just play with the padding characters; one byte can be encoded in multiple base64 chars. See [this jsFiddle](https://jsfiddle.net/prnav1Lq/) and [this page](https://blogs.oracle.com/rammenon/entry/base64_explained) on how base64 works. I've also edited my question. – qxz Aug 17 '16 at 07:26
1

in nodejs 

    function joinBase64Strings(base64Str1, base64Str2) {
      const bothData = Buffer.from(base64Str1, 'base64').toString('binary') 
            + Buffer.from(base64Str2, 'base64').toString('binary');
      const joinedBase64Result = Buffer.from(bothData.toString(), 'binary').toString('base64');
      console.log('joinedBase64Result', joinedBase64Result);
      return joinedBase64Result;
    }
аlex
  • 5,426
  • 1
  • 29
  • 38
  • I just made this a little more clear, but my question is specifically about browser-side javascript – B T Oct 09 '18 at 21:58
  • @BT It bad practice do convertation in browser, do it on backend – аlex Oct 10 '18 at 09:43
  • Convertation? What do you mean by that? I'm streaming data from the frontend to the backend, so its literally impossible to "do it on [the] backend". – B T Oct 10 '18 at 22:25
0

You can use .readAsArrayBuffer(), progress, load, loadend events of FileReader to create a new Blob or File object, or to stream each byte of data returned as .result by .readAsArrayBuffer(/* Blob or File object */)

guest271314
  • 1
  • 15
  • 104
  • 177