Splitting a File into Chunks with Javascript

Question

I'm trying to take a single file object and split it into chunks by a specified chunk size. In my example, trying to split a single file into 1MB chunks. So I figure out how many chunks it would take, then I'm trying to slice the file starting from the 'offset' (current chunk I'm on * chunk size), and slicing off a chunk size. My first slice comes out properly at 1MB but my subsequent slices turn out to 0, any ideas why? Have a working codepen here:

http://codepen.io/ngalluzzo/pen/VvpYKz?editors=001[1]

var file = $('#uploadFile')[0].files[0];
  var chunkSize = 1024 * 1024;
  var fileSize = file.size;
  var chunks = Math.ceil(file.size/chunkSize,chunkSize);
  var chunk = 0;

  console.log('file size..',fileSize);
  console.log('chunks...',chunks);

  while (chunk <= chunks) {
      var offset = chunk*chunkSize;
      console.log('current chunk..', chunk);
      console.log('offset...', chunk*chunkSize);
      console.log('file blob from offset...', offset)
      console.log(file.slice(offset,chunkSize));
      chunk++;
  }

Your codepen seems to be working for me. One difference between your codepen and the code above is that your codepen correctly says: `file.slice(offset, offset + chunkSize)` instead of `file.slice(offset, chunkSize)`. Also you're producing an empty blob at the end, because your while condition should be `while (chunk < chunks)`. — Noah Freitas, Oct 01 '15 at 23:20
Yeah I updated the codepen with the solution below, working great. Thanks for looking! — Nico Galluzzo, Oct 01 '15 at 23:25

score 21 · Accepted Answer · answered Oct 01 '15 at 23:14

21

Was slicing off the wrong ends:

console.log(file.slice(offset,chunkSize));

should have been

console.log(file.slice(offset,offset+chunkSize));

answered Oct 01 '15 at 23:14

Nico Galluzzo

485
1
3
8

score 6 · Answer 2 · edited Oct 06 '20 at 15:44

6

Use The function below to split a big file into multiple chunks. I have used it with react, it's works.

  createChunks = (file,cSize/* cSize should be byte 1024*1 = 1KB */) => {
   let startPointer = 0;
   let endPointer = file.size;
   let chunks = [];
   while(startPointer<endPointer){
    let newStartPointer = startPointer+cSize;
    chunks.push(file.slice(startPointer,newStartPointer));
    startPointer = newStartPointer;
   }
   return chunks;
  }

edited Oct 06 '20 at 15:44

jps

20,041
15
75
79

answered Oct 06 '20 at 15:07

Muhotasim Fuad

61
1
1

1

Hi Fuad, am currently facing the issue of uploading large files, please can you throw more light on how I can split big file into multiple and upload to an endpoint that accept chunks upload – Gabriel Geelario Jul 14 '21 at 21:24
wow very clean solution good day sir :) – Un1xCr3w Apr 27 '22 at 11:57
Will this actually work if there are _hundreds of thousands_ or even _millions_ of 1kB chunks? It's hard to fathom that a contemporary browser would accommodate JS trying to allocate so much dynamic/heap memory at once? – NetXpert Jun 18 '22 at 20:11

score 0 · Answer 3 · answered Aug 17 '22 at 11:46

Inheriting what was above, I write expanded, dir in folder named "in" and then split, cut files and name them by increasing number of files, it will output in "out" folder, name.z00, name.z01, name.z02...

const fs = require('fs')

function numberName (numberFile,index){
    if(numberFile<10){
        return "0"+index
    }
    if(numberFile<100){
        if(index<10)
            return "00"+index
        return "0"+index
    }
}

var createChunks = (file,cSize/* cSize should be byte 1024*1 = 1KB */) => {
    let startPointer = 0;
    let endPointer = file.length;
    let chunks = [];
    while(startPointer<endPointer){
    let newStartPointer = startPointer+cSize;
    chunks.push(file.slice(startPointer,newStartPointer));
    startPointer = newStartPointer;
    }
    return chunks;
}
function splitDirFile(cSize){
    var list = fs.readdirSync("./in");
    console.log(list)
    list.map(fileName=>{
        var chunks = createChunks(fs.readFileSync("./in/"+fileName),cSize)
        //console.log(list.length);
        var name = fileName.slice(0,fileName.length-4)
        chunks.map((buff,i)=>{
            fs.writeFileSync("./out/"+name+".z"+numberName(chunks.length,i),buff)
        })
    })

}

splitDirFile(1024*1024*10)

Splitting a File into Chunks with Javascript

3 Answers3

Linked