Axios post Large File

Question

I am trying to Post a large JSON into my Database, the JSON has at least 400.000 Objects.

If I cut the file, and try to Post 20.000 Objects everything works just fine, so the problem should be JSON's size.

I've split the JSON into 20 chunks, and my idea is to upload one at a time but i'm struggling to make it work.

This is what I'm using:

var rows = {};
Papa.parse(content, {
    header: false,
    delimiter: '|',
    worker: true,
    encoding: "utf16le",
    dynamicTyping: true,
    skipEmptyLines: true,
    complete: function(results) {
        rows = results.data;
        let obj = []
        for(var i=0; i < rows.length; i++){
            obj.push(rows[i])
        }

        let result = []

        for(let i in obj) {
            let temp = {}
            if(i > 0) {
                temp["id"] = obj[i][0]
                temp["name"] = obj[i][1]
                temp["tel"] = obj[i][2]
                temp["email"] = obj[i][3]
                temp["status"] = obj[i][5]
                
                result.push(temp)
            }
        }
        
        var array1 = result.map((e) => {
            return {
                id: e.id,
                name: e.name,
                email: e.email
            }
        })

        let chunked = []
        let size = 20000;

        Array.from({length: Math.ceil(array1.length / size)}, (val, i) => {
        chunked.push(array1.slice(i * size, i * size + size))
        })

        console.log(chunked); // at this point I have my array divided into chunks of 20000

        axios({
            url: 'url',
            method: 'post',
            data: chunked
          })
          .then(function (response) {
              // your action after success
              console.log(response);
          })
          .catch(function (error) {
             // your action on error successif (error.response) {
            console.log(error);
        
          });

You have divided into 20 chunks i.e. 'chunked' but still sending the entire array, which is the same as sending same number of data as without dividing it. Better loop through your 'chunked' send one by one. — Pramod Mali, Aug 08 '20 at 13:01
Hello, thank you for your answer, I've tried to loop, I just dont know how to send it one by one, No matter what I do I guess im always sendind the entire array at once — Retalhante, Aug 08 '20 at 13:25

score 0 · Answer 1 · answered Aug 08 '20 at 13:43

0

You can send it one by one as follow. Assuming your backend is able to accept the data format.

    for(let i=0;i<chunked.length;i++){
          axios({
            url: 'url',
            method: 'post',
            data: chunked[i]
          })
          .then(function (response) {
              // your action after success
              console.log(response);
          })
          .catch(function (error) {
             // your action on error successif (error.response) {
            console.log(error);
        
          });
     }

Or a modern solution could be to use Promise.all([])

let promiseArray = [];
for(let i=0;i<chunked.length;i++){
    promiseArray.push(axios({
       url: 'url',
       method: 'post',
       data: chunked[i]
    }))
}

Promise.all(promiseArray)
.then((responses) => {
  console.log(responses);   //Will return resolved/rejected promises in an array
})
.catch(error => { 
  console.error(error.response)
});;

You can read more about Promise.all([]) https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all

answered Aug 08 '20 at 13:43

Pramod Mali

1,588
1
17
29

I've tried both, and it manages to upload ~16.000 and then dies : / – Retalhante Aug 08 '20 at 14:01
What error you are getting? Also, check server logs, maybe backend could be failing. – Pramod Mali Aug 08 '20 at 14:19
I cant see the error because in the terminal it shows me the entire JSON, therefor I think there might be a problem with the api, i am sending you the code so you can see if there is something wrong with it. – Retalhante Aug 08 '20 at 15:15
I also got this error in phpmyadmin - mysqli_real_connect(): (HY000/2002) – Retalhante Aug 08 '20 at 15:23
For MySQL error there is similar question on StackOverflow that might help. https://stackoverflow.com/questions/41881123/mysqli-real-connect-hy000-2002-no-such-file-or-directory – Pramod Mali Aug 08 '20 at 15:32

Axios post Large File

1 Answers1