Using join() to process an array in batches?

Question

   var j = 0;
   var batchSize = 100;
   var json_left_to_process = json.length;
   while (json_left_to_process != 0) {
       var url_param = "";
       for (var i = j; i < j+batchSize; i++) {
           url_param += json[i].username + ",";
           json_left_to_process--;
       }
       j += batchSize;
       if (json_left_to_process < 100) {
           batchSize = json_left_to_process;
       }
       url_param.substring(0, url_param.lastIndexOf(','));

      //make ajax request
           $.ajax({
                type: "POST",
                url: "/api/1.0/getFollowers.php",
                data: {param: url_param}
           )};
  }

I don't want to use

url_param += json[i].username + ",";

Instead, I want to say

newArray.push(json[i].username)

and then

url_param = newArray.join(',');

But I also want to process up to 100 elements in the array at a time. How can I do so?

EDIT: Sorry, I meant up to 100 elements, and then up to another 100 elements, and then up to another 100 elements, etc etc until you've processed everything.

just do it, where is the problem? 100 elements really shouldnt be any problem. but be aware that pure string concatenation is much faster! — hereandnow78, Aug 30 '13 at 20:23
http://stackoverflow.com/questions/417142/what-is-the-maximum-length-of-a-url-in-different-browsers — Lorenz Meyer, Aug 30 '13 at 20:36
Right, so would a post work? See my edit to the question please. — dtgee, Aug 30 '13 at 20:43

ᴅᴇɴɪʀᴏ · Accepted Answer · 2013-08-30T21:02:37.017

3

If you mean you want to join 100 and then another 100 after then you can do it like this.

 newArray.slice(0, 100).join(',');

And then after that

 newArray.slice(100, 200).join(',');

You could create a function like this to automate it.

var array = [1...1000], // The array
    start = 0,          // The index to start at
    step  = 100;        // How many to get at a time

var getBatch = function() {
    var result = array.slice(start, start + step).join(',');
    start += step;  // Increment start
    return result;
};

getBatch(); // === 1, 2, 3, 4, ... 100
getBatch(); // === 100, 101, 102, ... 200

Documentation: https://developer.mozilla.org

edited Aug 30 '13 at 21:02

answered Aug 30 '13 at 20:22

ᴅᴇɴɪʀᴏ

46
3

The automation is really cool, except the problem is what if I have something like 777 elements? – dtgee Aug 30 '13 at 20:40
Then it the last batch will return `77` and won't throw an error if that's what you're thinking. – ᴅᴇɴɪʀᴏ Aug 30 '13 at 20:41
Oh...? So I guess the slice() function handles error checking? – dtgee Aug 30 '13 at 20:42
what do you mean, if i have this array `[1, 2, 3, 4, 5]` and do `array.slice(1, 10000);` it will return `[2, 3, 4, 5]` – ᴅᴇɴɪʀᴏ Aug 30 '13 at 20:43
Ah, I see. So yea I was just asking if the slice() function makes sure that it won't do anything that is out of bounds – dtgee Aug 30 '13 at 20:46
Also, I noticed that you should do the `start += step` part after the join and slice, because you'll never get indices 0..100 – dtgee Aug 30 '13 at 20:50

hereandnow78 · Answer 2 · 2013-08-30T20:39:11.493

1

i would use a modulo.

var i, start = 0;
var newArray = [];
for (i = 0; i <= json.length; i++) {

   // push whatever you want to newArray

   // if i is dividable by 100 slice that part out and join it
   if (i % 100 === 0) {
      console.log(newArray.slice(start, i).join(','));
      start = i;
   }
}

edited Aug 30 '13 at 20:39

answered Aug 30 '13 at 20:31

hereandnow78

14,094
8
42
48

Using join() to process an array in batches?

2 Answers2