0

I'm uploading data to a web-service by iterating through a precompiled local database of about 150000 images with metadata. To prevent node from throwing Error: EMFILE, too many open files, I'm batching the upload into small tasks of about 5000 images at a time.

To achieve this, I've written a script that accepts an upper and lower bounds:

node upload.js bot_bound top_bound;

In the code, there is a comparator that looks like this:

for(i in parsed_data){
    if(i >= bot_bound && i < top_bound){
        //...process the data
    }
}

it's the (i >= bot_bound && i < top_bound) that's really giving me trouble. It's behaving ...weird. For the first few batches, with bounds below like 40000, it was skipping over certain values. As I tried higher and higher batches, it skipped over more values until it skipped over all of them. by the time my bounding range was [95000, 100000) it was returning false for all values.

So, to troubleshoot, I've been throwing this log line in there:

(${i} > ${bot_range}) && (${i} < ${top_range}) :: ${(i >= bot_range && i< top_range)}

It outputs this at runtime:

...
(94999 >= 95000) && (94999 < 100000) :: false
(95000 >= 95000) && (95000 < 100000) :: false
(95001 >= 95000) && (95001 < 100000) :: false
(95002 >= 95000) && (95002 < 100000) :: false
(95003 >= 95000) && (95003 < 100000) :: false
...

However, when I enter a repl, it works as expected:

> var bot_range = 95000 
undefined
> var top_range = 100000
undefined
> var i = 95001
undefined
> `(${i} > ${bot_range}) && (${i} < ${top_range}) :: ${(i >= bot_range && i< top_range)}`
'(95001 > 95000) && (95001 < 100000) :: true'

So, what's going on here? I'm stumped.

pgulley
  • 21
  • 3
  • You are using a for in loop over an array? Why are you looping over all the indexes and not just the range? – epascarello May 23 '19 at 19:54
  • 6
    Check that everything you're dealing with is a number, and not a string or something else – CertainPerformance May 23 '19 at 19:54
  • Javascript's `for(i in...` syntax can do different things depending on the datatype of the variable you're iterating over. In some cases, it might be better to explicitly use `for (var i = 0; i < parsed_data.length; i++)` if you know that parsed_data is an array. – jwatts1980 May 23 '19 at 19:58
  • `typeof(i)` is going to be `string`. So that might be the problem. – Krisztián Balla May 23 '19 at 20:03
  • @CertainPerformance - oof. it's the little things, isn't it. That did it. I just added a `parseInt()` to where I read in the argv values, and now it works as expected. I'm still not clear on what the prior issue was exactly- like, why it worked for lower bounding values- but everything is working again. Thank you – pgulley May 23 '19 at 20:03
  • 1
    @pgulley that code is still horrible for performance, makes no sense to loop over everything.... Just loop over the indexes you need....`for( i = bot_bound; i < top_bound, i++){}` – epascarello May 23 '19 at 20:04
  • 1
    Why not use `for (let i = bot_bound; i < top_bound; ++i)`? – Krisztián Balla May 23 '19 at 20:05
  • See https://stackoverflow.com/questions/500504/why-is-using-for-in-with-array-iteration-such-a-bad-idea?lq=1 – Barmar May 23 '19 at 20:24

0 Answers0