0

I have a file named 'images.txt' which contains urls to images like this

http://www.foobar.com/assets/foobar/abc.jpg                                                
http://www.foobar.com/assets/foobar/xyz.jpg                                          
http://www.foobar.com/assets/foobar/aaa.jpg                                     
http://www.foobar.com/assets/foobar/bbb.jpg                    
http://www.foobar.com/assets/foobar/rrr.jpg

I am trying to iterate through the files and download the images one by one, but for some reason I can only download the last image rrr.jpg and it only has 0kb ..which is incorrect.

I am using the below nodejs code.

var wget = require('node-wget'); fs = require('fs')

fs.readFile(__dirname + '\\images.txt', 'utf8', function (err,data) {
  if (err) {
    return console.log(err);
  }else{
    wget(data);
  }
});

I have both nodejs and wget installed and working without any issue.

UPDATE

I have used this script

var lineReader = require('readline').createInterface({
  input: require('fs').createReadStream(__dirname + '\\images.txt')
});

lineReader.on('line', function (line) {
 wget(line);
});

It downloads all the images but they seem to be corrupted as all of them can not be opened or deleted. Even though nodejs finished running the script it seems there is something open from nodejs but not closed

Community
  • 1
  • 1
hidar
  • 5,449
  • 15
  • 46
  • 70

1 Answers1

1

it looks like you are not reading the file line by line. try this experiment:

// check.js
const fs = require('fs')

fs.readFile(__dirname + '/check.js', 'utf8', function(err, data) {
  if (err) {
    return console.warn(err)
  } else {
    console.log(`next line: ${data}`)
    // wget(data);
  }
})

(turn the slash around since it looks like you are using windows)

you will only see 'next line' once.

(I replaced the file name with the name of the script, which I called check.js. You will have to replace check.js with either the name of your script or the name of your list of images.)

---re update---

maybe this will work:

// check.js
const fs = require('fs')

fs.readFile(__dirname + '/check.js', 'utf8', function(err, data) {
  if (err) {
    return console.warn(err)
  } else {
    for (let line of data.split('\n')) { // might have to change to \r\n or something for windows
      console.log(`next line: ${line}`)
      // wget(data);
    }

  }
})

wget might not like when you start a second download before a former one is finished

Alex028502
  • 3,486
  • 2
  • 23
  • 50
  • https://stackoverflow.com/questions/6156501/read-a-file-one-line-at-a-time-in-node-js shows how to read line by line I think – Alex028502 Aug 02 '17 at 14:50
  • I only saw one 'next line' but the link from Alex seems to do the job. For some reason nodejs downloaded all file instead of one by one ... all of them are not yet complete but the size is changing... this are big image 5mb+ – hidar Aug 02 '17 at 14:58
  • Hmm... it seems there is another problem. The txt file has 150 image, and now nodejs has downloaded 150 image but they can't be opened or deleted... the script has finished running without errors, bu the files are not can not be opened. – hidar Aug 02 '17 at 15:02
  • @hidar Have you got any solution for that? – edam Jul 31 '19 at 08:41