4

I have a probably rather basic question to using the download.file function in R using the wget option and employing some of the wget extra options, but I just cannot get it to work.

What I want to do: download a local copy of a webpage (actually several webpages, but for now the challenge is to get it to work even with 1).

Challenge: I need the local copy to look exactly like the online version, which also means to include links/ icons, etc.. I found wget to be a good tool for this and I would like to specify some of the extra options, such as --random wait, -p, -r. I found some very helpful tutorials on this, however none of them employed the extra options in R, but rather in wget directly.

So here is the code I have put together for this: 

download.file('https://www.wikipedia.org/', destfile = "wikipage", method = "wget", extra = getOption("--random wait", "-r", "-p"))

which does not work. I suspect there are problems with both, the "wget" method and the specification of the extras.

Can anyone help, it would be much appreciated?

A bonus question: I know that the destfile is supposed to specify a file name for the downloaded document, but is there any way I could specify a folder through a path to which all downloaded files should be saved?

Thank you in advance!

Best Carolin

tk3
  • 990
  • 1
  • 13
  • 18
Carolin
  • 539
  • 2
  • 7
  • 15

1 Answers1

3

You can specify multiple options directly in the extra argument, without getOption().

Further, you can simply include the path to the file where you want to save your downloaded file in the destfile.

download.file('https://www.wikipedia.org/', destfile = "mydirectory/wikipage.html", method = "wget", extra = "-r -p --random-wait")

You will, however, have the problem that it will attempt to save all downloaded items into the same destfile.

Note that there was a similar question a while ago (I saw that only now). The suggested solution was to use system() instead of download.file to run the wget command. Adapted to your problem:

setwd("./mydirectory")
system("wget http://www.wikipedia.org -p -k --random-wait")

Edit: Please also note that both commands will only work on systems with wget installed. On Linux/BSD/Mac, the package to install should usually be called wget. On Windows, wget is (according to the download.file() help) available from packages like gnuwin32 and Cygwin. In this case, the system() command may still not work if the system does not know where the wget executable is. You may, in this case, need to specify the absolute path to the wget executable.

0range
  • 2,088
  • 1
  • 24
  • 32
  • Hi Orange, thank you very much for your answer on my question! If I run the first suggestion, I still get an error message though. It says the following: – Carolin May 11 '18 at 15:37
  • @Carolin Code is functional now. Provided that the path passed to `destfile` exists (otherwise it throws a `'wget' call had nonzero exit status`). – 0range May 11 '18 at 15:46
  • Thank you, I had pressed enter to early, sorry about that. so the error message I get is: download.file('https://www.wikipedia.org/', destfile = "mydirectory/wikipage.html", method = "wget", extra = "-r -p --random-wait") – Carolin May 11 '18 at 15:46
  • @Carolin ahm ...no. What you just posted is also not your error message, this is the `download.file` command. Besides: I am virtually certain this does not do what you want. It downloads all items, one after the other, and collects them into one single file. In the case of your wikipedia example this is just index.html and robots.txt. Unless you manage to slice the destination file up again in the right places, this will not do you any good. I suggest you look into the other solution that uses `system()` – 0range May 11 '18 at 15:56
  • Well, yes, sorry :-/. It actually gave me the same error as in your suggested link (i.e. 'wget' call had nonzero exit status). I have tried with the system() solution, but I get a "status 127" error message, which I think indicates that the command is not found. Do you know why this could be the case? – Carolin May 11 '18 at 16:04
  • @Carolin Strictly speaking, this is not related to the question any more and the comment section should not be used for extended conversations. I assumed you are on a Linux/Unix/Mac system with `wget` installed (In this case, command not found indicates that you must first install `wget`); if you are on Windows, you may want to refer to [this question](https://stackoverflow.com/q/13342488/1735215). – 0range May 11 '18 at 16:23
  • Don't forget to honor @Orange's effort by accepting the answer. – Dieter Menne May 16 '18 at 15:42