3

I have the following site http://www.asd.com.tr. I want to download all PDF files into one directory. I've tried a couple of commands but am not having much luck.

$ wget --random-wait -r -l inf -nd -A pdf http://www.asd.com.tr/

With this code only four PDF files were downloaded. Check this link, there are over several thousand PDFs available:

For instance, hundreds of files are in the following folder:

But I can't figure out how to access them correctly to see and download them all, there are some of folders in this subdirectory, http://www.asd.com.tr/Folders/, and thousands of PDFs in these folders.

I've tried to mirror site using -m command but it failed too.

Any more suggestions?

slm
  • 15,396
  • 12
  • 109
  • 124
eddie skywalker
  • 189
  • 1
  • 5
  • 16

1 Answers1

9

First, verify that the TOS of the web site permit to crawl it. Then, one solution is : 

mech-dump --links 'http://domain.com' |
    grep pdf$ |
    sed 's/\s+/%20/g' |
    xargs -I% wget http://domain.com/%

The mech-dump command comes with Perl's module WWW::Mechanize (libwww-mechanize-perl package on debian & debian likes distros)

Gilles Quénot
  • 173,512
  • 41
  • 224
  • 223