How to download all files from hidden directory

Question

I have do download all log files from a virtual directory within a site. The access to virtual directory is forbidden but files are accessible.

I have manually entered the file names to download

dir="Mar"
for ((i=1;i<100;i++)); do
   wget http://sz.dsyn.com/2014/$dir/log_$i.txt
done

The problem is the script is not generic and most of the time I need to find out how many files are there and tweak the for loop. Is there a way to trigger wget to fetch all files without me bothering to specify the exact count.

Note: If I use the browser to view http://sz.dsyn.com/2014/$dir, it is 403 forbidden. I cant pull all the files via browser tool/extension.

Is there a realistic maximum? Why not just loop to a high number? If it fails, it'll fail quickly and you can just ignore it. — arco444, Mar 11 '14 at 14:07

score 1 · Accepted Answer · edited May 23 '17 at 12:21

1

First of all check this similar question If this is not what you are looking for, you need to generate a file of URLs within and feed wget. e.g.

 wget --input-file=http://sz.dsyn.com/2014/$dir/filelist.txt

edited May 23 '17 at 12:21

Community

1
1

answered Mar 11 '14 at 18:15

hawk

5,409
2
23
29

score 0 · Answer 2 · answered Mar 11 '14 at 15:02

0

wget will have the same problem your browser has: it cannot read the directory. Just pull until your first failure then quit.

answered Mar 11 '14 at 15:02

Bruce K

749
5
15

How to download all files from hidden directory

2 Answers2