Download Images from a site via a command line

Question

I followed this answer here https://stackoverflow.com/a/4602181/4480164

I ran this command

wget -nd -r -P /Users/bheng/Desktop -A jpeg,jpg,bmp,gif,png  -e robots=off https://www.bunlongheng.com

I kept getting

    --2019-03-06 11:07:19--  https://www.bunlongheng.com/                                                               
Resolving www.bunlongheng.com... 104.27.153.38, 104.27.152.38                                                       
Connecting to www.bunlongheng.com|104.27.153.38|:443... connected.                                                  
HTTP request sent, awaiting response... 500 Internal Server Error                                                   
2019-03-06 11:07:19 ERROR 500: Internal Server Error.

My site is working fine if I go there via browser.

Why ? ‍♂️

How would one go about debugging this further?

@jww As specified in the [on-topic](https://stackoverflow.com/help/on-topic) page, "software tools commonly used by programmers", of which wget is certainly one, is also on-topic here. — Lynn Crumbling, Mar 06 '19 at 17:12
Thanks @Lynn. I think you missed the other part of the citation: *"and is ... a practical, answerable problem that is unique to software development"*. How to use Wget is certainly not unique to software development. Without that criteria everything on a Linux or Windows box would be on-topic here. — jww, Mar 06 '19 at 17:16
@jww But it says "Professional Full Stack Developer" right on the image he pasted :) — Lynn Crumbling, Mar 06 '19 at 17:17

score 2 · Answer 1 · answered Mar 06 '19 at 16:12

2

Try to specify User Agent in wget

wget -U "User Agent Here"

answered Mar 06 '19 at 16:12

mhdINbY

21
1

score 2 · Accepted Answer · answered Mar 06 '19 at 16:45

You get the same behavior when running wget https://www.bunlongheng.com/ without all that stuff with images. Running wget -d https://www.bunlongheng.com/ 2>&1 | less provides some information: there is an index error in a php file :

ErrorException: Undefined offset: 1 (View: /home/forge/bheng/resources/views/layouts/fe/meta.blade.php) (View: /home/forge/bheng/resources/views/layouts/fe/mSkipping 512 bytes of body: [eta.blade.php) in file /home/forge/bheng/storage/framework/views/0b4178e309ed0339363606e08a7e6d3f33347b7f.php on line 76
Stack trace:
  1. ErrorException-&gt;() /home/forge/bheng/storage/framework/views/0b4178e309ed0339363606e08a7e6d3f33347b7f.php:76
  ...
  etc

As proposed by @mhdINbY, if you put a user agent of an existing browser (I tried mine : -U "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0" everything went OK.

I would suspect that your framework analyses the user agent HTTP header in order to format the output accordingly and has a bug when it doesn't know the user agent you are using, here User-Agent: Wget/1.17.1 (linux-gnu)

Download Images from a site via a command line

2 Answers2