Scaning a URL pattern for images?

Question

I want to scan a website and download the images in it. For example, for a website URL like this: a.example.com/2vZBkE.jpg, I need a bot to scan from a.example.com/aaaaaa.jpg to a.example.com/AAAAAA.jpg to a.example.com/999999.jpg, and if there is an image, save the URL or download the image.

I tried using Python and Scrapy but I am very new to it. This is as far as I could go:

import scrapy

from scrapy.contrib.spiders import Rule, CrawlSpider
from scrapy.contrib.linkextractors import LinkExtractor
from example.items import ExampleItem

class exampleSpider(CrawlSpider):
    name = 'example'
    allowed_domains = ['example.com']
    start_urls = ['http://a.example.com/2vZBkE']
    #rules = [Rule(LinkExtractor(allow=['/.*']),'parse_example')]
     rules = (Rule(SgmlLinkExtractor(allow=('\/%s\/.*',)), callback='parse_example'),
    )

    def parse_example(self,response):
        image = ExampleItem()
        image['title']=response.xpath(\
            "//h5[@id='image-title']/text()").extract()

        rel = response.xpath("//img/@src").extract()
        image ['image_urls'] = ['http:'+rel[0]]
        return image

I think I need to change this line:

rules = (Rule(SgmlLinkExtractor(allow=('\/%s\/.*',)), callback='parse_example'),
        )

to somehow limit %s to 6 characters and make Scrapy try possible combinations. Any ideas?

So you want to download all images in the form of `a.example.com/{id}.jpg` ? — Chaker, Oct 17 '15 at 21:17
Yes. First I need to check if there is an image on that id and then download it. — Scraper, Oct 17 '15 at 21:24

score 0 · Answer 1 · answered Oct 17 '15 at 21:38

I don't know about Scrapy. But you can do it with requests and itertools

from string import ascii_letters, digits
from itertools import product
import requests

# You need to implement this function to download images
# check this http://stackoverflow.com/questions/13137817
def download_image(url):
    print url  

def check_link(url):
    r = requests.head(url)
    if r.status_code == 200:
        return True
    return False

# Check all possible combinations and check them
def generate_links():
    base_url = "http://a.example.com/{}.jpg"
    for combination in product(ascii_letters+digits, repeat=5):
        url = base_url.format("".join(combination))
        if check_link(url):
            download_image(url)

are you sure this is working? I change the URL and run but nothing happened. — Scraper, Oct 17 '15 at 22:34

score 0 · Answer 2 · answered Oct 18 '15 at 15:59

0

to extract links like href="a.example.com/123456.jpg"

use the following regex:

"=\"(\S+/[\w\d]{6}.jpg)"

answered Oct 18 '15 at 15:59

Gurpal Singh

1
1

Scaning a URL pattern for images?

2 Answers2