0

So i have written som code to download all the images from a site but it seems like there is some sort of limit(128) to the amount of pictures i can download from the site?

my code for getting the images is as follows:

This code somehow stops downloading images after image 128 even though the site has over 220 images on it?

The HTML code from the site is as follows:

The HTML code does include this thing called "data-page_size="128"" as you can se on the image. I dont know if this has anything to do with it. Also the command ""document.select("div.continuous-pagination.icon-preview-inline").select("img[src$=.png]").size"" returns a value of 128.

Is it some workaround to get the program to download all the images from the site?

link to site: https://www.iconfinder.com/search/icons?family=feather

For those wondering. I do have a licence to use and download the images from the site (Even though most of these are free), but there is no way of downloading the icons in mass from the site as of now and i would like to automate the task.

I tried getting the image information of all 220 images on the site.

olavSR
  • 1

1 Answers1

0

JSoup is great for pages that do not use Javascript to load content. The page you referred to loads more images as you scroll down, finally revealing all available images.

Try this: refresh the page in your browser but do not scroll down. Open the Inspector and execute $x('//div[contains(@class, "icon-preview-bg")]'). Expand the resulting array to see that there are 128 elements. Now, scroll all the way down the page (you should notice the vertical scrollbar changing positions up and down), and repeat the path search in the Inspector: result = 286.

To get all the images, you would have to use a web driver (e.g., Selenium) driving an instance of Chrome (you can do it in headless mode) on your machine to scroll down (Chrome and Firefox examples), forcing the reloads. Then scrape the page for the image urls, using driver.findElements(By.xpath(...)).

pfurbacher
  • 1,789
  • 3
  • 15
  • 23