0

I am trying to use the search option in https://www.homecentre.com/ae/en/ and store the number of products displayed in output table for each search

import requests
from bs4 import BeautifulSoup
import pandas as pd

r = requests.get("https://www.homecentre.com/ae/en/", params=dict(
query="baby toys",
page=2
))

text = r.text

The problem is it only shows the source code of the first page and not the one searched for.

I am trying to get the source code of the page below and save 22 Products as my output

Output Page Screen-shot I am not sure if this is a logical mistake or something.

albert
  • 8,112
  • 3
  • 47
  • 63
user3117837
  • 87
  • 1
  • 8

1 Answers1

1

First, there is not a second page on that url.

Second, you are sending that request to the wrong link.

What you need to do is:

response = requests.get('https://www.homecentre.com/ae/en/search/', params=dict(q='baby toys'))

print(response.url)
# https://www.homecentre.com/ae/en/search/?q=baby+toys
BcK
  • 2,548
  • 1
  • 13
  • 27
  • This returns the same source code and I am not being able to find the section of source code which displays the output of the search. Or is there some other logic which needs to be coded to parse that section – user3117837 Jun 25 '18 at 09:45
  • @user3117837 The reason for that is, sometimes websites generate content with javascript dinamically. Solution for this situation -> https://stackoverflow.com/questions/13960567/reading-dynamically-generated-web-pages-using-python – BcK Jun 25 '18 at 16:33