How to pass the value from a list in a url string

Question

I have a problem with this Python script. I'm attempting to pass the values from a list that has home strings in it. I've attached the script. In this command page = requests.get("https://www.google.dz/search?q=lista[url]") I have to put what I'm looking for on google after the search?q=. I want to search multiple keyword, so I made a list. I don't how to pass the values from the list in that command...

import requests
import re
from bs4 import BeautifulSoup

lista = []
lista.append("Samsung S9")
lista.append("Samsung S8")
lista.append("Samsung Note 9")

list_scrape = []

for url in lista:
    page = requests.get("https://www.google.dz/search?q=lista[url]")
    soup = BeautifulSoup(page.content)
    links = soup.findAll("a")
    for link in  soup.find_all("a",href=re.compile("(?<=/url\?q=) 
    (htt.*://.*)")):
        list_scrape.append(re.split(":(?=http)",link["href"].replace("/url?q=","")))

print(list_scrape)

Thank you!

Possible duplicate of [REST API Best practice: How to accept list of parameter values as input](https://stackoverflow.com/questions/2602043/rest-api-best-practice-how-to-accept-list-of-parameter-values-as-input) — Olivier Melançon, Sep 17 '18 at 12:04

score 2 · Accepted Answer · answered Sep 17 '18 at 12:04

2

Use format

for url in lista:
    page = requests.get("https://www.google.dz/search?q={}".format(url))

Or

page = requests.get("https://www.google.dz/search?q=%s" % url)

answered Sep 17 '18 at 12:04

Sushant

3,499
3
17
34

assuming `requests` url is some developmental / production box and not google, doing this can cause disaster if you are doing a blind append in `url`, you would need to [encode](https://stackoverflow.com/questions/5607551/how-to-urlencode-a-querystring-in-python) properly first – NoobEditor Sep 17 '18 at 12:09

score 1 · Answer 2 · answered Sep 17 '18 at 12:24

1

try this..

for url in lista:
    page = requests.get("https://www.google.dz/search?q="+url)

or

page = requests.get("https://www.google.dz/search?q={}".format(url))

answered Sep 17 '18 at 12:24

Sushant Kathuria

123
8

Dmitriy Zub · Answer 3 · 2021-09-10T11:38:49.130

You can use f-string instead which more pythonic way in my opinion to do string formatting:

requests.get(f"https://www.google.dz/search?q={url}")
# or
for query in queries:
   html = requests.get(f"https://www.google.dz/search?q={query}")

Note that the next problem might appear because of no user-agent specified thus Google blocked your request.

Because the default requests user-agent is python-requests. Google understands it and blocks a request since it's not the "real" user visit. Checks what's your user-agent.

Code:

from bs4 import BeautifulSoup
import requests, lxml

headers = {
    "User-agent":
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}

queries = ["Samsung S9", "Samsung S8", "Samsung Note 9"]

for query in queries:
  params = {
    "q": query,
    "gl": "uk",
    "hl": "en"
  }

  html = requests.get("https://www.google.com/search", headers=headers, params=params)
  soup = BeautifulSoup(html.text, "lxml")
  
  for result in soup.select('.tF2Cxc'):
    title = result.select_one('.DKV0Md').text
    link = result.select_one('.yuRUbf a')['href']

    print(f"{title}\n{link}\n")

-------
'''
Samsung Galaxy S9 and S9+ | Buy or See Specs
https://www.samsung.com/uk/smartphones/galaxy-s9/

Samsung Galaxy S9 - Full phone specifications - GSMArena ...
https://www.gsmarena.com/samsung_galaxy_s9-8966.php
...
Samsung Galaxy S8 - Wikipedia
https://en.wikipedia.org/wiki/Samsung_Galaxy_S8

Samsung Galaxy S8 Price in India - Gadgets 360
https://gadgets.ndtv.com/samsung-galaxy-s8-4009
...
Samsung Galaxy Note 9 Cases - Mobile Fun
https://www.mobilefun.co.uk/samsung/galaxy-note-9/cases

Samsung Galaxy Note 9 - Wikipedia
https://en.wikipedia.org/wiki/Samsung_Galaxy_Note_9
'''

Alternatively, you can achieve the same thing by using Google Organic Results API from SerpApi. It's a paid API with a free plan.

The difference in your case is that you don't need to think about how to extract certain things or figure out why something isn't working as it should work. All that really needs to be done is to iterate over structured JSON and get the data you want fast without any headache.

Code to integrate:

import os
from serpapi import GoogleSearch

queries = ["Samsung S9", "Samsung S8", "Samsung Note 9"]

for query in queries:
  params = {
      "engine": "google",
      "q": query,
      "hl": "en",
      "gl": "uk",
      "api_key": os.getenv("API_KEY"),
  }

  search = GoogleSearch(params)
  results = search.get_dict()

  for result in results["organic_results"]:
    print(result['title'])
    print(result['link'])
    print()

------
'''
Samsung Galaxy S9 and S9+ | Buy or See Specs
https://www.samsung.com/uk/smartphones/galaxy-s9/

Samsung Galaxy S9 - Full phone specifications - GSMArena ...
https://www.gsmarena.com/samsung_galaxy_s9-8966.php
...
Samsung Galaxy S8 - Wikipedia
https://en.wikipedia.org/wiki/Samsung_Galaxy_S8

Samsung Galaxy S8 Price in India - Gadgets 360
https://gadgets.ndtv.com/samsung-galaxy-s8-4009
...
Samsung Galaxy Note 9 Cases - Mobile Fun
https://www.mobilefun.co.uk/samsung/galaxy-note-9/cases

Samsung Galaxy Note 9 - Wikipedia
https://en.wikipedia.org/wiki/Samsung_Galaxy_Note_9
'''

Disclaimer, I work for SerpApi.

How to pass the value from a list in a url string

3 Answers3