Adding text to results in an array using beautifulsoup in python

Question

Anyone any ideas how to prepend each item in an array with text before its passed into the next loop?

Basically I have found the links that im after but they do not contain the main sites url, just the child elements

links = []
for link in soup.find_all("a", {"class": "product-info__caption"}):
links.append(link.attrs['href'])

#this returns the urls okay as /products/item
#whereas i need the https://www.example.com/products/item to pass into next loop

for x in links:

result = requests.get(x)
src = result.content
soup = BeautifulSoup(src, 'lxml')
Name = soup.find('h1', class_='product_name')
... and so on

You should include an example of what result you expect to make the question clearer. Also you should indent your code to make it more readable. It is not clear in which loop you want to add the text for the elements of the array. — Rafael Rios, Aug 21 '20 at 18:13

score 1 · Answer 1 · answered Aug 21 '20 at 17:26

1

You can prepend 'https://www.example.com' in your first loop, for example:

links = []
for link in soup.find_all("a", {"class": "product-info__caption"}):
    links.append('https://www.example.com' + link.attrs['href'])

for x in links:
    # your next stuff here
    # ...

answered Aug 21 '20 at 17:26

Andrej Kesely

168,389
15
48
91

1

Excellent working like a charm! im new to this whole coding malarkey so great to get such a swift response! – growntreee Aug 21 '20 at 20:14

theX · Answer 2 · 2020-08-21T17:45:21.243

Building on top of @Andrej Kesely's answer, I think you should use a list comprehension

links = [
   "https://www.example.com" + link.attrs['href']
   for link in soup.find_all("a", {"class": "product-info__caption"})
]

List comprehensions are faster than the conventional for loop. This StackOverflow answer will explain why list comprehensions are faster.

Note

Every list comprehension can be turned into a for loop.

Adding text to results in an array using beautifulsoup in python

2 Answers2

Note

Further reading