BeautifulSoup Issue: Get exact link url and Title

Question

This is my code. This code returns all h2 title not link

from bs4 import BeautifulSoup
import requests
url=requests.get("http://www.prothom-alo.com/")
data=url.text
soup=BeautifulSoup(data ,"lxml")
for link in soup.find_all("h2"):
    print(link)

I want to get link and title like as link+title. I was trying many methods but I cant it properly. How can I get the exact link to the title?

What does it mean `link+title`? Did you mean to scrape `link` and `title` separately? — SIM, Jan 09 '18 at 15:58

score 3 · Accepted Answer · answered Jan 09 '18 at 16:34

I'm guessing you want all the titles with their corresponding links(href).

import requests
from bs4 import BeautifulSoup

r = requests.get('http://www.prothomalo.com/')
soup = BeautifulSoup(r.text, 'html.parser')
titles = {}
for div in soup.find_all('div', {'class': 'col col1'}):
    title = div.find('span', {'class': 'title'}).text
    link = div.find('a', {'class': 'link_overlay'}).get('href')
    titles[title] = link

Now, we've got a dictionary with all the titles as keys and their corresponding links(href) as values.

To check what we've got

for t in titles.items():
    print(t)

Output:

('ঢাকা উত্তর সিটি উপনির্বাচনে সেনা চান ফখরুল', 'bangladesh/article/1405261/%E0%A6%A2%E0%A6%BE%E0%A6%95%E0%A6%BE-%E0%A6%89%E0%A6%A4%E0%A7%8D%E0%A6%A4%E0%A6%B0-%E0%A6%B8%E0%A6%BF%E0%A6%9F%E0%A6%BF-%E0%A6%89%E0%A6%AA%E0%A6%A8%E0%A6%BF%E0%A6%B0%E0%A7%8D%E0%A6%AC%E0%A6%BE%E0%A6%9A%E0%A6%A8%E0%A7%87-%E0%A6%B8%E0%A7%87%E0%A6%A8%E0%A6%BE-%E0%A6%9A%E0%A6%BE%E0%A6%A8-%E0%A6%AB%E0%A6%96%E0%A6%B0%E0%A7%81%E0%A6%B2')
('সোনার ভরি ৫০ হাজার টাকা ছাড়াল', 'economy/article/1405121/%E0%A6%B8%E0%A7%8B%E0%A6%A8%E0%A6%BE%E0%A6%B0-%E0%A6%AD%E0%A6%B0%E0%A6%BF-%E0%A7%AB%E0%A7%A6-%E0%A6%B9%E0%A6%BE%E0%A6%9C%E0%A6%BE%E0%A6%B0-%E0%A6%9F%E0%A6%BE%E0%A6%95%E0%A6%BE-%E0%A6%9B%E0%A6%BE%E0%A7%9C%E0%A6%BE%E0%A6%B2')
('ঘুষসংক্রান্ত বক্তব্যের জন্য শিক্ষামন্ত্রীর পদত্যাগ দাবি', 'bangladesh/article/1405246/%E0%A6%B6%E0%A6%BF%E0%A6%95%E0%A7%8D%E0%A6%B7%E0%A6%BE%E0%A6%AE%E0%A6%A8%E0%A7%8D%E0%A6%A4%E0%A7%8D%E0%A6%B0%E0%A7%80%E0%A6%B0-%E0%A6%AA%E0%A6%A6%E0%A6%A4%E0%A7%8D%E0%A6%AF%E0%A6%BE%E0%A6%97-%E0%A6%A6%E0%A6%BE%E0%A6%AC%E0%A6%BF')
('মুমিনুলকে ডাবল সেঞ্চুরির হাতছানি', 'sports/article/1405116/%E0%A6%AE%E0%A7%81%E0%A6%AE%E0%A6%BF%E0%A6%A8%E0%A7%81%E0%A6%B2%E0%A6%95%E0%A7%87-%E0%A6%A1%E0%A6%BE%E0%A6%AC%E0%A6%B2-%E0%A6%B8%E0%A7%87%E0%A6%9E%E0%A7%8D%E0%A6%9A%E0%A7%81%E0%A6%B0%E0%A6%BF%E0%A6%B0-%E0%A6%B9%E0%A6%BE%E0%A6%A4%E0%A6%9B%E0%A6%BE%E0%A6%A8%E0%A6%BF')
('সাংবাদিকদের প্রশ্ন এড়াতে...', 'international/article/1405181/%E0%A6%B8%E0%A6%BE%E0%A6%82%E0%A6%AC%E0%A6%BE%E0%A6%A6%E0%A6%BF%E0%A6%95%E0%A6%A6%E0%A7%87%E0%A6%B0-%E0%A6%AA%E0%A7%8D%E0%A6%B0%E0%A6%B6%E0%A7%8D%E0%A6%A8-%E0%A6%8F%E0%A7%9C%E0%A6%BE%E0%A6%A4%E0%A7%87')
('ভারতে সিনেমা হলে জাতীয় সংগীত বাধ্যতামূলক নয়', 'international/article/1405071/%E0%A6%AD%E0%A6%BE%E0%A6%B0%E0%A6%A4%E0%A7%87-%E0%A6%B8%E0%A6%BF%E0%A6%A8%E0%A7%87%E0%A6%AE%E0%A6%BE-%E0%A6%B9%E0%A6%B2%E0%A7%87-%E0%A6%9C%E0%A6%BE%E0%A6%A4%E0%A7%80%E0%A7%9F-%E0%A6%B8%E0%A6%82%E0%A6%97%E0%A7%80%E0%A6%A4-%E0%A6%AC%E0%A6%BE%E0%A6%9C%E0%A6%BE%E0%A6%A8%E0%A7%8B-%E0%A6%AC%E0%A6%BE%E0%A6%A7%E0%A7%8D%E0%A6%AF%E0%A6%A4%E0%A6%BE%E0%A6%AE%E0%A7%82%E0%A6%B2%E0%A6%95-%E0%A6%A8%E0%A7%9F')
('মেধাবীদের কত দিন দূরে রাখবে দেশ?', 'durporobash/article/1405241/%E0%A6%AE%E0%A7%87%E0%A6%A7%E0%A6%BE%E0%A6%AC%E0%A7%80%E0%A6%A6%E0%A7%87%E0%A6%B0-%E0%A6%95%E0%A6%A4-%E0%A6%A6%E0%A6%BF%E0%A6%A8-%E0%A6%A6%E0%A7%82%E0%A6%B0%E0%A7%87-%E0%A6%B0%E0%A6%BE%E0%A6%96%E0%A6%AC%E0%A7%87-%E0%A6%A6%E0%A7%87%E0%A6%B6')
('প্রিয় শৈশব, যেখানেই আছ ভালো থেকো!', 'durporobash/article/1405206/%E0%A6%AA%E0%A7%8D%E0%A6%B0%E0%A6%BF%E0%A7%9F-%E0%A6%B6%E0%A7%88%E0%A6%B6%E0%A6%AC-%E0%A6%AF%E0%A7%87%E0%A6%96%E0%A6%BE%E0%A6%A8%E0%A7%87%E0%A6%87-%E0%A6%86%E0%A6%9B-%E0%A6%AD%E0%A6%BE%E0%A6%B2%E0%A7%8B-%E0%A6%A5%E0%A7%87%E0%A6%95%E0%A7%8B')
('প্রেসিডেন্ট নির্বাচনে ট্রাম্পের প্রতিদ্বন্দ্বী অপরাহ্\u200c!', 'northamerica/article/1405231/%E0%A6%AA%E0%A7%8D%E0%A6%B0%E0%A7%87%E0%A6%B8%E0%A6%BF%E0%A6%A1%E0%A7%87%E0%A6%A8%E0%A7%8D%E0%A6%9F-%E0%A6%A8%E0%A6%BF%E0%A6%B0%E0%A7%8D%E0%A6%AC%E0%A6%BE%E0%A6%9A%E0%A6%A8%E0%A7%87-%E0%A6%9F%E0%A7%8D%E0%A6%B0%E0%A6%BE%E0%A6%AE%E0%A7%8D%E0%A6%AA%E0%A7%87%E0%A6%B0-%E0%A6%AA%E0%A7%8D%E0%A6%B0%E0%A6%A4%E0%A6%BF%E0%A6%A6%E0%A7%8D%E0%A6%AC%E0%A6%A8%E0%A7%8D%E0%A6%A6%E0%A7%8D%E0%A6%AC%E0%A7%80')
('একেই বলে ভাগ্য!', 'northamerica/article/1405091/%E0%A6%8F%E0%A6%95%E0%A7%87%E0%A6%87-%E0%A6%AC%E0%A6%B2%E0%A7%87-%E0%A6%AD%E0%A6%BE%E0%A6%97%E0%A7%8D%E0%A6%AF')
('নগরে শীতের হাওয়া', 'bangladesh/article/1405076/%E0%A6%A8%E0%A6%97%E0%A6%B0%E0%A7%87-%E0%A6%B6%E0%A7%80%E0%A6%A4%E0%A7%87%E0%A6%B0-%E0%A6%B9%E0%A6%BE%E0%A6%93%E0%A7%9F%E0%A6%BE')

I hope this is what you're looking for. I can't verify since I don't understand the language.

bla · Answer 2 · 2018-01-09T14:52:36.807

0

Perhaps you meant to fetch the text inside those elements. If this is the case, this may help:

from bs4 import BeautifulSoup
import requests
url=requests.get("http://www.prothom-alo.com/")
data=url.text
soup=BeautifulSoup(data ,"lxml")
for link in soup.find_all("h2"):
    print(link.text)

edited Jan 09 '18 at 14:52

answered Jan 09 '18 at 14:44

bla

1,840
1
13
17

score 0 · Answer 3 · answered Jan 09 '18 at 16:30

Your question is incomplete! Here, i separated titles and links. I think you need something like this :

For Python 3 :

from bs4 import BeautifulSoup
import requests
url = requests.get("http://www.prothomalo.com/")
data = url.text
soup = BeautifulSoup( data )


for link in soup.findAll('div',{'class':'row'}):
     print (link.find('h2').text)
     print (link.find('a')['href'])

Maybe this helps you , if not , here some links:

1) How to extract link from <a> inside the <h2 class=section-heading>:BeautifulSoup

2) BeautifulSoup getting href

3) retrieve links from web page using python and BeautifulSoup

4) How to find elements by class

Good luck!

BeautifulSoup Issue: Get exact link url and Title

3 Answers3