60

I am new to Python and I am learning it for scraping purposes I am using BeautifulSoup to collect links (i.e href of 'a' tag). I am trying to collect the links under the "UPCOMING EVENTS" tab of site http://allevents.in/lahore/. I am using Firebug to inspect the element and to get the CSS path but this code returns me nothing. I am looking for the fix and also some suggestions for how I can choose proper CSS selectors to retrieve desired links from any site. I wrote this piece of code:

from bs4 import BeautifulSoup

import requests

url = "http://allevents.in/lahore/"

r  = requests.get(url)

data = r.text

soup = BeautifulSoup(data)
for link in soup.select( 'html body div.non-overlay.gray-trans-back div.container div.row div.span8 div#eh-1748056798.events-horizontal div.eh-container.row ul.eh-slider li.h-item div.h-meta div.title a[href]'):
    print link.get('href')
Sebastian Zartner
  • 18,808
  • 10
  • 90
  • 132
Flecha
  • 615
  • 1
  • 5
  • 4
  • 6
    You really don't need to be *this specific* for your selector, really. – Martijn Pieters Jul 17 '14 at 10:53
  • Is there an actual problem, however? Does the code work, not work, what did you expect to happen? – Martijn Pieters Jul 17 '14 at 10:53
  • Also, [don't use `r.text`, use `r.content`](http://stackoverflow.com/questions/24790258/encoding-issue-of-a-character-in-utf-8) here. – Martijn Pieters Jul 17 '14 at 10:54
  • 1
    @Martijn Pieters the code doesn't work.I expect to get the links of all events listed under the tab of "Upcoming Events" on http://allevents.in/lahore/ – Flecha Jul 17 '14 at 10:56
  • @MartijnPieters am not using r.content because I plan to extract some text along with links, but at the moment I am unable to retrieve the desired links – Flecha Jul 17 '14 at 11:06
  • You need to leave content characterset detection to BeautifulSoup. You can always use `r.text` *elsewhere* if you need access to a Unicode value of the HTML source, but BeautifulSoup **will give you Unicode too**, but will do a better job of determining the correct codec. – Martijn Pieters Jul 17 '14 at 11:11

2 Answers2

69

The page is not the most friendly in the use of classes and markup, but even so your CSS selector is too specific to be useful here.

If you want Upcoming Events, you want just the first <div class="events-horizontal">, then just grab the <div class="title"><a href="..."></div> tags, so the links on titles:

upcoming_events_div = soup.select_one('div.events-horizontal')
for link in upcoming_events_div.select('div.title a[href]'):
    print(link['href'])

Note that you should not use r.text; use r.content and leave decoding to Unicode to BeautifulSoup. See Encoding issue of a character in utf-8

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • http://stackoverflow.com/questions/24789094/css-selectors-to-be-used-for-scraping-specific-links @Martijn Pieters I also asked a question similar to this I am not getting the point that how can I choose CSS selectors like you are suggesting me the edits could you please guide me throgh this how to choose desired CSS slectors ? – Flecha Jul 17 '14 at 11:16
  • 1
    @Flecha: in general, try using a selector that picks *just* the tag you are looking for, then add more context if that is not enough to identify the object(s) uniquely. – Martijn Pieters Jul 17 '14 at 11:18
  • 1
    @Flecha: And don't fixate on just using CSS selectors, they are not always enough to identify something uniquely. Here I first picked *one* `events-horizontal` element first, as we cannot easily pick just that one with CSS. – Martijn Pieters Jul 17 '14 at 11:19
27
import bs4 , requests

res = requests.get("http://allevents.in/lahore/")
soup = bs4.BeautifulSoup(res.text)
for link in soup.select('a[property="schema:url"]'):
    print link.get('href')

This code will work fine!!

Anuj Saraswat
  • 271
  • 3
  • 3