Python urllib2, how to get the next match

Question

Solved it myself Thank you guys for your concern and sorry.

---

I've made my bot to fetch some stuff from Twitter accounts, like Date of Join, Number of Tweets, Number of Followers and so on.
I tried to make it get the tweets of that account as well, but it ALWAYS gets only the latest tweet. In the page source code, ALL the tweets start like this:

dir="ltr" data-aria-label-part="0"
And now bot will return the first tweet. So, how can I make it skip the first tweet, and gets the second or third or any other tweet I want?
Thanks. P.S.: It's only for Python2.7

Here's my code:

url = 'http://www.twitter.com/'+account  
req = urllib2.Request(url)  
req.add_header = ('User-agent', 'Mozilla/5.0')  
r = urllib2.urlopen(req)    
target = r.read()  
od = re.search('dir="ltr" data-aria-label-part="0"',target)    
h1 = target[od.end():]  
h1 = h1[:re.search('</p>',h1).start()]
tweet = decode(h1)  

Answer 1

It looks like you're trying to parse HTML with regular expressions. Don't do that. It's a waste of time and generally can't be done. For that, you want to use lxml.html (http://lxml.de/lxmlhtml.html) or BeautifulSoup (http://www.crummy.com/software/BeautifulSoup/)

Furthermore, what your real problem seems to be to access Twitter through Python. Which means that what you should really be doing is use a Twitter lib for Python, such as Twython (http://twython.readthedocs.org/en/latest/) or Tweepy (https://github.com/tweepy/tweepy).

Answer 2

With BeautifulSoup you can find all instances and plus it have awesome features that isolate only the text and much more.

Something in the lines of:

from bs4 import BeautifulSoup
import urllib2

page = urllib2.urlopen(url)
soup = BeautifulSoup(page)

soup = soup.body.find_all('p', attrs={'class':'ProfileTweet-text'})

for t in soup:
    print t.get_text()

EDIT:

I'm not that familiar with BeautifulSoup too, but if you inspect the twitter page you will see that every tweet is acessed by 'div.ProfileTweet u-textBreak[...]' and inside it there's 'div.ProfileTweet-contents' with 'p.ProfileTweet-text[...]', so 'p' is just a class, and you are look for a string that start with a class named 'p' and that it has a sub-class 'ProfileTweet-text' in it.

From the docstring of find_all():

def find_all(self, name=None, attrs={}, recursive=True, text=None,
             limit=None, **kwargs):
"""Extracts a list of Tag objects that match the given
    criteria.  You can specify the name of the Tag and any
    attributes you want the Tag to have.

    The value of a key-value pair in the 'attrs' map can be a
    string, a list of strings, a regular expression object, or a
    callable that takes a string and returns whether or not the
    string matches for some custom definition of 'matches'. The
    same is true of the tag name."""