Regex - Getting Value from HTML with Changing Middle

Question

I am attempting to regex the end of the twitter link where the only identifier is the class value fl. Thus, the regex (to the best of my knowledge) must include:

class=\"fl\"

account for changing middle section, where \S+ does not work, then find and group on:

 data-href="http://www.twitter.com/(newyorklife)

where the group is found in the parentheses. The whole string I am trying to parse through is.

<g-link class="fl"><a href="/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=32&amp;cad=rja&amp;uact=8&amp;ved=0ahUKEwjknIy87oHWAhXHi1QKHXQdAJsQ9zAIyQEwHw&amp;url=http%3A%2F%2Fwww.twitter.com%2Fnewyorklife&amp;usg=AFQjCNHKcAcw6H6cYG3YH1j4V3UOxX1whw" onmousedown="return rwt(this,'','','','32','AFQjCNHKcAcw6H6cYG3YH1j4V3UOxX1whw','','0ahUKEwjknIy87oHWAhXHi1QKHXQdAJsQ9zAIyQEwHw','','',event)" data-href="http://www.twitter.com/newyorklife"><div jsl="$t t-XNwoAoU5dyo;$x 0;" class="r-iBA3fWkVHWLE"><g-img class="_tek"><img id="uid_4" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAIAAAD8GO2jAAABZUlEQVR4AWLQWfWQpmjUAjxo1IJRC2wBpJTDQkVRFIafKBujZrnGjbNd84wHiJNs27btrm3rrFbW1T73m20u/yMsW0cBg6zue5XCYLFQcC41JK0I3PsYaWvC+BkugYFljrbmWPp/H/86FOnhB2hGZbTg/dBhFoEBhsoEAO23Su9+5s/9nA0R/ANtXEgNJTtiAgObfB28gZaKt8Wen2ZarhRgjVL8nagGmetC+IFMb5lgqOtOZAtsLVgjcIhFZqD+RLYj0IFzGCwUcRctc7XgNNcyA7GBhAW+EWvnHK3XCjqDhg3OUpvAEegFTgAdA+nrwnuF4zCw7DSlwqOPscRxUAmtiYqY5NDXImz/6mPprlAP1sDgcjdFLokdCkPGW6Kstmbhtoim2IWNsRsvFXNsjURvBmvgiMROc11S0+BhVvmhFAUDhewrISgbg4/qlyUdeEnl+sBk7SOgfcBSb3jWaKMWjFoAABKespvtvzYlAAAAAElFTkSuQmCC" data-deferred="1" class="_WCg" height="32" width="32" alt="" onload="typeof google==='object'&amp;&amp;google.aft&amp;&amp;google.aft(this)"></g-img></div>Twitter</a></g-link>

I am not aware if regex has a method or can skip the entire middle section with so many special chars. I have been playing at pythex.org for a while and can't find a method which simply finds an initial value then skips everything until ... specified values. Any ideas?

Edit. I want the string 'Newyorklife' as the output. Though this is a changing value, so really i just want the \w+ which comes after twitter.com/. The issue is that the class=fl is the only unique identifier for the line on the webpage (as twitter and data-href show up elsewhere on the page).

Do I understand correctly in that you want to capture the bit from "rwt(" ...to... "event)" yes? — Martin Joiner, Aug 31 '17 at 17:37
I am attempting to acquire (newyorklife) in this example. Though this will be a changing value thus it will be a \w+ after twitter.com which I want to acquire. the only unique value is the class=fl value. — WolVes, Aug 31 '17 at 17:39

Martin Joiner · Accepted Answer · 2017-08-31T18:02:33.430

There will be a way to do it in one regex string but it will be pig ugly and difficult to read. So I would approach this in 2 steps. Firstly, capture the HTML tag with class of "fl", secondly find the Twitter handle in the attribute.

str = document.documentElement.innerHTML;

anchorTag = str.match( "class=\"fl\">([^>]+)" )[1];

matches = anchorTag.match("twitter\.com%2F([^&]+)&");
if( matches != null && matches.length > 1 ){
    var handle = matches[1];
}

console.log(handle);

<g-link class="fl"><a href="/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=32&amp;cad=rja&amp;uact=8&amp;ved=0ahUKEwjknIy87oHWAhXHi1QKHXQdAJsQ9zAIyQEwHw&amp;url=http%3A%2F%2Fwww.twitter.com%2Fnewyorklife&amp;usg=AFQjCNHKcAcw6H6cYG3YH1j4V3UOxX1whw" onmousedown="return rwt(this,'','','','32','AFQjCNHKcAcw6H6cYG3YH1j4V3UOxX1whw','','0ahUKEwjknIy87oHWAhXHi1QKHXQdAJsQ9zAIyQEwHw','','',event)" data-href="http://www.twitter.com/newyorklife"><div jsl="$t t-XNwoAoU5dyo;$x 0;" class="r-iBA3fWkVHWLE"><g-img class="_tek"><img id="uid_4" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAIAAAD8GO2jAAABZUlEQVR4AWLQWfWQpmjUAjxo1IJRC2wBpJTDQkVRFIafKBujZrnGjbNd84wHiJNs27btrm3rrFbW1T73m20u/yMsW0cBg6zue5XCYLFQcC41JK0I3PsYaWvC+BkugYFljrbmWPp/H/86FOnhB2hGZbTg/dBhFoEBhsoEAO23Su9+5s/9nA0R/ANtXEgNJTtiAgObfB28gZaKt8Wen2ZarhRgjVL8nagGmetC+IFMb5lgqOtOZAtsLVgjcIhFZqD+RLYj0IFzGCwUcRctc7XgNNcyA7GBhAW+EWvnHK3XCjqDhg3OUpvAEegFTgAdA+nrwnuF4zCw7DSlwqOPscRxUAmtiYqY5NDXImz/6mPprlAP1sDgcjdFLokdCkPGW6Kstmbhtoim2IWNsRsvFXNsjURvBmvgiMROc11S0+BhVvmhFAUDhewrISgbg4/qlyUdeEnl+sBk7SOgfcBSb3jWaKMWjFoAABKespvtvzYlAAAAAElFTkSuQmCC" data-deferred="1" class="_WCg" height="32" width="32" alt="" onload="typeof google==='object'&amp;&amp;google.aft&amp;&amp;google.aft(this)"></g-img></div>Twitter</a></g-link>

Come on, this is not even `Python` as requested in the question! — Jan, Aug 31 '17 at 19:03
Its the one i was able to modify the concept to get to work. though both of your results were good, for whatever reason I couldn't get them to function. I was particularly confused as to why i couldn't get your regex to work @ekhumoro — WolVes, Aug 31 '17 at 20:11
@WolVes. The only thing I can think of is that your real html data contains newlines, which the `.` operator won't normally match unless the `re.S` flag is set. FWIW, I will update my answer to deal with that issue. — ekhumoro, Aug 31 '17 at 21:48

Jan · Answer 2 · 2017-08-31T18:39:44.883

No regex needed, use a decent parser instead:

from bs4 import BeautifulSoup

html = """<g-link class="fl"><a href="/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=32&amp;cad=rja&amp;uact=8&amp;ved=0ahUKEwjknIy87oHWAhXHi1QKHXQdAJsQ9zAIyQEwHw&amp;url=http%3A%2F%2Fwww.twitter.com%2Fnewyorklife&amp;usg=AFQjCNHKcAcw6H6cYG3YH1j4V3UOxX1whw" onmousedown="return rwt(this,'','','','32','AFQjCNHKcAcw6H6cYG3YH1j4V3UOxX1whw','','0ahUKEwjknIy87oHWAhXHi1QKHXQdAJsQ9zAIyQEwHw','','',event)" data-href="http://www.twitter.com/newyorklife"><div jsl="$t t-XNwoAoU5dyo;$x 0;" class="r-iBA3fWkVHWLE"><g-img class="_tek"><img id="uid_4" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAIAAAD8GO2jAAABZUlEQVR4AWLQWfWQpmjUAjxo1IJRC2wBpJTDQkVRFIafKBujZrnGjbNd84wHiJNs27btrm3rrFbW1T73m20u/yMsW0cBg6zue5XCYLFQcC41JK0I3PsYaWvC+BkugYFljrbmWPp/H/86FOnhB2hGZbTg/dBhFoEBhsoEAO23Su9+5s/9nA0R/ANtXEgNJTtiAgObfB28gZaKt8Wen2ZarhRgjVL8nagGmetC+IFMb5lgqOtOZAtsLVgjcIhFZqD+RLYj0IFzGCwUcRctc7XgNNcyA7GBhAW+EWvnHK3XCjqDhg3OUpvAEegFTgAdA+nrwnuF4zCw7DSlwqOPscRxUAmtiYqY5NDXImz/6mPprlAP1sDgcjdFLokdCkPGW6Kstmbhtoim2IWNsRsvFXNsjURvBmvgiMROc11S0+BhVvmhFAUDhewrISgbg4/qlyUdeEnl+sBk7SOgfcBSb3jWaKMWjFoAABKespvtvzYlAAAAAElFTkSuQmCC" data-deferred="1" class="_WCg" height="32" width="32" alt="" onload="typeof google==='object'&amp;&amp;google.aft&amp;&amp;google.aft(this)"></g-img></div>Twitter</a></g-link>"""

soup = BeautifulSoup(html, 'html5lib')

# select one
user = soup.select_one('.fl > a')["data-href"].split('/')[-1]
print(user)
# newyorklife

To select multiple links, use soup.findAll(), see the documentation for more information.

ekhumoro · Answer 3 · 2017-08-31T21:47:43.723

Here is a working regexp:

>>> r = re.compile(r'\bclass="fl".*?\bdata-href="http://www\.twitter\.com/(\w+)"', re.S)
>>> r.search(s).group(1)
'newyorklife'

The key concept here is non-greedy matching. Since there may be more than one data-href on the page, you must take care to find the first occurence of it after class="fl" has been matched. So the .*? expression is used here to match as few characters as possible before trying to match the next data-href.

Ajax1234 · Answer 4 · 2017-08-31T17:47:25.730

You can try this:

import re
s = '<g-link class="fl"><a href="/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=32&amp;cad=rja&amp;uact=8&amp;ved=0ahUKEwjknIy87oHWAhXHi1QKHXQdAJsQ9zAIyQEwHw&amp;url=http%3A%2F%2Fwww.twitter.com%2Fnewyorklife&amp;usg=AFQjCNHKcAcw6H6cYG3YH1j4V3UOxX1whw" onmousedown="return rwt(this,'','','','32','AFQjCNHKcAcw6H6cYG3YH1j4V3UOxX1whw','','0ahUKEwjknIy87oHWAhXHi1QKHXQdAJsQ9zAIyQEwHw','','',event)" data-href="http://www.twitter.com/newyorklife"><div jsl="$t t-XNwoAoU5dyo;$x 0;" class="r-iBA3fWkVHWLE"><g-img class="_tek"><img id="uid_4" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAIAAAD8GO2jAAABZUlEQVR4AWLQWfWQpmjUAjxo1IJRC2wBpJTDQkVRFIafKBujZrnGjbNd84wHiJNs27btrm3rrFbW1T73m20u/yMsW0cBg6zue5XCYLFQcC41JK0I3PsYaWvC+BkugYFljrbmWPp/H/86FOnhB2hGZbTg/dBhFoEBhsoEAO23Su9+5s/9nA0R/ANtXEgNJTtiAgObfB28gZaKt8Wen2ZarhRgjVL8nagGmetC+IFMb5lgqOtOZAtsLVgjcIhFZqD+RLYj0IFzGCwUcRctc7XgNNcyA7GBhAW+EWvnHK3XCjqDhg3OUpvAEegFTgAdA+nrwnuF4zCw7DSlwqOPscRxUAmtiYqY5NDXImz/6mPprlAP1sDgcjdFLokdCkPGW6Kstmbhtoim2IWNsRsvFXNsjURvBmvgiMROc11S0+BhVvmhFAUDhewrISgbg4/qlyUdeEnl+sBk7SOgfcBSb3jWaKMWjFoAABKespvtvzYlAAAAAElFTkSuQmCC" data-deferred="1" class="_WCg" height="32" width="32" alt="" onload="typeof google==='object'&amp;&amp;google.aft&amp;&amp;google.aft(this)"></g-img></div>Twitter</a></g-link>'
r = 'data-href="http://www.twitter.com/\((.*?\))'
data = re.findall(r, s)

print(data)

Output:

['newyorklife']

newyorklife is not a constant and twitter shows up frequently as a link. thus, it has to have the unique identifier of class=fl incorporated. — WolVes, Aug 31 '17 at 17:45

Regex - Getting Value from HTML with Changing Middle

4 Answers4