python regular expression find and replace html tag with specific attribute value

Question

Am trying to write a regular expression in python that would find all img tags where src attribute equal to a specific value.. I tried to write the below

   # where thm equal /public_media/cache/84/b5/84b59e293cbdb7041b68a84977d62cf3.jpg?image_pk=82
   p = re.compile(r'<img.*?%s.*?>' % thm)
   print p.pattern
   print p.sub(linked_image, c)

and below what I got as an output

<img.*?/public_media/cache/84/b5/84b59e293cbdb7041b68a84977d62cf3.jpg?image_pk=82.*?>

<p><img src="/public_media/cache/84/b5/84b59e293cbdb7041b68a84977d62cf3.jpg?image_pk=82" alt=""></p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf </p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf 
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf 
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf 
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf 
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf 
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf 
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf 
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf 
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf 
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf 
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf

what about using BeautifulSoup for this job? http://www.crummy.com/software/BeautifulSoup/ — Totem, Dec 15 '13 at 15:12
I could use the beautifulSoup, but am altering content that is unstructured and it should maintain its structure without change. problem with beautifulSoup, I won't be able to output the processed soup to the previous structure as output... that's why i need to change replace string method — Mo J. Mughrabi, Dec 15 '13 at 20:33
Please take a look at the new version - I found the error in the raw `thm` variable and proposed two solutions below. — Jon, Dec 15 '13 at 21:30

score 2 · Accepted Answer · answered Dec 17 '13 at 08:57

A solution with LXML

To compare a solution with Regular expressions and LXML I created another post:

An easier and more stable solution is to use lxml with etree. In that solution you access certain DOM elements and edit them.

Convert the HTML string and get it through the right xpath e.g. .//img. The xpath returns a list of all found elements where you can get and set the src attribute. The function etree.tostring(tree) returns an edited string:

from lxml import etree
tree = etree.HTML('''<html>
                     <body>
                        <h1>Title</h1>
                        <img src="/media/old/another_logo.png" alt="" />
                        <p>Lorem Ipsum</p>
                        <p><img src="/media/old/logo.png" alt=""/></p>
                     </body>
                  </html>''')

imgs = tree.xpath('.//img')

for img in imgs:
    print 'OLD_SOURCE', img.get('src')
    img.set('src', '/media/new/python.jpg')

print etree.tostring(tree)

Output

OLD_SOURCE /media/old/another_logo.png
OLD_SOURCE /media/old/logo.png

<html>
    <body>
        <h1>Title</h1>
            <img src="/media/new/python.jpg" alt=""/>
            <p>Lorem Ipsum</p>
            <p><img src="/media/new/python.jpg" alt=""/></p>
    </body>
</html>

score 1 · Answer 2 · edited May 23 '17 at 12:03

A solution with Regular Expressions:

I realized that the string which was inserted into thm was not escaped. So before adding it to your regular expression you need to escape all symbols with an additional signification in the Regular Expression language - here, ? and ..

I replaced ? with [?]{1} and . with \.. The resulting regular expressions matches now the test string.

import re
thm = '/public_media/cache/84/b5/84b59e293cbdb7041b68a84977d62cf3.jpg?image_pk=82'

all_html_code = '''<img.*?/public_media/cache/84/b5/84b59e293cbdb7041b68a84977d62cf3.jpg?image_pk=82.*?>

<p><img src="/public_media/cache/84/b5/84b59e293cbdb7041b68a84977d62cf3.jpg?image_pk=82" alt=""></p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf </p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf
</p><p>lksj lksdfj lsdkfj sldkfj sldkfj lskdfj lsjf lksjf lksj flksdjf klsj flk dkj sdlkfj sdlkfj sldkjf sldkfj lsdkjf lskjflsjfsl lksdjf '''

escaped_thm = thm.replace('.', '\.').replace('?','[?]{1}')
p = re.compile(r'<img.*?src="(%s)".*?>' % escaped_thm)

test_img = '''<img src="/public_media/cache/84/b5/84b59e293cbdb7041b68a84977d62cf3.jpg?image_pk=82" alt="">'''
print p.match(test_img)

new_img_tag = '<img src="/python/logo.jpg" alt=""/>'
print p.sub(new_img_tag, all_html_code)

By the way, why are you looking for <img src=""...>? You could replace the src attribute directly:

escaped_thm = thm.replace('.', '\.').replace('?','[?]{1}')
p = re.compile(r'src="(%s)"' % escaped_thm)

replacement = '''src="/python/logo.jpg"'''
print p.sub(replacement, all_html_code)

Output 1

<_sre.SRE_Match object at 0x... >
<img.*?/public_media/cache/84/b5/84b59e293cbdb7041b68a84977d62cf3.jpg?image_pk=82.*?>

<p><img src="/python/logo.jpg" alt=""/></p><p>lksj lksdfj ... lksdjf

Output 2

<p><img src="/python/logo.jpg" alt=""></p><p>lksj lksdfj ... lksdjf

After asking about a proper way to escape symbols of regular expressions (Regular expression to escape regular expressions) - I can recommend an re.escape instead of the two replace methods.

Using LXML

Do you need to use regular expressions? Regular expressions for HTML can be highly problematic. See more information and wonderful post here(RegEx match open tags except XHTML self-contained tags).

I would rather use XPath like here.

I could use the beautifulSoup, but am altering content that is unstructured and it should maintain its structure without change. problem with beautifulSoup, I won't be able to output the processed soup to the previous structure as output... that's why i need to change replace string method — Mo J. Mughrabi, Dec 15 '13 at 20:33
I added a solution with regex. Please check out the example above. — Jon, Dec 15 '13 at 23:17

python regular expression find and replace html tag with specific attribute value

2 Answers2

Linked