Parsing a url in Python

Question

I want to scrape some info about couple of companies from a website, and below you can see two companies url's of the same info requsted :

url for Company "A1" :

url = 'http://www.example.com/wps/portal/!ut/p/c1/lYuxDoIwGAYf6f8oVMIoDLVIGrTQ0C6mAxqiFAejr2_dXNSYG--OHEWCv08nf5uW4C80kFsdFCqzlxmDSIoEcqt3Rc_WDHUevX3z0ByQWWO6qhEpgL9u0Rr-ujXTbZ1Csh-3HgOpzTKPZMnlH8uSU0e2jJH9Gl3nfnio4_kJwpcKOw!!/dl2/d1/L3dJMjIyQSEhL3dQRUJGUUJnTlFCaGpRQmhyUUJoSFFCZ1BRQmdBQSEhL1lJNXcvN19OMENWUkk0MjBHMTkxMElLU1E5VTJBMjBCNQ!!/?CompanyName=A1&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/'

url for Company "A2":

url = 'http://www.example.com/wps/portal/!ut/p/c1/lYuxDoIwGAYf6f8oVMKomNQiadBCQ7uYDmiIUhyMvr51c1Fjbrw7chQJ_j6e_G2cg79QT25xUCjNXmYMIikSyK3eFR1bMlR59PbNQ3NAZrVpy1qkAP66RWP469ZMN1UKyX7cegikNvM0kCWXfyxXnFqy6xjZr9F16vqHOp6fJIF9Sw!!/dl2/d1/L3dJMjIyQSEhL3dQRUJGUUJnTlFCaGpRQmhyUUJoSFFCZ1BRQmdBQSEhL1lJNXcvN19OMENWUkk0MjBHMTkxMElLU1E5VTJBMjBCNQ!!/?CompanyName=A2&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/'

As you can see the url keeps changing before this part:

/?CompanyName=A2&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/

and after this part:

http://www.example.com/wps/portal/!ut/p/c1/

and what i want is to ignore any changing part in the url and only pass the company names below:

Companies = ['A1', 'A2', 'A3']

as exapmle , of course it will be with a for loop:

url = 'http://www.example.com/wps/portal/!ut/p/c1/04_SB8K8xLLM9MSSzPy8xBz9CP0os3g_A-ewIE8TIwN3Q0tDA0_v4EDLUCNHIwMvc6B8JJK8QbCpgYGniU9YiLOPu7GBgQFJut0DwkxBuoONggO8jA08jQjo9vPIz03Vj9SPMsepyslUP0Q_0hWoKBKvooLc0IhyQ91AAHb2Eas!/dl2/d1/L0lDUmlTUSEhL3dHa0FKRnNBL1lCUlp3QSEhL2Vu/?typeofsearch=1&company=&CompanyName=' + companies +'A1&tabOrder=1&x=0&y=0'

How can i do it ?

Basically, you want to get a subString from the url string, where companyName = something, and do a loop on which you add to an Array of Strings each found company name ? :) — Chaibi Alaa, Oct 27 '15 at 22:45
what research have you done? Are you aware of the python libraries for parsing and creating URLs? — Bryan Oakley, Oct 27 '15 at 22:52
sorry for not replying earlier, yes Peter i asked it before and i was having 3 websites at the time for parsing and the code work on two of them and i thought it worked for the third and important one , but the important website didn't work. that's why i asked here for another way and an easy solution here .. ""only passing the company name and ignore everything else"" .. sorry if you saw it dublicate .. but i didn't meant it to be .. — T.M, Oct 28 '15 at 20:53

Harish Talanki · Accepted Answer · 2015-10-27T23:25:43.490

You can do it without using regular expressions,

url = 'http://www.example.com/wps/portal/!ut/p/c1/lYuxDJBMjBCNQ!!/?CompanyName=A1&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/'
url = url.replace('Name=A1', 'Name=')
compaines = ['a1','a2','a3']
for ele in compaines:
    start = url.find("CompanyName=") + 12
    newUrl1 = url[:start] + ele
    newUrl2 = newUrl1 + url[start:]
    print newUrl2

Note: I have simplified the URL for testing purpose.

If you run the above program, you will get the below output,

    http://www.example.com/wps/portal/!ut/p/c1/lYuxDJBMjBCNQ!!/?CompanyName=a1&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/
http://www.example.com/wps/portal/!ut/p/c1/lYuxDJBMjBCNQ!!/?CompanyName=a2&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/
http://www.example.com/wps/portal/!ut/p/c1/lYuxDJBMjBCNQ!!/?CompanyName=a3&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/

@PeterWood this solution worked perfectly , Thanks Peter for your previous answer and i appreciate it .. this solution is what i was searching for it was so simple that it works for a couple of websites.. Thanks everyone for your help. — T.M, Oct 28 '15 at 21:16

score 0 · Answer 2 · answered Oct 27 '15 at 22:56

l = ['http://www.example.com/wps/portal/!ut/p/c1/lYuxDoIwGAYf6f8oVMKomNQiadBCQ7uYDmiIUhyMvr51c1Fjbrw7chQJ_j6e_G2cg79QT25xUCjNXmYMIikSyK3eFR1bMlR59PbNQ3NAZrVpy1qkAP66RWP469ZMN1UKyX7cegikNvM0kCWXfyxXnFqy6xjZr9F16vqHOp6fJIF9Sw!!/dl2/d1/L3dJMjIyQSEhL3dQRUJGUUJnTlFCaGpRQmhyUUJoSFFCZ1BRQmdBQSEhL1lJNXcvN19OMENWUkk0MjBHMTkxMElLU1E5VTJBMjBCNQ!!/?CompanyName=A2&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/','http://www.example.com/wps/portal/!ut/p/c1/lYuxDoIwGAYf6f8oVMIoDLVIGrTQ0C6mAxqiFAejr2_dXNSYG--OHEWCv08nf5uW4C80kFsdFCqzlxmDSIoEcqt3Rc_WDHUevX3z0ByQWWO6qhEpgL9u0Rr-ujXTbZ1Csh-3HgOpzTKPZMnlH8uSU0e2jJH9Gl3nfnio4_kJwpcKOw!!/dl2/d1/L3dJMjIyQSEhL3dQRUJGUUJnTlFCaGpRQmhyUUJoSFFCZ1BRQmdBQSEhL1lJNXcvN19OMENWUkk0MjBHMTkxMElLU1E5VTJBMjBCNQ!!/?CompanyName=A1&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/']

import re
for el in l:
    mo = re.search(r'CompanyName=(.*?)&', el)
    if mo:
        print(mo.group(1))

A2
A1

Parsing a url in Python

2 Answers2