How to log on into Costco.com using Python requests

Question

I'm trying to automate log-in into Costco.com to check some member only prices. I used dev tool and the Network tab to identify the request that handles the Logon, from which I inferred the POST URL and the parameters. Code looks like:

import requests

s = requests.session()

payload = {'logonId': 'email@email.com',
           'logonPassword': 'mypassword'
          }

#get this data from Google-ing "my user agent"
user_agent = {"User-Agent" : "myusergent"}

url = 'https://www.costco.com/Logon'
response = s.post(url, headers=user_agent,data=payload)
print(response.status_code)

When I run this, it just runs and runs and never returns anything. Waited 5 minutes and still running. What am I going worng?

I don't know much about this, but perhaps Costco.com wants to prevent such code so that attackers can't brute-force their way in. — theknightD2, Jan 23 '21 at 22:53
@theknightD2 I don't know, might be the case. Just want to make sure I'm not overlooking anything in the code. — ADJ, Jan 23 '21 at 22:54
@ADJ Is the `User-Agent` the only request header the URL cares about? — Paul M., Jan 23 '21 at 22:56
@PaulM.Not sure what "care for" means in this context (mandatory vs optional headers), but there are plenty more request header in the Request Headers tab. Not sure which ones matter. — ADJ, Jan 23 '21 at 23:05

score 0 · Answer 1 · answered Jan 25 '21 at 00:15

0

maybe you should try to make a get requests to get some cookies before make the post requests, if the post requests doesnt work, maybe you should add a timeout so the script stop and you know that it doesnt work.

r = requests.get(w, verify=False, timeout=10)

answered Jan 25 '21 at 00:15

edouardv

18
3

Tried passing cookies from a get request to no avail. Still not returning anything. – ADJ Jan 25 '21 at 04:20
did you try to add timeout ? to add headers ? please post what's the payload you use. – edouardv Jan 25 '21 at 17:09

bbd108 · Answer 2 · 2021-02-13T05:55:19.687

This one is tough. Usually, in order to set the proper cookies, a get request to the url is first required. We can go directly to https://www.costco.com/LogonForm so long as we change the user agent from the default python requests one. This is accomplished as follows:

import requests

agent = (
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) "
    "Chrome/85.0.4183.102 Safari/537.36"
)

with requests.Session() as s:
    headers = {'user-agent': agent}
    s.headers.update(headers)
    
    logon = s.get('https://www.costco.com/LogonForm')
    # Saved the cookies in variable, explanation below
    cks = s.cookies

Logon get request is successful, ie status code 200! Taking a look at cks:

print(sorted([c.name for c in cks]))
['C_LOC',
 'CriteoSessionUserId',
 'JSESSIONID',
 'WC_ACTIVEPOINTER',
 'WC_AUTHENTICATION_-1002',
 'WC_GENERIC_ACTIVITYDATA',
 'WC_PERSISTENT',
 'WC_SESSION_ESTABLISHED',
 'WC_USERACTIVITY_-1002',
 '_abck',
 'ak_bmsc',
 'akaas_AS01',
 'bm_sz',
 'client-zip-short']

Then using the inspect network in google chrome and clicking login yields the following form data for the post in order to login. (place this below cks)

data = {'logonId': username,
'logonPassword': password,
'reLogonURL': 'LogonForm',
'isPharmacy': 'false',
'fromCheckout': '',
'authToken': '-1002,5M9R2fZEDWOZ1d8MBwy40LOFIV0=',
'URL':'Lw=='}

login = s.post('https://www.costco.com/Logon', data=data, allow_redirects=True)

However, simply trying this makes the request just sit there and infinitely redirect.

Using burp suite, I stepped into the post and and found the post request when done via browser. This post has many more cookies than obtained in the initial get request.

Quite a few more in fact

# cookies is equal to the curl from burp, then converted curl to python req
sorted(cookies.keys())
['$JSESSIONID',
 'AKA_A2',
 'AMCVS_97B21CFE5329614E0A490D45%40AdobeOrg',
 'AMCV_97B21CFE5329614E0A490D45%40AdobeOrg',
 'C_LOC',
 'CriteoSessionUserId',
 'OptanonConsent',
 'RT',
 'WAREHOUSEDELIVERY_WHS',
 'WC_ACTIVEPOINTER',
 'WC_AUTHENTICATION_-1002',
 'WC_GENERIC_ACTIVITYDATA',
 'WC_PERSISTENT',
 'WC_SESSION_ESTABLISHED',
 'WC_USERACTIVITY_-1002',
 'WRIgnore',
 'WRUIDCD20200731',
 '__CT_Data',
 '_abck',
 '_cs_c',
 '_cs_cvars',
 '_cs_id',
 '_cs_s',
 '_fbp',
 'ajs_anonymous_id_2',
 'ak_bmsc',
 'akaas_AS01',
 'at_check',
 'bm_sz',
 'client-zip-short',
 'invCheckPostalCode',
 'invCheckStateCode',
 'mbox',
 'rememberedLogonId',
 's_cc',
 's_sq',
 'sto__count',
 'sto__session']

Most of these look to be static, however because there are so many its hard to tell which is which and what each is supposed to be. It's here where I myself get stuck, and I am actually really curious how this would be accomplished. In some of the cookie data I can also see some sort of ibm commerce information, so I am linking Prevent Encryption (Krypto) Of Url Paramaters in IBM Commerce Server 6 as its the only other relevant SO answer question pertaining somewhat remotely to this.

Essentially though the steps would be to determine the proper cookies to pass for this post (and then the proper cookies and info for the redirect!). I believe some of these are being set by some js or something since they are not in the get response from the site. Sorry I can't be more help here.

If you absolutely need to login, try using selenium as it simulates a browser. Otherwise, if you just want to check if an item is in stock, this guide uses requests and doesn't need to be logged in https://aryaboudaie.com/python/technical/educational/2020/07/05/using-python-to-buy-a-gift.html

How to log on into Costco.com using Python requests

2 Answers2