2

I am trying to write a script to log in to my university course website (http://nalanda.bits-pilani.ac.in/login/) to automatically download all my course files. I'm using the Requests package to communicate with the server. It's working but is taking a lot of time.

This is the code I'm using:

import requests
import logging

logging.basicConfig(format='%(asctime)s %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p', level = logging.DEBUG)
with requests.Session() as c:
    url = 'http://nalanda.bits-pilani.ac.in/login/'
    USERNAME = '<my username>'
    PASSWORD = '<my password>'
    headers = {'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1'}
    login_data = dict(username = USERNAME, password = PASSWORD)
    print('Sending authentication details...')
    r = c.post(url,data = login_data)
    print('Authentication done in '+str(end-start))
    r.raise_for_status()
    print('Getting list of courses...')
    page = c.get('http://nalanda.bits-pilani.ac.in/my/')

and here is the output:

Sending authentication details...
07/17/2018 07:13:29 PM Starting new HTTP connection (1): nalanda.bits-pilani.ac.in
07/17/2018 07:15:41 PM http://nalanda.bits-pilani.ac.in:80 "POST /login/ HTTP/1.1" 303 463
07/17/2018 07:15:41 PM http://nalanda.bits-pilani.ac.in:80 "GET /login/index.php?testsession=3053 HTTP/1.1" 303 434
07/17/2018 07:15:41 PM http://nalanda.bits-pilani.ac.in:80 "GET /my/ HTTP/1.1" 200 15578
Getting list of courses...
07/17/2018 07:15:42 PM http://nalanda.bits-pilani.ac.in:80 "GET /my/ HTTP/1.1" 200 15577

It takes over 2 minutes to start the HTTP connection. All the POSTS and GETS after that work smoothly. For other websites, say http://www.google.com, everything works fine and starting the HTTP connection takes less than a second.

Any fixes/workarounds will be appreciated. Note that the website loads fast on my browser with cache cleared.

Naveen Unnikrishnan
  • 141
  • 1
  • 2
  • 7
  • 3
    I think your question might be a bit more actionable if you were to focus on how to figure out where requests is spending its time. As Is, your question is a bit broad. – Sam Hartman Jul 17 '18 at 12:49
  • Please follow [this answer](https://stackoverflow.com/a/16337639/3929826) and increase the log level to see details. – Klaus D. Jul 17 '18 at 12:50

0 Answers0