1

I am trying to scrape an internal web page with BeautifulSoup. The web page itself is only open to certain users on our network and uses integrated Windows authentication. I have a simple script below that works just fine, but I have to provide credentials.

import requests
from requests_ntlm import HttpNtlmAuth
from bs4 import BeautifulSoup

url = "internal page"
content = requests.get(url, auth=HttpNtlmAuth('domain\\myusername', 'mypw'))

result = BeautifulSoup(content.text, 'html.parser')

print(result)

Is it possible for me to run this in such a way that I don't have to provide my credentials, using integrated Windows authentication?

Thank you.

Mike
  • 399
  • 2
  • 9
  • 19

1 Answers1

2

There is a closed issue at requests-ntlm repo with a question:

Is there a way to authenticate with currently logged user's credentials instead of providing login/password directly inside the script?

And response from author of the lib:

Not as far as I know....

Also take a look at this question. There are some alternative ways to store your secrets.

Personally I prefer environment variables and something like secrets.py:

import os

user = os.getenv('MY_USER')
password = os.getenv('MY_PASSWORD')

And then you could import your secrets from there:

from secrets import user, password

Another non so lightweight option is to use playwriht or selenium to open real browser, which might be able to load page without hardcoded login and password

Alex Kosh
  • 2,206
  • 2
  • 19
  • 18