0

For privacy concerns, I cannot distribute the url publicly.

I have been able to access this site successfully using python requests session = requests.Session(); r = session.post(url, auth = HttpNtlmAuth(USERNAME, PASSWORD), proxies = proxies) which works great and I can parse the webpage with bs4. I have tried to return cookies using session.cookies.get_dict() but it returns an empty dict (assuming b/c site is hosted using sharepoint). My original thought was to retrieve cookies then use them to access the site.

The issue that I'm facing is when you redirect to the url, a box comes up asking for credentials - which when entered directs you to the url. You can not inspect the page that the box is on- which means that I can't use send.keys() etc. to login using selenium/chromedriver.

I read through some documentation but was unable to find a way to enter pass/username when calling driver = webdriver.Chrome(path_driver) or following calls.

Any help/thoughts would be appreciated.

When right clicking the below - no option to inspect webpage.

enter image description here

Zachary Wyman
  • 299
  • 2
  • 11
  • Are you using selenium or requests? Pick one, don't mix them. – Code-Apprentice Mar 02 '21 at 18:57
  • Read above - I tried using ```requests``` to pass cookies to ```selenium``` and then log in. There is no way to use ```selenium``` to my knowledge to ```send.keys``` to a site that does not provide html inspection, etc. I also need to interact with the site, so I need to use a webdriver to perform tasks. I.e. there may be no way around it except combining the two. If you have knowledge of some good ```http``` authentications that work well with ```selenium``` please direct them my way. – Zachary Wyman Mar 02 '21 at 19:01
  • What do you mean by "a site that does not provide html inspection"? If you open any webpage in a browser, you can inspect the html. – Code-Apprentice Mar 02 '21 at 19:04
  • The site is blocked behind authentication. There is no option to inspect the page. Once you provide authentication, sure you can inspect the webpage - but not until credentials have been entered. I'm assuming its similar for other sites on sharepoint. – Zachary Wyman Mar 02 '21 at 19:09
  • "blocked behind authentication" is not the same as "does not provide html inspection". You just need to log in, then you can otherwise inspect the webpage. – Code-Apprentice Mar 02 '21 at 19:10
  • Mm, I have searched every bit of the html, there are no keys to provide login credentials. – Zachary Wyman Mar 02 '21 at 19:10
  • A page that asks for a username and password must have a URL and must have HTML that defines the `` elements. – Code-Apprentice Mar 02 '21 at 19:10
  • Here's the process that happens: give webdriver the url - redirects you to a blank white page that does not have a url above - simply a box appears asking for user/pass and once entered, you are directed to the site w/ url appearing. It's confusing as I would think the same as you originally – Zachary Wyman Mar 02 '21 at 19:13
  • Will you show a screenshot of the box that asks for user/pass? – Code-Apprentice Mar 02 '21 at 19:14
  • I'm guessing it is a login dialog similar to what you get when the server has a `.htaccess` file. This isn't HTML. Maybe https://stackoverflow.com/questions/29516740/how-to-access-popup-login-form-with-selenium-in-python answers your question? – Code-Apprentice Mar 02 '21 at 19:16
  • I will give it a look - thank you – Zachary Wyman Mar 02 '21 at 19:17
  • Does this answer your question? [How to access popup login form with Selenium in Python](https://stackoverflow.com/questions/29516740/how-to-access-popup-login-form-with-selenium-in-python) – Code-Apprentice Mar 02 '21 at 19:18
  • Now that I understand better what you are seeing. This is a browser-level dialog. It isn't rendered in HTML, so it isn't accessible by either requests or selenium for parsing. Instead, you have to pass the credentials directly. See the linked answer for details. – Code-Apprentice Mar 02 '21 at 19:19
  • 1
    Hmm interesting. I was actually able to login using ```r = session.post(url, auth = HttpNtlmAuth(USERNAME, PASSWORD), proxies = proxies)``` and then parse with ```bs4```. Is that capability due to the ```HttpNtlmAuth``` package? – Zachary Wyman Mar 02 '21 at 19:21
  • I will add that the url extension is ```.aspx``` if that provides any clarity. – Zachary Wyman Mar 02 '21 at 19:26
  • Unfortunately, the linked answer does not provide the right solution for this particular website. Will look into accessing ```.aspx``` url extensions. – Zachary Wyman Mar 02 '21 at 19:37
  • 1
    Googling "python selenium ntlm authentication" lead me to https://stackoverflow.com/questions/56251199/how-to-by-pass-ntlm-authentication-pop-up-while-performing-automation-testing-us which might help you here. – Code-Apprentice Mar 02 '21 at 20:04
  • 1
    Hah - using an encoded separator ```%5C``` for the request did the trick. Thank you! – Zachary Wyman Mar 02 '21 at 20:34
  • You should close this question as a duplicate of the one that gave the answer. That way future visitors can find the answer more easily, too. – Code-Apprentice Mar 02 '21 at 20:59

0 Answers0