0

I am trying to run some automatic scripts on a webpage with selenium and python, but the problem I'm having is that the webpage is loaded using document.write(). I have to find some elements, but they are not shown because when I view the source, it is shown as document.write(A lot of JS) instead of the html.

How can I do it so that I can view the HTML source code? I know there is the function driver.execute_script(), but I have to specify the script to run, and I don't think it will work. The page is correctly rendered, only problem is the source cannot be parsed.

Hamperfait
  • 465
  • 1
  • 6
  • 18
  • "*I have to find some elements, but they are not shown because when I view the source, it is shown as document.write(A lot of JS) instead of the html.*" do you mean that happens when *you* look at the source code or when Selenium gets to the page? If *you* want to see the current code, then you should use the developer tools (F12 usually) as opposed to opening the page source in the browser. If Selenium is getting the page as a bunch of un-executed JS, then it probably does it too soon and you need to wait for it to be rendered properly. – VLAZ Feb 20 '19 at 08:49
  • The webdriver does show the correct page, but driver.page_source shows me the document.write(). When I look at the source code with the browser it also shows me the document.write() code, so I'd say it's not a problem that I have to wait, it's just how it works, the source code contains the JS with the document.write, but the thig is that it's the browser that renders it. – Hamperfait Feb 20 '19 at 09:20
  • I've only used Selenium in Java so I'm not sure if `driver.page_source` is even in that but I *think* you shouldn't be using that. Normally, you would grab a page via the driver and instruct the driver to lookup elements for you. Selenium should internally be using the rendered page. The page source in the browser (I suspect it's the same with the driver) is in many cases what you *initially* get as the page code and *before* the browser interprets it as the page, thus before any JS executions that might manipulate the DOM. The dev tools in the browser show you the current state. – VLAZ Feb 20 '19 at 09:47
  • I have searched for the input box with selenium (findbyid, findbyname, all possibilities), but it won't find the box, although it is correctly rendered, so my guess is that it searches within the source code pre-render, not post-render, as you suggest (and I hoped it would). But it seems selenium is not using the dev tools but the source code... Thanks anyway for the responses! – Hamperfait Feb 20 '19 at 10:52
  • That would be weird but it seems in line with what you've found. I've never used the Python version of Selenium, so I'm not sure if that's the problem. Although if anything, I'd suspect the webdriver. As a guess - try swapping the driver for a different one. Or updating it? Dunno. – VLAZ Feb 20 '19 at 12:30
  • I found my problem. The page was using a lot of iframes, which selenium can't handle as one frame, so all I had to do was switch to the desired frame and voilá, problem solved. – Hamperfait Feb 21 '19 at 06:51

1 Answers1

0

As it turns out after some digging into the code, Selenium does the search in the rendered final view, but the problem was not the document.write(), but the fact that the field I was looking for was in an iframe, which selenium could not find on the default frame.

All I had to do was search through the iframes and find the ones that I needed.

Hamperfait
  • 465
  • 1
  • 6
  • 18