2

I have a webpage that contains some information that I am interested in. However, those information are generated by Javascript.

If you do something similar like below:

browser = webdriver.Chrome() 
browser.set_window_size(1000, 1000)    
browser.get('https://www.xxx.com')  # cannot make the web public, sorry
print browser.page_source

It only print out a few javascript functions and some headers which doesn't contain that information that I want - Description of Suppliers, etc... So, when I try to collect those information using Selenium, the browser.find_element_by_class_name would not find the element I want successfully either. I tried the code below assuming it would has the same effect as typing document in the javascript console, but obviously not. 

result = browser.execute_script("document")
print result

and it returns NULL...

However, if I open up the page in Chrome, right click the element and inspect element. I could see the populated source code. See the attached picture.

Also, I was inspired by this commend that helps a lot. I could open up the javascript console in Chrome, and if I type in 

document

I could see the complete html sitting there, which is exactly what I want. I am wondering is there a way to store the js populated source code using selenium?

I've read some posts saying that it requires some security work to store the populated document to client's side.

enter image description here

Hope I have made myself clear and appreciates any suggestion or correction. (Note, I have zero experience with JS so detailed explaination would be gratefully appreciated!)

Community
  • 1
  • 1
B.Mr.W.
  • 18,910
  • 35
  • 114
  • 178
  • What...have you tried? Why don't you try to do this, then come back when you're stuck? – Marcin Aug 19 '13 at 17:55
  • result = browser.execute_script("document"); print result and it returns null, I am not sure where the data is stored.. I assume it run the javascript code: document and get the complete populated html but obviously not... – B.Mr.W. Aug 19 '13 at 18:01
  • Put that *in your question*. Also, why is that what you have tried? What did you do to actually retrieve the document? – Marcin Aug 19 '13 at 18:03
  • @Marcin I have modified my question and hope it is clear enough. Let me know what do you think – B.Mr.W. Aug 19 '13 at 18:24
  • Well `page_source` should print everything. If not, then there is an issue. What version of Selenium and Firefox are you running? Why are you not finding elements and then getting their text values? What is this information that you cannot get by going *directly* to a specific element. – Arran Aug 19 '13 at 18:31
  • @Arran I am using selenium-2.33.0-py2.7, and I am using Chrome(becuase I would get some weird error a is null...etc in Firefox). As I mentioned in my post, I cannot find the elements in selenium... and print page_source doesn't work, it only prints out a few javascript functions...without the populated html, which is exactly what I want. – B.Mr.W. Aug 19 '13 at 18:36

0 Answers0