Selenium Unable to process further after executing script

Question

This question is actually continued from here I have following link I got help from community regarding using script in selenium but main part here is I want to collect atleast 2-3 news and store in some json which can further be manipulated. I am not able to click the get those news section which is in bottom part of page any way I can scrape those news at least 1-2 new?

   stocks_data=["AKPL","NICA"]
   for stock_data in stocks_data:
            self.driver.get(f'https://merolagani.com/CompanyDetail.aspx?symbol={stock_data[0].lower()}')
            self.driver.execute_script(
                "document.getElementById('ctl00_ContentPlaceHolder1_CompanyDetail1_lnkNewsTab').click()")

score 1 · Answer 1 · answered Jun 27 '22 at 15:43

1

Use stocks_data=["AKPL","NICA"] instead of stocks_data="AKPL","NICA"]

answered Jun 27 '22 at 15:43

Md. Fazlul Hoque

15,806
5
12
32

EvgenyKolyakov · Answer 2 · 2022-06-27T17:22:55.843

1

As for the multiple news, just use a loop over document.querySelectorAll('[id*="_lnkNewsTab"]') or some similar selector if that one is too wide.

As for the json, try to populate some imaginary div (that you can create in Javascript by document.body.innerHTML += '<div id="my_json"></div>') with the content of the elements you need (document.getElementById('my_json').innerHTML += document.getElementById('the id of the element you want the data from').innerHTML;) and finally get that div from python (see How to get text with Selenium WebDriver in Python)

Then you can manipulate however you like in python.

Something like:

   stocks_data=["AKPL","NICA"]
   for stock_data in stocks_data:
            self.driver.get(f'https://merolagani.com/CompanyDetail.aspx?symbol={stock_data[0].lower()}')
            self.driver.execute_script("document.getElementById('ctl00_ContentPlaceHolder1_CompanyDetail1_lnkNewsTab').click()")
            self.driver.execute_script("""
document.body.innerHTML += '<div id="my_json"></div>';
var x = document.querySelectorAll('[id*="_lnkNewsTab"]');
for (let i = x.length - 1; i >= 0; --i) {
  document.getElementById('my_json').innerHTML += x[i].innerHTML + '--DELIMITER--'
});
            """)
            content = self.driver.find_element_by_id("my_json")
            content = content.text.split('--DELIMITER--')[:-1]

edited Jun 27 '22 at 17:22

answered Jun 27 '22 at 15:59

EvgenyKolyakov

3,310
2
21
31

` ``self.driver.execute_script("document.querySelectorAll('[id*=\"_lnkNewsTab\"]')") for z in self.driver.find_elements_by_xpath("//table[@class='table table-striped table-hover']//tr//a"): print(z.click())``` I did upto here but no result – lord stock Jun 27 '22 at 16:57
I dont know js but I got this errro ` TypeError: x.map is not a function` – lord stock Jun 27 '22 at 17:12
1

Fixed, edited answer – EvgenyKolyakov Jun 27 '22 at 17:23
just a silly qsn what is content supposed to print? – lord stock Jun 27 '22 at 17:34
I just got `NEWS` printed – lord stock Jun 27 '22 at 17:36
content should hold an array of your collected "articles" – EvgenyKolyakov Jun 27 '22 at 19:55
Also, can you run `document.querySelectorAll('[id*="_lnkNewsTab"]')` in DevTool and show me what it prints ? – EvgenyKolyakov Jun 27 '22 at 19:56
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/245958/discussion-between-nava-and-evgenykolyakov). – lord stock Jun 28 '22 at 03:34

Selenium Unable to process further after executing script

2 Answers2

Linked