0

I'm using Python and Selenium in PyCharm to go to the SEC website to download a 10-K CSV file. Ideally, the program should ask for user input for a "ticker symbol", then go to the SEC's website, input the ticker symbol provided and download the 10-K and 10-Q CSV files from the page. I was using Microsoft's ticker symbol (MSFT) as an example test. The SEC's Edgar search website is this:

https://www.sec.gov/edgar/searchedgar/companysearch.html

and I am using the 'Fast Search' search engine. I created a function 'get_edgar_results' to perform this download. It might be that I'm new to web scraping, but I thought I identified the HTML tags correctly on where to put my search term. Previous problems suggested that I might need to have the program wait before searching for the HTML element, so I added code for the program to wait. I continue getting this error: line 242, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: [id="Find"]

My code is below:

import selenium.webdriver.support.ui as ui
from pathlib import Path
import selenium.webdriver as webdriver
ticker_symbol = input("please provide a ticker symbol: ")

def get_edgar_results(ticker_symbol):
    url = "https://www.sec.gov/edgar/searchedgar/companysearch.html"  
    driver = webdriver.Firefox(executable_path=r"C:\Program Files\JetBrains\geckodriver.exe") 
    wait = ui.WebDriverWait(driver,30) 
    driver.set_page_load_timeout(30) 
    driver.get(url)
    search_box = driver.find_element_by_id("Find")
    search_box.send_keys(ticker_symbol)
    search_box.submit()
    annual_links = driver.find_elements_by_class_name("10-K")
    quarterly_links = driver.find_elements_by_class_name("10-Q")
    results = []
    driver.close()
    driver.quit()
    return results


get_edgar_results(ticker_symbol)

Any help would be greatly appreciated.

Fred B
  • 1
  • 4
  • Which is the element referred as `[id="Find"]`? With in **Company Name** or within **Fast Search**? What text do you intent to send? – undetected Selenium Jun 19 '20 at 14:32
  • within Fast Search. I was hoping to put the four-letter ticker symbol into Fast Search. So if you put 'MSFT' for Microsoft into Fast Search, you'd get this as your search result. https://www.sec.gov/cgi-bin/browse-edgar?CIK=MSFT&owner=exclude&action=getcompany&Find=Search – Fred B Jun 19 '20 at 16:49
  • @FredB Is the problem solved? – AzyCrw4282 Jun 21 '20 at 18:37
  • @AzyCrw4282 Sorry, I'm still learning python, and it appears I have to put your code into python syntax and use the variables I defined. When you referenced 'lesscompany' I don't know what that's referring to, as well as 'yourvalue'. I thought my code was clear about defining the value for ticker_symbol, but I probably should have said that my first example, ticker_symbol = MSFT. I didn't have as much time over the weekend to work on this as I'd hoped. I hope to get to complete this this afternoon. – Fred B Jun 22 '20 at 08:01
  • `When you referenced 'lesscompany' I don't know what that's referring to, as well as 'yourvalue'?` This is the input box in which you have to send the value before you make the button click. The id is in reference to the box and the `yourvalue` is what you wanna send to that box. Try it and lmk if you are stuck – AzyCrw4282 Jun 22 '20 at 19:27
  • @AzyCrw4282 - I get this error using the code as you suggested. ``` line 22 driver.findElement(By.xpath("//*[@id="lesscompany"])).sendKeys("your value") ^ SyntaxError: invalid syntax``` I had substituted these lines in my original code '' search_box = driver.find_element_by_id("Find") search_box.send_keys(ticker_symbol) search_box.submit() ``` for the lines you suggested and got the error. – Fred B Jun 23 '20 at 09:34
  • Oh Sorry... it seems like due to a missing a quotation mark, so should be `driver.findElement(By.xpath("//*[@id="lesscompany"]")).sendKeys("your value") ` – AzyCrw4282 Jun 23 '20 at 19:47
  • this code is still giving the 'invalid syntax' error 'driver.findElement(By.xpath("//*[@id="lesscompany"])).sendKeys("your value")' – Fred B Jun 24 '20 at 08:12
  • @AzyCrw4282 This line worked, ' ` driver.set_page_load_timeout(30) driver.get(url) search_box = driver.find_element_by_xpath('//*[@id="cik"]') search_box.send_keys(ticker_symbol) search_box.submit()`' so maybe the problem was that I had invisible spaces in my code? – Fred B Jun 24 '20 at 11:38
  • Likely. Where are you with the problem and have you solved it? – AzyCrw4282 Jun 24 '20 at 19:14
  • It looks like its solved. No further issues. – Fred B Jun 25 '20 at 09:15
  • @FredB please accept an answer if it has solved your problem. see [here](https://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work) for more information – AzyCrw4282 Jul 02 '20 at 07:08

1 Answers1

0

Consider using a waitUntil or Implicit/Explicit waits method to wait until an element is loaded. This way you can circumvent the error shown above, code below for wait until method. 

browser = webdriver.Firefox()
browser.get("url")
delay = 3 # seconds
try:
    myElem = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'IdOfMyElement')))
    print "Page is ready!"
except TimeoutException:
    print "Loading took too much time!"

You also seem to have an error in the following code

search_box = driver.find_element_by_id("Find")
search_box.send_keys(ticker_symbol)
search_box.submit()

The id=find locates the Search Box and not the input element and therefore sending keys value to a button is incorrect. I would recommend you to use the xpath to uniquely locate the element of your choice.

The following will send a value to the input box and will make a button click on the SEARCH button.

driver.findElement(By.xpath("//*[@id="lesscompany"])).sendKeys("your value");
driver.findElement(By.xpath("//*[@id="search_button_1"]")).click();
AzyCrw4282
  • 7,222
  • 5
  • 19
  • 35