1

I've just started to learn coding this month and started with Python. I would like to automate a simple task (my first project) - visit a company's career website, retrieve all the jobs posted for the day and store them in a file. So this is what I would like to do, in sequence:

  1. Go to http://www.nov.com/careers/jobsearch.aspx
  2. Select the option - 25 Jobs per page
  3. Select the date option - Today
  4. Click on Search for Jobs
  5. Store results in a file (just the job titles)

I looked around and found that Selenium is the best way to go about handling .aspx pages.

I have done steps 1-4 using Selenium. However, there are two issues:

  1. I do not want the browser opening up. I just need the output saved to a file.
  2. Even if I am ok with the browser popping up, using the Python code (exported from Selenium as Web Driver) on IDLE (i have windows OS) results in errors. When I run the Python code, the browser opens up and the link is loaded. But none of the form selections happen and I get the foll error message (link below), before the browser closes. So what does the error message mean? https://i.stack.imgur.com/lmcDz.png

Any help/guidance will be appreciated...Thanks!

user2226459
  • 11
  • 1
  • 3

2 Answers2

1

First about the error you've got, I should say that according to the expression NoSuchElementException and the message Unable to locate element, the selector you provided for the web-driver is wrong and web-driver can't find the element.

Well, since you did not post your code and I can't open the link of the website you entered, I can just give you a sample code and I will count as much details as I can.

from selenium import webdriver

driver = webdriver.Firefox()
driver.get("url")
number_option = driver.find_element_by_id("id_for_25_option_indicator")
number_option.click()
date_option = driver.find_element_by_id("id_for_today_option_indicator")
date_option.click()
search_button = driver.find_element_by_id("id_for_search_button")
search_button.click()
all_results = driver.find_elements_by_xpath("some_xpath_that_is_common_between_all_job_results")

result_file = open("result_file.txt", "w")

for result in all_results:
    result_file.write(result.text + "\n")

driver.close()
result_file.close()

Since you said you just started to learn coding recently, I think I have to give some explanations:

  1. I recommend you to use driver.find_element_by_id in all cases that elements have ID property. It's more robust.
  2. Instead of result.text, you can use result.get_attribute("value") or result.get_attribute("innerHTML").

That's all came into my mind by now; but it's better if you post your code and we see what is wrong with that. Additionally, it would be great if you gave me a new link to the website, so I can add more details to the code; your current link is broken.

Zeinab Abbasimazar
  • 9,835
  • 23
  • 82
  • 131
0

Concerning the first issue, you can simply use a headless browser. This is possible with Chrome as well as Firefox. Check Grey Li's answer here for example: Python - Firefox Headless

from selenium import webdriver
options = webdriver.FirefoxOptions()
options.add_argument('headless')
driver = webdriver.Firefox(options=options)
Ideophagous
  • 1
  • 2