Submitting a Form using Selenium in Python

Question

I need to scrape some data behind those hyperlinks from this Site. However, those hyperlinks are javascript function calls, which later submits a form using post method. After some search, selenium seems to be a candidate. So my question is that how should I properly set a value to an input tag and submit the form which does not a submit a button.

from selenium import webdriver

url = "http://www.echemportal.org/echemportal/propertysearch/treeselect_input.action?queryID=PROQ3h3n"
driver = webdriver.Firefox()
driver.get(url)
treePath_tag = driver.find_element_by_name("treePath")

Before submitting the form, I need to assign value to tag <input>. However, I got an error

Message: Element is not currently visible and so may not be interacted with

treePath_tag.send_keys('/TR.SE00.00/QU.SE.DATA_ENV/QU.SE.ENV_ENVIRONMENT_DATA/QU.SE.EN_MONITORING')

IF above is correct, I would like to submit form this way. Is it correct?

selenium.find_element_by_name("add_form").submit()

Below are sources from the web page.

JavaScript function

<script type="text/javascript">
    function AddBlock(path){
        document.add_form.treePath.value=path;
        document.add_form.submit();
    }
</script>

form "add_form"

<form id="addblock_input" name="add_form" action="/echemportal/propertysearch/addblock_input.action" method="post" style="display:none;">
<table class="wwFormTable" style="display:none;"><tr style="display:none;">
  <td colspan="2">
<input type="hidden" name="queryID" value="PROQ3h1w" id="addblock_input_queryID"/>  </td>
</tr>
<tr style="display:none;">
  <td colspan="2">
<input type="hidden" name="treePath" value="" id="addblock_input_treePath"/>  </td>
</tr>
</table></form>

div with javascript call

<div id="querytree">
    <h1>Property Search</h1>
    <h2>Select Query Block Type</h2>
    <p>Select a section for which to define query criteria.</p>
    <div class="queryblocktools"><a href="javascript:document.load_form.submit();"><img style="vertical-align:top;" alt="Load" src="/echemportal/etc/img/load.gif"/>&nbsp;Load Query</a></div>
    <ul class="listexpander">   
    <li>Physical and chemical properties<ul>
        <li><a href="javascript:AddBlock('/TR.SE00.00/QU.SE.DATA_PHYS/QU.SE.PC_MELTING');">Melting point/freezing point</a></li>
        <li><a href="javascript:AddBlock('/TR.SE00.00/QU.SE.DATA_PHYS/QU.SE.PC_BOILING');">Boiling point</a></li>
    </ul>
</div>

Have you tried [this](http://stackoverflow.com/questions/27927964/selenium-element-not-visible-exception/27931726#27931726) with `execute_script`? — Anzel, Jul 22 '16 at 08:10
Y should you want to assign value to a hidden field? won't display in the browser GUI — Aishu, Jul 22 '16 at 08:33

Saurabh Gaur · Accepted Answer · 2016-07-22T16:57:23.303

7

You are trying to set value on hidden input is which not visible on the page, that's why error has occurred. If you want to set value on hidden field try using execute_script as below :-

treePath_tag = driver.find_element_by_name("treePath")
driver.execute_script('arguments[0].value = arguments[1]', treePath_tag, '/TR.SE00.00/QU.SE.DATA_ENV/QU.SE.ENV_ENVIRONMENT_DATA/QU.SE.EN_MONITORING')

After setting value on hidden field you can use following to submit the form :-

selenium.find_element_by_name("add_form").submit()

Hope it helps..:)

edited Jul 22 '16 at 16:57

answered Jul 22 '16 at 08:25

Saurabh Gaur

23,507
10
54
73

Thanks for your suggestion! What is the meaning of `arguments[0].value=argument[1]' since I got this error `selenium.common.exceptions.WebDriverException: Message: arguements is not defined` – TTT Jul 22 '16 at 16:21
@tao.hong sorry its `arguments[1]`..just typo mistake..here arguments[0] is your hidden element and arguments[1] is your value which you want to set...:) – Saurabh Gaur Jul 22 '16 at 16:59
No worries. But I still think it requires to define `arguments` since I got this msg `selenium.common.exceptions.WebDriverException: Message: arguements is not defined` – TTT Jul 22 '16 at 18:20
Thanks again. But this time, it looks the program was halted... It opened firefox then my python console did not get any return/response... – TTT Jul 22 '16 at 18:30
@tao.hong Is there any exception?? WHAT you expecte in return?? could you explain more?? – Saurabh Gaur Jul 22 '16 at 18:44
Sure. There is no exception, which is quite strange. I would like to fill that hidden input tag and submit the form, which will go to a different page to fill another form and scrap some data. Thanks again! – TTT Jul 22 '16 at 18:58
SO how did you know the code didn't fill the data in hidden field while its not visible?? – Saurabh Gaur Jul 22 '16 at 19:00
Here is how I plan to check. `driver.execute_script('arguments[1].value = arguments[1]', treePath_tag, '/TR.SE00.00/QU.SE.DATA_ENV/QU.SE.ENV_ENVIRONMENT_DATA/QU.SE.EN_MONITORING') treePath_tag2 = driver.find_element_by_name("treePath") print treePath_tag2.get_attribute('value')` – TTT Jul 22 '16 at 19:03
@tao.hong Your are executing wrong script try as :- `driver.execute_script('arguments[0].value = arguments[1]', treePath_tag, '/TR.SE00.00/QU.SE.DATA_ENV/QU.SE.ENV_ENVIRONMENT_DATA/QU.SE.EN_MONITORING')`... here first is `arguments[0]` and another is `arguments[1]`..then get and print.. – Saurabh Gaur Jul 22 '16 at 19:10
Sorry, my mistake. Any reference for those `arguments`? just try to understand this better. thanks so much for your help! – TTT Jul 22 '16 at 19:13
Ok I'm trying make it clear...in selenium `execute_script()` method expect two parameters where first is JavaScript expression as string and another is array of string...here in the second parameter which array of string is used as input for the java script expression...in JavaScript expression it suppose array of string input in `arguments` variable, means array..as you see in your case in execute_script three param passed where first is JavaScript expression and other two is arguments array where arguments[0] is your hidden element and another arguments[1] is your value.. – Saurabh Gaur Jul 22 '16 at 19:23
Thanks so much for explaining this process! – TTT Jul 22 '16 at 19:26
@tao.hong Have a look for more details..https://seleniumhq.github.io/selenium/docs/api/java/org/openqa/selenium/JavascriptExecutor.html.. – Saurabh Gaur Jul 22 '16 at 19:26

Submitting a Form using Selenium in Python

JavaScript function

form "add_form"

div with javascript call

1 Answers1

Linked