Python: Entering a large list into web form efficiently?

Question

I'm using modules selenium and xlrd to input many cells of data (~1000) from excel into a web form. I've successfully written code that does this using selenium's send_keys, but there is currently a large delay in time due to processing of the input before the form gets submitted. I currently compile one large string that contains the value of every cell and then use send_keys to put it into the form all at once like this:

for z in range(1000):
        ss = ss + (str(sh.row_values(z)[1])) + "\n"
form.send_keys(ss)

I've noticed that when I simply copy and paste the 1000 excel cells into the form manually, there is almost no processing delay before submission, so I imagine there is a more efficient way to accomplish what I want without using selenium's send_keys. I would appreciate any help.

Are you able to consider using a headless system and just submitting the form as an HTTP request then inspecting the response? — Silas Ray, May 11 '12 at 13:45

score 0 · Answer 1 · answered May 11 '12 at 13:51

0

If there is no javascript involved, you can try Mechanize

http://wwwsearch.sourceforge.net/mechanize/

This is how you can handle forms with Mechanize -

http://wwwsearch.sourceforge.net/mechanize/forms.html

answered May 11 '12 at 13:51

theharshest

7,767
11
41
51

I can't use Mechanize for this site, and I'd prefer to continue using Selenium. – user1382685 May 11 '12 at 14:43

score 0 · Answer 2 · edited May 23 '17 at 12:20

What you might do is consider using javascript instead to just set the values of the fields yourself.

The answer here shows how to execute javascript in python with selenium. When setting the value of text fields, the action is near instantaneous and you shouldn't get that effect of the form processing slowly.

Remember though, that as soon as you start using javascript instead of the selenium given API, you'll begin losing some of its functionality that is built in. For instance, if the page is not loaded yet and you try executing javascript, the action will fail immediately as it fails to find the element whereas Selenium will contently wait for a bit if it doesn't find the element to see if it will load eventually.

Python: Entering a large list into web form efficiently?

2 Answers2