Why isn't python's selenium button.click() working on a twitter follow button?

Question

So I know I'm locating the correct element. I've tried it with xpath and css selectors, both are correctly retrieving the correct button.

Then when I call .click() on the button, it does not follow the account.

button = self.driver.find_element(By.CSS_SELECTOR, ".user-actions-follow-button.js-follow-btn.follow-button.btn")
print(button.text)
button.click()
print('should have followed')

Does anyone know why it's behaving like this?

Edit: Here's the whole class code:

class Scraper:
    def __init__(self):
        self.driver = webdriver.PhantomJS(executable_path='phantomjs')
        self.loggedIn = False;


    def login(self, url, username, password):
        self.driver.get(url)

        try:
            usernameTextField = self.driver.find_element_by_css_selector(".js-username-field.email-input.js-initial-focus")

            passwordTextField = self.driver.find_element_by_css_selector('.js-password-field')

            usernameTextField.send_keys(username)
            passwordTextField.send_keys(password + '\n')
        except:
            print('Already logged in')

        try:
            WebDriverWait(self.driver, 10).until(
                    EC.presence_of_element_located((By.ID, 'dashboard-profile-prompt'))
            )

        except:
            self.loggedIn = False

        finally:
            print('succesfully logged in')
            self.loggedIn = True

    def followByUrl(self, url):
        if self.loggedIn:
            self.driver.get(url)

            actions = ActionChains(self.driver)
            wait = WebDriverWait(self.driver, 10)
            follow = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".user-actions-follow-button.js-follow-btn.follow-button.btn")))


            print(follow.text)

            actions.move_to_element(follow).click().perform()

            print('should have followed')

And here's a picture of the element

What error are you getting? An `Element not visible` error suggests the element isn't loaded yet, and is common on dynamically-loaded pages. — alphazwest, Dec 15 '16 at 19:35
@Frank Not getting any errors. It's also correctly printing out the button.text — joe, Dec 15 '16 at 19:36
Usually when I run into that sort of issue, it means the wrong element is receiving the action. — alphazwest, Dec 15 '16 at 19:46
@Frank I don't see this being wrong? The same method has worked for me on other scrapers also it's printing out the text correctly. http://imgur.com/a/bio5K — joe, Dec 15 '16 at 19:51

score 1 · Answer 1 · edited May 23 '17 at 12:09

1

First of all, since you are using PhantomJS, pretending not to be PhantomJS might solve the problem, see this post with a working Python/Selenium sample:

Set phantomjs user-agent string

And, make sure you are clicking the same button you are checking to be clicked. There can be multiple "Follow" buttons on a page.

Also, add an Explicit Wait to wait for the button to become clickable:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)

follow = wait.until(element_to_be_clickable((By.CSS_SELECTOR, "button.follow-button")))
follow.click()

You may also try moving to the element and then clicking via "ActionChains":

from selenium.webdriver import ActionChains

actions = ActionChains(driver)
actions.move_to_element(follow).click().perform()

Or, you can click "via JavaScript":

driver.execute_script("arguments[0].click();", follow)

And don't forget to stay on the legal side, follow the Terms of Use, be a good web-scraping citizen.

edited May 23 '17 at 12:09

Community

1
1

answered Dec 15 '16 at 19:39

alecxe

462,703
120
1,088
1,195

Tried everything, still nothing is working. Do you think twitter may be detecting the request is coming from a bot? – joe Dec 15 '16 at 19:46
@joe doubt so, though you have to follow the rules to stay on the legal side of things. What browser do you use? Could you show the complete code you have so far? Thanks. – alecxe Dec 15 '16 at 19:48
I'm using PhantomJS, going to edit with my whole class so far – joe Dec 15 '16 at 19:53
@joe ah, that might be important. Try changing the user-agent (https://coderwall.com/p/9jgaeq/set-phantomjs-user-agent-string). – alecxe Dec 15 '16 at 19:54
1

Another means of diagnosis may be to try to use chromedriver or the firefox in your webdriver instantiation to see if that resolves the issue. I have had occasions where what worked in chrome driver didn't work in phantomjs. @alecxe 's user agent suggestion would likely resolve that issue. – alphazwest Dec 15 '16 at 19:59
Changed my driver to match the link and still nothing :/ – joe Dec 15 '16 at 19:59
@joe you can also try providing the usual python+selenium+phantomjs "troubleshooting" flags (http://stackoverflow.com/questions/23581291/python-selenium-with-phantomjs-empty-page-source). – alecxe Dec 15 '16 at 19:59
@joe how about clicking two times? – alecxe Dec 15 '16 at 19:59
@joe also, can you try making the screenshot from PhantomJS via `driver.take_screenshot("screenshot.png")` before clicking the button and after? Thanks. – alecxe Dec 15 '16 at 20:00
Took a screenshot before and after, no difference. – joe Dec 15 '16 at 20:05

score 0 · Answer 2 · answered Dec 15 '16 at 20:32

0

I got it to work by switching from the PhantomJS driver to chromedriver.

answered Dec 15 '16 at 20:32

joe

1,563
4
16
35

Why isn't python's selenium button.click() working on a twitter follow button?

2 Answers2