How to "send keys" to a canvas element for longer duration?

Question

My aim:

I am trying to make a python bot to win chrome's dino game.

The game allows 2 types of jumps:

short jumps
long jumps

Using main_body.send_keys(Keys.SPACE) (as shown in the code below) gives short jumps.

My problem:

I am having difficulty executing long jumps.

My approach:

Currently, for long jumps, I am using the Keyboard library:

keyboard.press(keyboard.KEY_UP)

This, unfortunately, requires the browser windows to be in focus all the time. Later on, I wish to run this program headlessly, so this approach won't work.

Alternatively:

I tried ActionChains:

ActionChains(driver) \
.key_down(Keys.SPACE) \
.pause(0.2) \
.key_up(Keys.SPACE) \
.perform()

But this ends up scrolling the entire page. And doesn't fulfill the intended purpose.

I simply wish to "send" these actions to the canvas element directly instead of performing them on the entire page...

I wish to do something like this:

main_body.key_down(Keys.SPACE) 
time.sleep(0.2)
main_body.key_up(Keys.SPACE)

Although this will, of course, give me this error: AttributeError: 'FirefoxWebElement' object has no attribute 'key_down' because canvas has no attribute key_down or key_up.

Here is an MCVE:

NOTE: In the code, the dino will keep jumping continuously but that is not the point, this is just to check the hights of the jumps and not to win the game.

import keyboard
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains

driver = webdriver.Firefox()
driver.get('https://chromedino.com/')
canvas = driver.find_element_by_css_selector('.runner-canvas')
main_body = driver.find_element_by_xpath("//html")

try:
    canvas.click()
except:
    main_body.send_keys(Keys.SPACE)    


while True:
    #short jump
    main_body.send_keys(Keys.SPACE)

    #long jump
    ActionChains(driver) \
    .key_down(Keys.SPACE) \
    .pause(0.2) \
    .key_up(Keys.SPACE) \
    .perform()

    #long jump using keyboard:
    keyboard.press(keyboard.KEY_UP)

Please see the effect of each type of jump by commenting out the code for others.

If possible suggest some other alternative way of getting a long jump without using Keyboard and without scrolling the entire page...

Using `ActionChains` wouldn't scroll the entire page.Works fine on my PC.(I am using `Chrome`) — jizhihaoSAMA, Jul 19 '20 at 14:12
@jizhihaoSAMA oh. I am using firefox. Let me give it a shot in chrome... — Sabito stands with Ukraine, Jul 19 '20 at 14:16

score 4 · Accepted Answer · answered Jul 19 '20 at 20:45

Unfortunately I cannot see a reproducible behaviour for jumps in that game. When i press UP o SPACE, I randomly see short or long jumps, so I cannot be really sure if my approach will work for you.

However, I think that, with a small effort, you could create a suitable event that will fit you needs. Basically, since Selenium can execute arbitrary javascript, my approach here is to send a keydown event to the canvas element (tested with Firefox 77).

A screenshot is made for each iteration to ensure the dino actually jumps.

Have fun.

from selenium.webdriver.firefox.options import Options as FirefoxOptions
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

options = FirefoxOptions()
options.add_argument("--headless")
driver = webdriver.Firefox(options=options)
driver.get('https://chromedino.com/')

canvas = driver.find_element_by_css_selector('.runner-canvas')
main_body = driver.find_element_by_xpath("//html")

try:
    canvas.click()
except:
    main_body.send_keys(Keys.SPACE)

while True:
    driver.execute_script('''
    var keydownEvt = new KeyboardEvent('keydown', {
        altKey:false,
        altKey: false,
        bubbles: true,
        cancelBubble: false,
        cancelable: true,
        charCode: 0,
        code: "Space",
        composed: true,
        ctrlKey: false,
        currentTarget: null,
        defaultPrevented: true,
        detail: 0,
        eventPhase: 0,
        isComposing: false,
        isTrusted: true,
        key: " ",
        keyCode: 32,
        location: 0,
        metaKey: false,
        repeat: false,
        returnValue: false,
        shiftKey: false,
        type: "keydown",
        which: 32,
    });
    arguments[0].dispatchEvent(keydownEvt);
    ''', canvas)
    driver.get_screenshot_as_file('proof_%s.png' % int(time.time()))
    time.sleep(0.2)

driver.quit()

Your assessment that you `randomly see short or long jumps` is correct. I too noticed this too while trying out @jizhihaoSAMA 's [suggestion](https://stackoverflow.com/questions/62501219/how-to-send-keys-to-a-canvas-element-for-longer-duration#comment111374719_62501219)... Your code is working correctly too. Can you please explain why you set the arguments in `new KeyboardEvent` to be what the are..? Also, you seem to have repeated `altKey:false` twice. — Sabito stands with Ukraine, Jul 20 '20 at 11:28
In order to obtain a valid event, I just added a keydown event to a random element in firefox console, and then attached a function that dumped that event. The duplication of "altKey" is surely a copy-paste refuse. Maybe you don't need every property of that event, you could try yourself and check against the target browser you wanto to use. — Mattia Galati, Jul 21 '20 at 08:03

score 3 · Answer 2 · answered Jul 19 '20 at 22:37

You were so close. However a few words:

The T-RexDinosaur within T-Rex Chrome Dino Game only animates within the canvas
The control, i.e. pressing the space bar doesn't needs to be sent within the canvas but only can be initiated from the keyboard.

In your code block for short jumps you don't need to send Keys.SPACE to any element.
In your code block for long jumps as well you don't need to send Keys.SPACE to any element and the variable keyboard wasn't defined as well.

Solution

As a solution once you open the url you need to induce WebDriverWait for the visibility_of_element_located() of the canvas element and you can use either of the following Locator Strategy based solutions.

Short Jumps

For short jumps you can use:

driver.get("https://chromedino.com/")
WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "canvas.runner-canvas")))
while True:
    #short jump
    ActionChains(driver).key_down(Keys.SPACE).key_up(Keys.SPACE).perform()

I was consistently able to score 69-72

Snapshot:

Long Jumps with 0.2 seconds pause

For long jumps with pause(0.2) you can use:

driver.get("https://chromedino.com/")
WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "canvas.runner-canvas")))
while True:
    #short jump
    # ActionChains(driver).key_down(Keys.SPACE).key_up(Keys.SPACE).perform()

    #long jump
    ActionChains(driver).key_down(Keys.SPACE).pause(0.2).key_up(Keys.SPACE).perform()

I was consistently able to score 65

Snapshot:

Long Jumps with 0.5 seconds pause

For long jumps with pause(0.5) you can use:

driver.get("https://chromedino.com/")
WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "canvas.runner-canvas")))
while True:
    #short jump
    # ActionChains(driver).key_down(Keys.SPACE).key_up(Keys.SPACE).perform()

    #long jump
    ActionChains(driver).key_down(Keys.SPACE).pause(0.5).key_up(Keys.SPACE).perform()

I was consistently able to score 57-60

Snapshot:

score -1 · Answer 3 · answered Jul 20 '20 at 16:56

-1

Here is a simple solution to it -

import keyboard

keyboard.press_and_release('space', 'space', 'space')

It worked when i tried so should work. You can increase and decrease the number of time 'space' has been written to increase the time duration or decrease it.

Hope it helped :)

answered Jul 20 '20 at 16:56

codeboy80

19
7

I can't use `keyboard`. I want to run this program headless. So no browser window will be open... – Sabito stands with Ukraine Jul 20 '20 at 17:34
I think you still will be able to use. Also can you explain what you mean by headless – codeboy80 Jul 22 '20 at 09:25
When you normally open selenium, it opens a browser on your device (firefox, chrome, etc). In headless mode, you are opening the browser in the background. You can't directly interact with it as you normally would. No browser will actually pop open. – Sabito stands with Ukraine Jul 22 '20 at 09:37

How to "send keys" to a canvas element for longer duration?

3 Answers3

Solution

Short Jumps

Long Jumps with 0.2 seconds pause

Long Jumps with 0.5 seconds pause