How to use Selenium Webdriver on Heroku?

Question

I am developing a Node.js app, and I use Selenium Webdriver on it for scraping purposes. However, when I deploy on Heroku, Selenium doesn't work. How can I make Selenium work on Heroku?

Answer 1

Below is a javaScript sample code using selenium-webdriver npm package with chrome browser.

const webdriver = require('selenium-webdriver');
const chrome = require('selenium-webdriver/chrome');

let options = new chrome.Options();
//Below arguments are critical for Heroku deployment
options.addArguments("--headless");
options.addArguments("--disable-gpu");
options.addArguments("--no-sandbox");

let driver = new webdriver.Builder()
  .forBrowser('chrome')
  .setChromeOptions(options)
  .build();

driver.get('http://www.google.com');
driver.quit();

Before you are ready to deploy, you would need to add two buildpacks to Heroku.

• Using Heroku buildpacks command:

$ heroku buildpacks:add --index 1 https://github.com/heroku/heroku-buildpack-chromedriver
$ heroku buildpacks:add --index 2 https://github.com/heroku/heroku-buildpack-google-chrome

or

• Config in Heroku dashboard:
  Settings -> Add buildpacks -> https://github.com/heroku/heroku-buildpack-chromedriver -> Save changes
  Settings -> Add buildpacks -> https://github.com/heroku/heroku-buildpack-google-chrome -> Save changes
  

Answer 2

I was able to get the Selenium Webdriver working on Node/Heroku using PhantomJs as the headless browser. I installed the PhantomJs buildpack to my Heroku app and it just worked. I struggled to get Chrome and Firefox drivers working on Heroku... I wrote a blog with the steps and code I used to get it working:

http://www.viderman.com/2017/05/selenium-on-heroku.html

Answer 3

Pasting my answer from here: https://stackoverflow.com/a/60586343/11135757

---

Note: I am using React, Express, and Selenium with chrome

Step 1: Create a new Heroku app.

Step 2: From your terminal, login to heroku using heroku login

step 3: Once you're logged in, cd to your project directory and set its remote to your heroku app. heroku git:remote -a YOUR-HEROKU-APP-NAME

step 4: Run all the following commands in your terminal

heroku buildpacks:add https://github.com/heroku/heroku-buildpack-chromedriver
heroku buildpacks:add https://github.com/heroku/heroku-buildpack-google-chrome
heroku config:set CHROME_DRIVER_PATH=/app/.chromedriver/bin/chromedriver
heroku config:set CHROME_BINARY_PATH=/app/.apt/opt/google/chrome/chrome

step 5: Login to heroku from your browser and navigate to your app. Go to settings and under buildpacks, add heroku/nodejs

step 6: This is how my index.js looks like. Note: my express entry point is inside root-dir/server/index.js and my react files are inside root-dir/client/

const express = require('express');
const app = express();
const path = require('path');

// Serve static files from the React app. 
app.use(express.static(path.join(__dirname, '..', 'client/build')));


app.get('/api', async (req, res) => {
    const webdriver = require('selenium-webdriver');
    require('chromedriver');
    const chrome = require('selenium-webdriver/chrome');

    let options = new chrome.Options();
    options.setChromeBinaryPath(process.env.CHROME_BINARY_PATH);
    let serviceBuilder = new chrome.ServiceBuilder(process.env.CHROME_DRIVER_PATH);
    
    //Don't forget to add these for heroku
    options.addArguments("--headless");
    options.addArguments("--disable-gpu");
    options.addArguments("--no-sandbox");
  

    let driver = new webdriver.Builder()
        .forBrowser('chrome')
        .setChromeOptions(options)
        .setChromeService(serviceBuilder)
        .build();

    await driver.get('http://www.google.com');
    res.send(await driver.getTitle());
});

app.get('*', (req, res) => {
    res.sendFile(path.join(__dirname, '..', 'client/build/index.html'));
});

const port = process.env.PORT || 5000;
app.listen(port, () => {
    console.log(`listening to port ${port} now...`);
});

step 7 (if you are using React): Now inside your package.json in root-dir/, add this

"scripts": {
...
"heroku-postbuild": "cd client && npm install && npm run build"
}

step 8 (if you are using react): inside your package.json in root-dir/client/ (i.e: package.json for react app), add the following line:

  "proxy": "http://localhost:5000/",

step 8: (if you're using react): inside root-dir/client/src/, create a new file called setupProxy.js and paste the following code:

const proxy = require("http-proxy-middleware");

module.exports = function(app) {
    app.use(proxy('/api', { target: `http://localhost:${process.env.PORT || 5000}/`}));
};

step 9: Now, you are ready for deployment. Make sure you have the following packages installed: express, selenium-webdriver, and chromedriver

step 10: now push it to heroku

git add .
git commit -m "my app"
git push heroku master