2

I want to do some web crawling with scrapy and python. I have found few code examples from internet where they use selenium with scrapy.

I don't know much about selenium but only knows that it automates some web tasks. and browser actually opens and do stuff. but i don't want the actual browser to open but i want everything to happen from command line.

Can i do that in selenium and scrapy

Mirage
  • 30,868
  • 62
  • 166
  • 261

2 Answers2

8

Updated: PhantomJS is abandoned, and you can use headless browsers directly now, such like Firefox and Chrome!

Use PhantomJS instead.

You can do browser = webdriver.PhantomJS() in selenium v2.32.0.

jifeng.yin
  • 2,081
  • 1
  • 21
  • 19
  • 2
    @user172409255 This should have been the best answer, as you requested command line. So you could use for example webdriver.Chrome() to view what is going on, then fall back to webdriver.PhantomJS() for production. Or echo results to view and stick with the much faster Phantom.js – Shane Aug 03 '13 at 08:56
  • PhantomJS is now abandoned in favor of headless modes that major browsers themselves provide. – ivan_pozdeev Jun 10 '18 at 14:00
5

You can use selenium with PyVirtualDisplay, at least on linux.

from pyvirtualdisplay import Display
from selenium import webdriver
display = Display(visible=0, size=(1024, 768))
display.start()
browser = webdriver.Chrome()
root
  • 76,608
  • 25
  • 108
  • 120