1

I have a Python program that uses BeautifulSoup to extract some data from a website.

In Google Chrome, there is this option called the Developers Console; it is used to execute javascript commands live right on the webpage.

Is there a way, or a work-around-of-a-way to execute javascript commands remotely using Python? Like casting a webpage to an object and running a javascript command in the background (without launching a new Chrome window)?

Boris
  • 805
  • 1
  • 8
  • 20
  • You mean like execute JS using selenium or something? – SuperStew Jan 12 '18 at 19:13
  • Selenium does exactly this, I think you can use it with Python - though I've one done it with Java – cbll Jan 12 '18 at 19:14
  • 1
    @sushain97 eh he didn't mention selenium, yet, not sure if that's what he means – SuperStew Jan 12 '18 at 19:15
  • @SuperStew Fair. I see Selenium as one of the possible (and most advisable) solutions since OP asks for "casting a webpage to an object". – sushain97 Jan 12 '18 at 19:19
  • @sushain97 A web page is already an object. By "most advisable" you are referencing your own opinion, correct? – guest271314 Jan 12 '18 at 19:22
  • @SuperStew I'll look into it, thank you! – Boris Jan 12 '18 at 19:28
  • @guest271314 Sure, my opinion. There are lots of Selenium resources out there already. A web page is an object but not a Python object. From OPs desire to "cast", I'm guessing they want a Python object. – sushain97 Jan 12 '18 at 19:29
  • @sushain97 The Native Messaging API provides a means to communicate between the browser and a shell script, there is a [sample app](https://developer.chrome.com/extensions/examples/api/nativeMessaging/app.zip) which uses python specifically to do so. It is also possible to use remote debugging. Not sure how Selenium is related to the inquiry. It is not clear from the original Question what JavaScript commands should be executed and what the expected result is. Perhaps consider posting an Answer describing the approach that you are suggesting. – guest271314 Jan 12 '18 at 19:31
  • @sushain97 CGI or `chrome.sockets` could be used to communicate with a python script. There are many options available. – guest271314 Jan 12 '18 at 19:36
  • @Darker1000 Which specific JavaScript commands are you trying to execute? What is the expected result? – guest271314 Jan 12 '18 at 19:40
  • This question is not related to Selenium alone – guest271314 Jan 12 '18 at 19:55
  • @guest271314 I'm trying to run a command from one of the javascripts attached to the website, that command returns a String array of some of the website's elements. – Boris Jan 12 '18 at 19:59
  • @Darker1000 We need greater clarity as to what should be executed by which language. You last comment appears to convey that you want to send data from JavaScript to python? – guest271314 Jan 12 '18 at 20:00
  • Are you trying to do something like `$ chromium --headless --disable-gpu --dump-dom http://example.com` or from python `>>> os.sytem('chromium --headless --disable-gpu --repl http://example.com') # Type a Javascript expression to evaluate or "quit" to exit. >>> document.querySelectorAll("p") {"result":{"className":"NodeList","description":"NodeList(2)","objectId":"{\"injectedScriptId\":1,\"id\":3}","subtype":"array","type":"object"}}`? – guest271314 Jan 12 '18 at 20:21
  • 1
    @guest271314 Using Python, I need to run javascript commands remotely from Python. And possibly, have the site as an object. Kind of like Beautiful Soup. – Boris Jan 12 '18 at 22:10

1 Answers1

-1

You can use Native Messaging API to communicate between a shell script and the browser.

You can also launch Chrome or Chromium with --headless flag set. See also puppeteer.

guest271314
  • 1
  • 15
  • 104
  • 177