0

I looked into this question which asked how a bot could input text on a webpage. One of the answers recommended using Selenium, but a comment there suggested using it was an inefficient way of accomplishing that task.

Say I wanted to create a bot that looks up a set of words on Wikipedia (using the search bar on Wikipedia) and gives me the first 20 words in each article. Would Selenium be the best tool for this?

(Note that I'm aware I could do this manually by just looking up https://en.wikipedia.org/wiki/<word I want> for each item in the list, but I'm specifically looking for how a bot would interact with search bars.)

Stephen
  • 35
  • 8

1 Answers1

2

Efficient and bot for what you're doing doesn't seem to intersect from what you described - why bother using a framework that renders the entire view as a human would see it when you are not using any of that visual content? The most efficient way to utilize a python bot to search on wiki would be to utilize the api and get the results as json to be parsed by the bot.

Searching Wikipedia using API

There is nothing magical about a search bar - when the input is put in there, the browser is redirected to the other url location as you stated https://en.wikipedia.org/wiki/<word you want>. I believe the inefficiency that is being referenced is this exact fact that you can just search manually without the search bar. Rendering and finding the bar to type something in and then submit takes hundreds of milliseconds. Searching directly on the API can be done in milliseconds - much more efficient.

PressingOnAlways
  • 11,948
  • 6
  • 32
  • 59
  • Ah, that makes sense. I didn't realize that you could just use the API, I thought that only worked for websites that directly linked to the desired URL location (like wikipedia, reddit, etc.). – Stephen Aug 16 '20 at 04:16