1

I have a Rails app. I need to implement automatic text categorization algorithms and possibly more NLP capabilities in app. I believe Ruby does not have good NLP tools available as Python has. I am using a separate resque server for process background jobs. I believe I have following

  1. Run python scripts using resque jobs
  2. Run a flask application on a separate server which can either talk to resque job or can automatically update the app database with processed results.
  3. Use Ruby tools mentioned in this thread
  4. Any other suggestions welcome

Please let me know what is the best way to do it. Are there any similar working examples?

davidism
  • 121,510
  • 29
  • 395
  • 339
Rahul
  • 442
  • 6
  • 17

1 Answers1

1

I had the same problem a few months ago. After a bit of research and testing this is the solution I implemented

Run several python processes As many as one machine can hold. And use as many machines as you need.

Use ZeroMQ to communicate between the web servers and the machines running python processes

Do not use HTTP to communicate because the overhead is significant and it will be very slow compared to ZeroMQ. You will also not need an as complex handler with ZeroMQ as you would with HTTP

Take care to expose ZeroMQ sockets to internal networks only, or you would need to set up authentication on each python server

Another option is to just use one of the many available NLP APIs, if don't need any corpus based algorithms (such as POS tagging, Sentiment Analysis, etc).

saq7
  • 1,528
  • 1
  • 12
  • 25
  • Thanks a lot. I was not familiar with zmq, though after running some quick experiments I feel it is much better and easier than my original plan of sending http requests. – Rahul Jun 01 '16 at 10:27