module initialization error: 'twisted.internet.reactor' when running scrapy on AWS Lambda

Asked Apr 14 '17 at 08:43

Active Apr 14 '17 at 08:43

Viewed 441 times

I m trying to run a scrapy spider in AWS Lambda.

I am getting this error :

module initialization error: 'twisted.internet.reactor'

The spider is running fine on an EC2 linux AMI

So far I have tried :

using crochet to make scrapy behave like a blocking process. https://pypi.python.org/pypi/crochet
using ScrapyScript to have twisted run outside of the main thread.
re-installing twisted (pip install twisted)
running scrapy with stop_after_crawl=False as it seemed to act as a workaround for the author of Scrapy throws error ReactorNotRestartable when runnning on AWS Lambda

edited May 23 '17 at 12:02

Community

asked Apr 14 '17 at 08:43

Hugo

Did it work in the end? – Ravindranath Akila Oct 29 '17 at 06:56
2

No I never made it work on AWS lambda. I fell back on using an EC2 server – Hugo Oct 31 '17 at 09:25
Thanks Hugo! Saves time! – Ravindranath Akila Nov 06 '17 at 07:39

0 Answers0