1

I'm trying to setup scrapy settings to work with test and production environment on local and also on scrapinghub. And I would like to know if there is any way to set this variable (for example as the following) on shub deploy:

enter image description here

And then at settings.py:

if env == "test":
     var1 = some_ip
     var2 = username
elif env == "prod":
     var1 = some_ip
     var2 = username

Or... maybe there is a cleaner way to this?

Thank you for reading!

PS: I want to automate the settings depending of the environment where is the spider/project located instead of changing the variables manually.

EDIT: Found a better solution. Check the answer.

Community
  • 1
  • 1
Alberto
  • 1,423
  • 18
  • 32

2 Answers2

2

If you want to control it using environment variables you could do the following: export an environment variable DEPLOY_TARGET that would contain test or prod and execute the following command to deploy

shub deploy $DEPLOY_TARGET
Wilfredo
  • 1,548
  • 1
  • 9
  • 9
  • That helped me to thinking on environment variables but did not solved my problem. Thank you anyways! – Alberto Nov 15 '17 at 11:16
1

Finally, I solved my problem following these steps:

So now my settings look like this:

settings tree

  • And then my __init__.py look like this:

__init__.py file

  • Next, modified _environment.py as the following:

_environment.py. this is a function helper that gets programmatically the environment where is Scrapy at the moment

I hope I have been helpful!

PS: Remember that you need to have two projects on scrapinghub, one for production and one for testing to do this, also you need to deploy from the same project as the following:

scrapinghub.yml

And then you just have to (reference):

  • shub deploy test Deploys to test.
  • shub deploy prod Deploys to production.
  • shub deploy Deploys to test.
Alberto
  • 1,423
  • 18
  • 32