Scrapy: using a separate test database for integration tests

Question

I have a scrapy project that writes the data it scrapes to a database. It was based on this great tutorial: http://newcoder.io/scrape/part-3/

I have hit an issue now that I am trying to write some integration tests for the project. I am following the guidelines here: Scrapy Unit Testing

It's not clear to me how best to pass in the appropriate database settings. I'd like the tests to use their own database that I can ensure is in a known state before the tests start running.

So just import settings won't do the trick as, if the project is being run in test mode then it needs to use a different settings file.

I am familiar with Ruby on Rails projects where you specify a RAILS_ENV environment variable, and based on this environment variable, the framework will use settings from different files. Is there a similar concept that can apply when testing scrapy projects? Or is there a more pythonic alternative approach?

You can use `custom_settings` in your code – gangabass Mar 02 '18 at 01:15 — gangabass, Mar 02 '18 at 01:15

score 0 · Accepted Answer · answered Oct 08 '18 at 13:06

In the end I edited the settings.py file to support using an environment variable to determine which additional files to get the settings from, like this:

from importlib import import_module
import logging
import os

SCRAPY_ENV=os.environ.get('SCRAPY_ENV',None)
if SCRAPY_ENV == None:
    raise ValueError("Must set SCRAPY_ENV environment var")

# Load if file exists; incorporate any names started with an
# uppercase letter into globals()
def load_extra_settings(fname):
    if not os.path.isfile("config/%s.py" % fname):
        logger = logging.getLogger(__name__) 
        logger.warning("Couldn't find %s, skipping" % fname)
        return
    mdl=import_module("config.%s" % fname)
    names = [x for x in mdl.__dict__ if x[0].isupper()]
    globals().update({k: getattr(mdl,k) for k in names})

load_extra_settings("secrets")
load_extra_settings("secrets_%s" % SCRAPY_ENV)
load_extra_settings("settings_%s" % SCRAPY_ENV)

I made an example github repo showing how this worked: https://github.com/alanbuxton/scrapy_local_settings

Keen to find out if there is a neater way of doing it.

Scrapy: using a separate test database for integration tests

1 Answers1