Mock in production code

Question

I'll try to be direct here:

I'm a Ruby programmer, beginning with Python, and I'm trying something with Scrapy. I'm looking into a code my company received from a third-party, and one thing is breaking my Scrapyd deploy (on the cloud):

ImportError: no module named mock

I have installed locally, it's on my requirements.txt, but it's breaking upstream, in the deploy process. To be precise, this is the only place I grep'd a reference to mock on production code:

def parse(self, response):
    ...
    with mock.patch('lxml.html', lxml.html):                            
        article.parse()
    ...

Can anyone see some light in the end of this tunnel? Does anyone have any idea why would you use that mock on a production code (and what is it supposed to do?)

Thanks =]

EDIT 1: Maybe I should clarify something: in their Scrapyd interface, they have this place for the developer to upload 'Python eggs'. I'm not sure what that is (sounds like just a Gemfile, or...a pack of modules to be imported by the server), but I ran python setup.py bdist_egg to have this egg generated, I suppose it includes all the dependencies (setup.py is kinda confusing for newcomers to Python). Long story short: I ran this command and uploaded it on the server's Scrapyd interface. So, I have no access to no apt-get or even ssh to the Scrapyd machine.

EDIT 2: I have no access to apt-get on the server, it's a normal Scrapyd server. The answer being called 'duplicate' to this one does not answer my question.

possible duplicate of [ImportError: No Module named simplejson](http://stackoverflow.com/questions/2604841/importerror-no-module-named-simplejson) — Arseni Mourzenko, Apr 08 '15 at 11:56
If that's the only place, then where is the import that's raising that error? — Martijn Arts, Apr 08 '15 at 11:57
Sorry, I meant: the only place being invoked. there is a `import mock` up there in the spider. — Thyago B. Rodrigues, Apr 08 '15 at 12:02
Where did you run `python setup.py bdist_egg`? It should be ran in the `mock` source directory, to generate the `mock` egg to be uploaded. — bosnjak, Apr 08 '15 at 12:03
I ran it on project root. I have many dependencies, some on system cache, and, apparently, some on src folder. I have to run the code on each dependency module's source directory? — Thyago B. Rodrigues, Apr 08 '15 at 12:08
Yes. Running it in your project root will get you the egg of your project, not the dependency egg. You should build the egg from the `mock` source repo, and then upload that egg to be used. I'll put more details in an answer. — bosnjak, Apr 08 '15 at 13:05

score 2 · Accepted Answer · answered Apr 08 '15 at 13:07

2

You should create eggs of the dependencies, not of your entire project. To do so, first clone the mock repository:

hg clone https://code.google.com/p/mock/

Then build the egg:

python setup.py bdist_egg

and find the egg to be used under dist folder. Upload it to the Scrapyd server and it should be satisfy the dependency.

answered Apr 08 '15 at 13:07

bosnjak

8,424
2
21
47

Since my error response now changed to "ImportError: No module named newspaper", I'll be doing this same process to all modules, and I'm accepting this answer. Thanks a lot! – Thyago B. Rodrigues Apr 08 '15 at 13:16
If somone would care to elaborate for a side-question: is this the best way in Python to manage dependencies on the server side? I mean, in Ruby we send the Gemfile and Bundler will look up our dependencies automatically. Isn't there something a bit less...painful? Thanks =] – Thyago B. Rodrigues Apr 08 '15 at 13:19

Mock in production code

1 Answers1