3

I'm creating a tool for my company's developers that will take a python project, zip it up with the required modules from the site-packages of a virtualenv, and run the code in an AWS Lambda function. I need to do this because Lambda doesn't come with 3rd party modules, and also won't let you install using pip, so I have to bring them all myself. The problem is that there's also a 250mb limit for the total amount of code, and if I have numpy+pandas, that's already over the limit. However, the developers aren't using nearly the full functionality of these modules, so the actual amount of code being used is tiny in comparison. I'd like something that'll strip out at least some of the unused code based on what the developer's project uses, or at least give me enough information to where I can write a script to remove the dead code automatically. Does this exist, or is it at least partially implemented?

nicolashahn
  • 539
  • 7
  • 19
  • How is this tool supposed to handle the compiled parts of numpy, including linked libraries? – hpaulj Feb 01 '16 at 23:20
  • Well what I'm doing now is installing these modules (including building C extensions) in a `virtualenv` on an EC2 server, which has pretty much the same OS and hardware as Lambda servers, then just moving them all from the `site-packages`. This worked for using just the `numpy` module alone. Even if the tool could just shrink the .py files and move every other file, that'd be a huge boon. – nicolashahn Feb 01 '16 at 23:37
  • Just from a quick glance at `numpy` `sitepackages` directory, I'd suggest omiting or clearing an directory labeled `tests`. Also make sure you don't have both `py` and `pyc` files. A few directories like `ma` or `fft` might go. But most of the other `numpy` is too interrelated. `scipy`, on the the other hand, if used, is composed of independent packages, and can be used piecemeal. – hpaulj Feb 02 '16 at 01:07
  • We have solved this problem for a code with dependencies of numpy, scipy and pandas. Here is how we followed it. http://stackoverflow.com/questions/34749806/using-moviepy-scipy-and-numpy-in-amazon-lambda – ZZzzZZzz Feb 15 '16 at 00:48

1 Answers1

0

You could try PyMinifier which looks like it can reduce size by about half. It supports obfusticating code, compressing your project, and it looks like it has an analyzer to look for and exclude unused imports.

Edit: Linked to the documentation in my answer, here's the GitHub repo.

mcnnowak
  • 265
  • 3
  • 17