2

For installing python packages, is there a way to only install the specific modules I need (plus dependencies) into the virtual environment?

For example, I import the following in my code

from statsmodels.tsa.arima_model import ARIMA

I only need ARIMA (and dependencies), and won't need the complete statsmodels library.

How do I accomplish that?

Rationale:

  • I am trying to zip up the packages and deploy to AWS lambda. However, I am exceeding the 50MB file size limit.
  • If there is a way to install only the modules I need from the packages, I can reduce the file size.
martineau
  • 119,623
  • 25
  • 170
  • 301
Ben Yi
  • 23
  • 3

1 Answers1

1

Here's a horrible brute-force approach. I hope someone gives you a better answer!

Assuming the source you want to install is public, its licensing allows this kind of use, and you are willing to deal with the hassle, I imagine you could:

  • copy the source
  • remove anything unused by your required module
  • place that module's source inside your package, and wire it up from there. (This answer proposes a reasonable file structure for vendored software)

Alternately, if the assumptions above hold but you prefer not to vendor directly within your software, you could probably:

  • copy the source (e.g. fork the repo)
  • remove anything not used by your required module, making sure to leave a working python package
  • use pip or your environment's native dependency management tools to download and install your package in the virtual env.

Please note that both of these approaches will expose you to a number of risks, including but not limited to:

  • breaking the module: what happens to your project if you accidentally created a logical or security flaw?
  • loss of currency: how do you plan to keep your new mini-package up to date?
  • liability: Does the vendored software's license allow this kind of use? Are you adequately protected if there is something wrong with the software you have modified?
Chris Keefe
  • 797
  • 7
  • 17
  • Thanks for your suggestion! Very practical and outlines all the risks. – Ben Yi Aug 13 '20 at 00:06
  • Thanks @Ben Yi. Please upvote if you found the answer useful, and consider accepting it if you believe it to be the best correct/viable answer. – Chris Keefe Aug 13 '20 at 22:12
  • I would love to, and I did! Unfortunately I don't have enough reputation yet, so it doesn't show. – Ben Yi Aug 18 '20 at 06:46
  • Absolutely no pressure to accept this answer, but as the asker you have the privilege accept an answer regardless your rep. It will give you +2, and you can change the accepted answer later if you get a better one. Here's how: https://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work – Chris Keefe Aug 19 '20 at 06:46