0

I use Conda and have a private Conda repo.

The max amount of layers a function can use is five. So I can't just upload every one of my private Conda packages as a separate layer- because functions might need to use more than 5 of them

I'm unclear on how layers are supposed to solve this problem or if they are not designed for this sort of thing.

Would I have just one lambda layer for the base Conda environment and then have the lambda use that layer to import my private Conda packages? Or should I be uploading my entire function as a new layer itself, where I would build my Conda Python app locally with all my private Conda deps, upload it as a new layer and then have a different function call it?

halfer
  • 19,824
  • 17
  • 99
  • 186
red888
  • 27,709
  • 55
  • 204
  • 392

1 Answers1

1

You could try building all you private packages into a separate - single - layer, and then using that (single) layer of your own with your functions. ...would that work for you? (there is no restriction in layers that they only have to contain single packages/modules)

If your base conda env also has public packages, you could also have a separate layer you build for that environment - so then in your functions you'd include two layers - one for your base conda env, and one for all your private packages.

With my code, I use conda for dev, but not for deployment. At present I've built and use a private layer containing skimage, but I build all my private packages in with the lambda's themselves, rather than bundling them as a layer. But I've thought about moving that way in the future.

I also use AWS's SAM for building and packaging, which makes things a lot easier. But I'm not sure how easy or not it is to make it work with conda instead of pip.

Here's a post I did on using SAM to make layers - might be some help if you go that route (...you can use SAM to build/install layers right now, but it takes a bit of arm twisting)

Richard
  • 3,024
  • 2
  • 17
  • 40
  • this is what I was thinking I would have to do, deploy the specific app as a layer- which runs against the use case of layers a bit because I believe they are supposed to be for shared dependencies across functions? Could I, in the same SAM template, deploy a layer and also a function that uses that layer? Will the SAM template zip up and upload my layer for me and point the function to it? – red888 Nov 18 '19 at 15:34
  • In this example: https://bryson3gps.wordpress.com/2018/12/06/trick-sam-into-building-your-lambda-layers/. Hes using a hack to get SAM to build a layer. I guess you need to define a function for SAM to build anything and he says `This does result in an orphan Lambda function that will never be used`- for my use case could that "orphan" function actually be the lambda I want to use my layer? Sounds like it would do exactly what I want. Im not uploading a layer I intend to share with any other functions – red888 Nov 18 '19 at 15:59
  • In my mind there are two ways of doing this: in a 'dynamic linking' model you use layers for all your associated modules (private or public). In a 'static linking' model, you don't use layers, but instead have those modules included into the function package at build time. I think in what you're describing above "for my use case that "orphan" function actually be the lambda I want to use my layer" you're somewhat doing it as a static library, as your layer and function are one and the same (package wise). – Richard Nov 18 '19 at 20:05
  • 1
    For answers to your first comment: yes - SAM can build/deploy a layer (with bryson's hacks) and function at the same time. Yes, in the same template you can also (in fact you need to) deploy a function that uses that layer. Yes, SAM will zip up, upload (ie "package" in SAM speak) the layer, as well as "deploy" it (register with AWS lambda etc). Note, once that layer is built, you can then use it later in any other functions you build/deploy - it's a true layer, so not restricted later for who uses it. – Richard Nov 18 '19 at 20:08
  • Could this also be achieved with a custom runtime? https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html – red888 Feb 04 '20 at 17:57
  • I'm not sure - I haven't used them at all. I did a quick read of the runtime docs, but wasn't sure exactly what their use case definitely is. – Richard Feb 04 '20 at 19:48