1

I have a Lambda function in Python that uses several libraries with binary dependencies: numpy, scipy, Pillow, etc. In the past, I have successfully compiled them on Amazon Linux & deployed to Lambda.

I recently added Grequest to the pile of dependencies, though, and it's throwing errors that don't seem direclty related to Grequest. Other questions on this topic have resulted in dead-ends, so putting my hat in the ring.

The first error was a simple Grequests requires gevent or something similar. To solve this, I tried running pip install gevent --no-binary :all: on an Amazon Linux instance, bundling that with my code and uploading to Lambda. This had no effect on the error.

I then downloaded the src from the gevent repo and compiled it using make (the commands in the repo's README threw errors I didn't record). This produced an egg file, which I converted to a whl file & ran pip install gevent.whl -t ..

I bundled the resulting code with my lambda.zip and uploaded. This led to a new error: module initialization error: lib/libptf77blas.a.gz: invalid ELF header

libptf77blas.a.gz is a file in the lib folder in my lambda.zip. This folder contains several .so and .a files, which I built on AWS Linux while assembling numpy, scipy, Pillow, etc.

As far as I know, this is a dependency for numpy. The part I don't understand is: my function used numpy (and presumably libptf77blas.a.gz just fine prior to adding grequests.

So I assume something about compiling gevent broke the dependencies for my other binary-dependant libs, or gevent itself requires libptf77blas.a.gz and is confused at what it finds.

I ran ldd on the .so files gevent depends on, but none referenced libptf77blas.a.gz. But, I'm a compiling newb, so that's hardly conclusive.

I'd like to achieve the parallel upload Grequests enables, but am unclear how to fix this, or which alternative libraries to use (I usually write JS, so Python is a second language).

Community
  • 1
  • 1
Brandon
  • 7,736
  • 9
  • 47
  • 72

2 Answers2

3

The problem was entirely a stupid user error from an unrelated matter--follow the steps in the OP and you should be fine.

Or just use the copy of grequests I posted on github. It's a zip with grequests, gevent, and greenlet compiled on an Amazon Linux instance. It works in my Lambda Function just fine.

Brandon
  • 7,736
  • 9
  • 47
  • 72
  • You should accept this answer even if it is your own as it is the correct one. – vks Feb 03 '17 at 09:16
  • I know I'm replying to a year old post/answer but I'd be much obliged if you would happen to have the same kind of zip file for Python 3.6 :D – kilokahn Dec 19 '17 at 05:43
0

As the other questions have stated, you need to make sure the binaries for libev and other things, are compiled using the Amazon Linux Image. Basically, you need to install things like numpy, and Grequests on a machine running Amazon Linux as they use C code that needs to be compiled.

The tutorial link in that question looks quite good for building things from source, albeit a little bit more than you need for this.

If you're that new to compiling binaries you should read a bit about make, and look at some examples like building curl. Since you're on Amazon Linux, it uses yum and is quite similar to CentOS.

You might be able to get away with installing numpy and Grequests in a venv on a docker image in CentOS and zipping up the contents from that similar to the tutorial, but I'd first go with launching an EC2 instance using amazon's image and installing stuff there like the tutorial does.

Community
  • 1
  • 1
Kirk
  • 1,779
  • 14
  • 20
  • thank you. as i note in my first paragraph, I've compiled numerous libraries on Amazon Linux before, including `numpy`, which up until adding `grequests`, worked fine in my `Lambda` function. I tried that tutorial you linked earlier, however the `gevent` library doesn't follow the same `make` steps, as far as i can tell. I was able to `make` the `gevent` repo into an `egg`, convert that into a `whl`, and install from the `whl` (on AWS Linux), but hit comparable errors to what my OP details. `gevents` also fails using less involved procedures, such as `pip install --no-binaries :all:`. – Brandon Dec 27 '16 at 21:50
  • My apologies. I clearly glanced right over most of that first paragraph. – Kirk Dec 27 '16 at 21:56
  • np--honestly I need to learn to write more succinct SO questions. It's unreasonable to expect people to dig out substance from my flowing prose :) – Brandon Dec 27 '16 at 21:57
  • Just to clarify, running a `pip install grequests`, and testing the grequests lib itself on the EC2 instance produces this error as well, or is it just on Lambda? – Kirk Dec 27 '16 at 22:00
  • The original `lambda` error is something like "Grequests requires gevent", so on Amazon Linux I installed only `gevent`. it failed at first as outlined in this [github issue](https://github.com/gevent/gevent/issues/782), but after a few diff't approaches, the command `pip install gevent -t . --no-binary :all:` completes w/o error. But, packaging the resultant files into my `lambda.zip` does not solve the original error. I'll edit my OP to reflect this, but it was building `gevent` from the `src` that got me from generic `Grequests requires gevent` error to the `ELF header` error (progress?) – Brandon Dec 27 '16 at 22:09
  • i added some clarity to the OP re: the steps I've taken and the results they've had. thanks for your interest, @Kirk. – Brandon Dec 27 '16 at 22:19
  • I'd try to get a sample script to run on the Amazon Linux image itself, just to be sure you got it compiled once correctly. Something dumb like https://github.com/kennethreitz/grequests#usage just to make sure the thing works. Also, it looks like the issue you linked seems to be a problem with pip itself. Perhaps something like `sudo /usr/local/bin/pip install --upgrade pip` would help out here too. You should be able to install the thing without being fancy with pip it seems. – Kirk Dec 27 '16 at 22:44