1

I'm deploying a flask (flask-restplus) REST API to an AWS Elastic Beanstalk instance, and I'm running into a weird failure mode.

One of my API endpoints has a dependency on OpenCV, which requires some dependencies as outlined at: ImportError: libGL.so.1: cannot open shared object file: No such file or directory while importing OCC. Per the answers there, I created an .ebextensions directory and created two files, one to install the libGL packages, which looks like this:

packages:
  yum:
    mesa-libGL : []
    mesa-libGL-devel : []

I saved that file as packages.config, if that matters.

The second file in .ebextensions downloads and installs zlib:

commands:
    00_download_zlib:
        command: |
            wget https://github.com/madler/zlib/archive/v1.2.9.tar.gz
            tar xzvf v1.2.9.tar.gz
            cd zlib-1.2.9
            ./configure
            make
            make install
            ln -fs /usr/local/lib/libz.so.1.2.9 /lib64/libz.so
            ln -fs /usr/local/lib/libz.so.1.2.9 /lib64/libz.so.1

I saved that file as zlib.config.

When I first ran eb deploy, everything worked great. Deployment was successful, my API responded to requests, and the code that depended on OpenCV worked. So far so good.

However, on subsequent deployments, I've gotten the following errors:

2020-11-18 23:47:44    ERROR   Instance deployment failed. For details, see 'eb-engine.log'.
2020-11-18 23:47:45    ERROR   [Instance: i-XXXXXXXXXXXXX] Command failed on instance. Return code: 1 Output: Engine execution has encountered an error..
2020-11-18 23:47:45    INFO    Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
2020-11-18 23:47:45    ERROR   Unsuccessful command execution on instance id(s) 'i-XXXXXXXXXXXXX'. Aborting the operation.
2020-11-18 23:47:46    ERROR   Failed to deploy application.

I went in and pulled down the logs from the instance, first looking at eb-engine.log. The only error there is:

2020/11/18 23:47:44.131837 [ERROR] An error occurred during execution of command [app-deploy] - [PreBuildEbExtension]. Stop running the command. Error: EbExtension build failed. Please refer to /var/log/cfn-init.log for more details.

However, looking at cfn-init.log just indicates that everything succeeded:

2020-11-18 23:47:34,297 [INFO] -----------------------Starting build-----------------------
2020-11-18 23:47:34,306 [INFO] Running configSets: Infra-EmbeddedPreBuild
2020-11-18 23:47:34,309 [INFO] Running configSet Infra-EmbeddedPreBuild
2020-11-18 23:47:34,313 [INFO] Running config prebuild_0_newapi
2020-11-18 23:47:36,512 [INFO] Running config prebuild_1_newapi
2020-11-18 23:47:44,106 [INFO] Command 00_download_zlib succeeded
2020-11-18 23:47:44,108 [INFO] ConfigSets completed
2020-11-18 23:47:44,108 [INFO] -----------------------Build complete-----------------------

I then tried removing the entire .ebextensions directory and re-deploying, and the deployment succeeded. Then I tried adding back the .ebextensions directory and adding the files one at a time, and discovered that the deployment worked fine when I added packages.config, but failed again when I added zlib.config.

My question boils down to: why is this happening, and is there anything I can do to resolve it? My understanding is that I need both of these files deployed to my instance in case I migrate to a different enviroment, or AutoScaling migrates my instance, etc.

The only thing I can think of is that the instance doesn't like the fact that I keep re-installing zlib, but the cfn-init-cmd.log indicates that all the commands in zlib.config are succeeding, as does cfn-init.log. So why is eb-engine.log reporting an error? Is it telling me to look in the wrong place for logs that may be relevant? I've looked in every log file and I don't see anything else indicating any issues.

I did find one tangentially-related possible solution relating to Immutable Environment Updates, which looks like it may work but feels like a bit of unnecessary work. At the very least I'd like to understand why I need to make that change and why Elastic Beanstalk isn't playing nicer with my .ebextensions.

darrenmunk
  • 23
  • 6
  • 1
    Can you ssh to new EB instance and run commands `00_download_zlib` manually, just to see that they work as expected when manually executed? – Marcin Nov 19 '20 at 00:23
  • @Marcin Just tried that, all commands worked as expected. cfn-init-cmd.log also indicates that they are all running as expected during EB deployment. – darrenmunk Nov 19 '20 at 00:27
  • 1
    Have you tried take out most commands from zlib.config, and put them back in one by one? Also out of curiosity, is there a particular reason to build from the source instead of installing one of the packages? – jingx Nov 19 '20 at 01:13
  • @jingx I haven't tried doing the commands one by one yet, I'll give that a try in the morning. The only reason I'm building from source is because that was the answer in the original [question I saw](https://stackoverflow.com/questions/62921554/importerror-libgl-so-1-cannot-open-shared-object-file-no-such-file-or-directo), so I figured there was a reason they said to build from source. – darrenmunk Nov 19 '20 at 01:19

1 Answers1

0

Just in case anyone runs across this in the future, I wanted to share the solution. I was never able to determine why the zlib install process was failing after the first deployment on an instance, so I ended up using the Immutable Environment Updates settings I linked in my original question.

Deployments take a bit longer to process as the deployment creates an autoscaling group and a new instance on each deployment, but my deployments just work every time now.

darrenmunk
  • 23
  • 6