How to deploy private python pip dependency with Amazon AWS Elastic Beanstalk?

Question

When I tried to set one of the services into AWS Elastic Beanstalk, the problem appeared. One of our python pip dependencies set up on private repository was causing errors during deployment process, since it was not accessible for pip process. Bellow I present description how we solved this issue.

It is worth to mention that there are other solutions that bind SSH keys used during deployment, to application project git repository. I find them a bit dirty, so I would like to share this one that allows to keep SSH keys separated in S3 bucket, separated from application git repository.

While Q&A is always welcomed, you need to have a quality answer **and** a quality question. Your question, as is, is a loose one-liner, please edit it a bit to match the quality required. It's a shame when one puts effort in self-answering and then gets downvoted. — Dimitris Fasarakis Hilliard, Jan 11 '16 at 17:24
@smentek Was the private repository referenced above the application itself (aka the one being installed)? — Drew Burnett, Jul 17 '16 at 18:24
@AndrewBurnett By private dependency I mean library for the application (that also is private). The fact that application is private was never an issue. — smentek, Jul 17 '16 at 18:30
@smentek Ok, understood. I'm having an issue deploying a Pyramid application to EB. Btw, this is my first attempt to deploy an application of any sort. The DistributionNotFound error I'm encountering is referencing the application itself (aka the one containing requirements.txt). When I run pip freeze a git reference to my app is included but AWS can't pull from the repo without SSH keys setup. I feel like I'm missing something here. How is *my* application installed along with all the third-party libraries. Can/do I need to include a local reference to it in requirements.txt? — Drew Burnett, Jul 17 '16 at 18:40
Do you have your application as dependency in requirements.txt? It should not be there in any case. EB will handle deployment of code for your application, since you "attaching" your local git index, to deployment process by 'activating' your git branch for deploy (eb init; eb use). — smentek, Jul 17 '16 at 18:48
@smentek Thank you, good to know. I have and haven't (tried both scenarios). In both cases, I'm met with the DistributionNotFound error referencing my application. Although, I haven't run an `eb use` command at any point. (Side note: when I run `pip freeze` it includes a `git+myonlinerepo` dependency.) — Drew Burnett, Jul 17 '16 at 18:54
Do you use eb cli tool? What do you see after running 'eb list' command? requirements.txt should point to all your dependencies libs that you may find on: pypi.python.org/pypi. I guess you do not use any private libs so it should be rather easy. Read more about using elastic beanstalk cli tool... — smentek, Jul 17 '16 at 18:59
@smentek I am using the eb cli tool. I originally ran eb init in the virtualenv created and used by my application. When I run `eb list` it returns the name used to initialize the environment on EB. Does/should this name match the name of my local virtualenv? Or local application name? Can it be different? I'll look into it more. Thank you much for all your help. — Drew Burnett, Jul 17 '16 at 19:13
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/117555/discussion-between-andrew-burnett-and-smentek). — Drew Burnett, Jul 17 '16 at 19:23

smentek · Accepted Answer · 2016-07-17T18:39:53.303

Works on: "64bit Amazon Linux 2015.09 v2.0.6 running Python 3.4" for dependency from private bitbucket repository. it may be modified for github, etc.

Add you private dependency to pip requirements.txt:

  -e git+git@bitbucket.org:account_name/dependency-name.git#egg=dependency-name

Generate public & private SSH keys (HOWTO can be found elsewhere) so you have id_rsa (private) and id_rsa.pub (public) key files.

On bitbucket.org project tab, find settings and go under "Deployment keys". Use form to set your public SSH key there.

Upload both generated keys (just private should be enough) on S3 bucket using amazon AWS console:

  bucket-with-keys/bitbucket/:  
 - id_rsa
 - id_rsa.pub

Where bucket-with-keys/bitbucket - is [BUCKET_NAME]/[PATH].

Add packages file to your project: project_name/.ebextensions/packages.config

    packages:
      yum:
        git: []

Add configuration file to your project: project_name/.ebextensions/03-pip-install-from-bitbucket.config:

    files:
        "/root/.ssh/config":
            owner: root
            group: root
            mode: "000600"
            content: |
                Host *
                    StrictHostKeyChecking no
                    UserKnownHostsFile=/dev/null
        "/root/.ssh/known_hosts":
            owner: root
            group: root
            mode: "000644"
            content: |
                #
                # paste output of `ssh-keyscan -H github.com` here
                #
    commands:
        01-command:
            command: aws s3 cp --recursive s3://bucket-with-keys/bitbucket/ /root/.ssh
        02-command:
            command: chmod 600 /root/.ssh/id_rsa

If you modify this file, keep in mind that:

commands are being executed in alphabetical order so "01-" starts before "02-" even if you would switch its order.
file name also follows that rule.

Go to AWS console IAM (Identity & Access Management) "policies" tab and find AmazonS3FullAccess on a list and attache it for aws-elasticbeanstalk-ec2-role. If you cannot find policy you can create one like below:

  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "s3:*",
        "Resource": "*"
      }
    ]
  }

Look for "Create and Attach Your First Customer Managed Policy" Tutorial if needed. This policy is quite open and it is advised to create one that would be more narrow... aws-elasticbeanstalk-ec2-role is role created by default by Elastic Beanstalk but you can use your own as long as it is set with CLI tool eb config under: "IamInstanceProfile: aws-elasticbeanstalk-ec2-role"

Now you can create your environmant with eb CLI tool:

      eb create project_name-prod-env  \
        --instance_type t2.micro \
        --tier webserver \
        --region eu-west-1 \
        --cname project_name-prod-env \
        --keyname identifier-of-ssh-key-accessed-from-console-here \
        --platform "64bit Amazon Linux 2015.09 v2.0.6 running Python 3.4"

Should work now!

If sth goes wrong you may debug if SSH files got to its place:

  eb ssh
  sudo su
  ls -la /root/.ssh/

And check logs on AWS console or directly on instance:

  eb ssh
  sudo su
  less /var/log/eb-activity.log

Try manually execute commands from project_name/.ebextensions/03-pip-install-from-bitbucket.config as a root user bu in the way they appear in log file, use switches to get more verbose output.

score 2 · Answer 2 · answered Feb 17 '16 at 20:49

2

I cannot comment so I'm answering the question. smentek's answer is very detailed and solves the issue. The only missing part is that you need to add the git package to the config:

packages:
    yum:
        git-all: ""

files:
    "/root/.ssh/config":
# ...

Git isn't installed on Amazon Linux by default.

answered Feb 17 '16 at 20:49

henadzit

332
2
8

1

Thanks, I've added it to my Answer. – smentek Jul 17 '16 at 18:40

How to deploy private python pip dependency with Amazon AWS Elastic Beanstalk?

2 Answers2

Linked