4

We are storing our PHP project on github.

For fast deployment we are using .bat file for git pushing changes to AWS Elastic Beanstalk cloud:

"C:\Program Files (x86)\Git\bin\sh.exe" --login -i -c "git aws.push --environment envname"

We are making commit every time before push, and it's working just perfect, as expected.

Unfortunately, for some reason, sometime it is pushing really quick (just pushing difference in PHP Code changes), but sometimes it is sending whole 300mb project (with all media).

Is there any way to git push only changed diference? Maybe there is any additional parameters on push or preparation git commands before send? Or maybe there is some way to tell AWS EB to pull last commit from github repository?

Thanks for any help.

Kainax
  • 1,431
  • 19
  • 29
  • I'm confused by this because Elastic Beanstalk's application version doesn't allow a bundle that large. There is a hard limit on somewhere around ~400MB – Nick Humrich Nov 15 '14 at 02:57
  • @Humdinger 5Gb is a realative value here :) it is actually around 300mb, but still, with our country internet connection it takes around 30 min to upload that. – Kainax Nov 15 '14 at 04:02
  • Why downvote? It's doesn't matter how much project weight 300mb or 5gb, question is the same - How to prevent full repo push instead of pushing only changes? – Kainax Nov 15 '14 at 12:58
  • 1
    @Kainax I experiment the same problem, some times git aws.push upload the whole 300MB (same case that me) and sometimes only the changes. – Jose Nobile Apr 04 '15 at 15:08

3 Answers3

2

When you issue git aws.push your entire git repository is zipped and uploaded to Elastic Beanstalk, hence the long push time. That's the way this command is implemented. There's a nice discussion and request for change in this thread.
I'm not sure what files weigh so much in your code, but in general it's best not to store them in the application git.
If these are media files, store them in S3 (or any other web accessible place). If the majority of files are external libs sitting in your git, you can pre-install them by an EB pre-deploy script.

Tal
  • 7,827
  • 6
  • 38
  • 61
  • Regarding media files size - it is actually dynamically generated 360 degree images that are changing every time when 3d Artists changing models, so they are changing during development period constantly and must be uploaded on every change, so currently manual S3 bucked upload is not a solution for us. If we not gonna find a solution, I think that I will extend custom imgs-generation softmare with automatic S3 upload. – Kainax Nov 17 '14 at 07:42
  • This answer is wrong, at least in Windows, git aws.push is implemented using powershell and it what does is get the user and url where to push, and after it does a normal git push. I think, because amazon is distributed, and the URL is not the same where push, it doesn't push to the same server and need push the whole repo. Would you accept this info as an answer? – Jose Nobile Apr 04 '15 at 15:11
  • @JoseNobile Hi, thanks for the tip. The reason is actually obvious, but what is the solution for this issue? Question is still open, maybe someone knows how to fix it. – Kainax Apr 13 '15 at 01:52
2

I implemented a workaround because this is a bug of Amazon.

I setup a git account in a server (I used Gitlab, free private repositories and unlimited collaborators), you can use any that you want (github, gitlab, your server).

The process between you and the git server is reliable, you only push the changes, always. In the EC2 server, I setup a git repository (you can use the configuration file .ebextensions/*.config or do manual, I suggest manual to not setup in all your instances) which is connected to my git server and from EC2 server execute git pull and a git aws.push.

You are able to setup for example a Web Hook to notify when you do a git push and then do the git pull and git aws.push automatically. in my case I prefer decide when deploy using SSH, and this is not always when I push commits to my git server.

In this way, do the git aws.push from EC2 server to Elastic Beanstalk server is very fast, doesn't matter if it push everything or only the changes.

Jose Nobile
  • 3,243
  • 4
  • 23
  • 30
2

OK, after months of troubles I resolved the reason of the problem.

The problem is that on

aws.push

command Elastic Beanstalk is checking if your current files are committed to the git or they are ahead of the repository. If your files are identical to a last git commit, then EB pulls only difference, but if your files are ahead of the last git commit, then it pulls whole folder!

To avoid problem I made a deploy.bat file for windows that tries to commit changes before aws.push command:

cls
git add .
git commit -a
git aws.push --environment myenvironment
pause

Now, if you already did commit before running this .bat file then you will upload only difference. Otherwise, you will be asked for a git commit message before uploading difference.

PS - Please note that if you will try to push version that is already on EB one more time, it will also try to upload entire folder.

PSS - If by any reason you will see that EB tries to pull entire folder, just close the window, make small changes to a source code (like adding new line or a comment), save, commit changes to git and run deploy.bat file again.

Works like a charm!

Edit:

For those who are using new standalone eb cli, the code will be:

cls
git add .
git commit -a
eb deploy your_environment_name
pause
Kainax
  • 1,431
  • 19
  • 29