4

I have an AWS EC2 instance on linux running 24/7 with a set of python files that are working properly. From the shell, I can input python python_script.py var1 var2. var1 and var2 are simply integers with values under 5,000.

I am able to set up the AWS API to receive HTTPS requests that deliver these variables as environmental variables. Then I can use AWS lambda to run some code when the API is triggered.

I can't for the life of me figure out what code to include in the lambda function in order to run the python script on the EC2 instance with the two variables. All of the results that I have found related to this issue are related to scripts on startup of an EC2 instance, run commands without any variables, or setting up an entire web server with domain and everything to receive the HTTPS request. I simply want to be able to access the API from a remote location, deliver two integers, then run the python file with those integers.

I did find advice on How to run a script on your aws-ec2 from a script on your local machine?

ssh -i key.pem ubuntu@ec2-instance "bash /path/to/your/script/data.sh"

So for my own use case, I have tried modifying it to:

ssh -i python_bot.pem ubuntu@ec2-instance "python Python_Project/python_script.py 2 166"

However, when I run this code from my own shell, I get bash: python: command not found. This is strange, because if I do ssh -i python_bot.pem ubuntu@ec2-instance and then once logged in do python Python_Project/python_script.py 2 166 the file runs successfully. I am guessing if I can get this to work, then I can try some solutions from Can bash script be written inside a AWS Lambda function

I have also found Running Python Script in an existing EC2 instance on AWS and while the answer gave me some direction, it is too broad to help with a specific solution.

Jared C
  • 362
  • 7
  • 19
  • 3
    Why do you have Lambda, AWS API Gateway, and SSH involved in this? I think it'd be easier to just have a Python-based web server on the EC2 box itself (e.g. Flask). – AKX Mar 06 '20 at 14:34
  • @AKX do you have a link to a proper walkthrough for setting up Flask to accomplish this? – Jared C Mar 06 '20 at 14:39
  • @jarmod I have been using search engines for several hours over the past few days. The results that I have found are about setting up web servers with domains and webpages and etc. I just want to be able to have an external web server perform an ad hoc request of the file with the two integer variables. – Jared C Mar 06 '20 at 14:43
  • 1
    In response to the issue of "bash: python: command not found" when attempting this via SSH, clearly this indicates that the PATH you were expecting to be there is not there. If you do a little research on SSH you will find that SSH command execution invokes a non-interactive shell, which is different to what you get when you actually log into Linux. For more, see https://stackoverflow.com/questions/216202/why-does-an-ssh-remote-command-get-fewer-environment-variables-then-when-run-man – jarmod Mar 06 '20 at 14:46
  • 1
    No, no walkthrough tutorial, but here're some pointers. https://gist.github.com/akx/b0af7a9c7a0e46792556f1c0e24f796b – AKX Mar 06 '20 at 14:55
  • 2
    On the Flask topic, a quick search for "simple flask app with api" yields good results. Here's the first: https://programminghistorian.org/en/lessons/creating-apis-with-python-and-flask#creating-a-basic-flask-application. If you still prefer to do this via SSH then Python has libraries to do what you want e.g. paramiko. But a design that includes API call to API Gateway to Lambda to SSH to EC2 to invoke Python script is not ideal. – jarmod Mar 06 '20 at 14:57
  • Thank you, I'll read up on Flask and try out the suggestions. If I can get this working rather than the lambda approach it will obviously be preferred. – Jared C Mar 06 '20 at 15:01
  • I managed to get everything working with Flask, thanks for the suggestions! – Jared C Mar 07 '20 at 16:56

2 Answers2

6

Trigger a script on an Amazon EC2 instance from an AWS Lambda function is not a good architectural design.

As described in Running Python Script in an existing EC2 instance on AWS (to which you linked), a computer doesn't generally like things being "pushed" to it, unless it is running as a web server.

Instead, I would like to suggest to you a different architecture:

  • The Lambda function pushes a 'job' onto an Amazon SQS queue
  • Code on the Amazon EC2 instance regularly checks ('polls') the SQS queue to see whether any messages are available
  • If a message is available, it uses the content of the message to trigger some local work on the EC2 instance

This way, the two systems are disconnected, but are communicating via the SQS queue. It has the additional benefit that, if the EC2 instance is busy or offline, the messages (jobs) accumulate in the SQS queue until they can be processed.

This architecture is better than attempting to use SSH or SCP to trigger a job (which is not a very 'cloud' way to architect a system).

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • I saw this approach, but it did not seem reasonable for my use case. The reason is that I need the python file on the EC2 instance to be executed immediately, for results to occur in real time. Having the SQS queue check every 1 or 2 seconds seems like it would be an unnecessary expenditure of resources, when the actual ad hoc requests would occur a few times per day. – Jared C Mar 07 '20 at 14:59
  • Just checking... what was the reason for not doing the processing in Lambda itself? – John Rotenstein Mar 07 '20 at 23:43
  • lambda does not support complicated projects with lots of extra packages. – Jared C Mar 09 '20 at 20:14
  • @JaredC Did you find a solution for your case? – ambigus9 Jan 14 '22 at 17:29
  • @ambigus9 I ended up using Flask on the EC2 server to receive requests, rather than involving lambda in any way. – Jared C Jan 15 '22 at 08:06
3

Login shells and running commands via SSH may have different environment variables, especially PATH here. Being explicit about the interpreter should help.

Log in to the instance and see which Python you're using:

which python

Then substitute that into your SSH command line, e.g.

ssh -i python_bot.pem ubuntu@ec2-instance "/usr/bin/python Python_Project/python_script.py 2 166"
AKX
  • 152,115
  • 15
  • 115
  • 172
  • Great, that solved the command not found error! Now I just need to figure out how to execute this ssh script from a lambda function to get this approach to work. But maybe the Flask suggestion will be a better approach? – Jared C Mar 06 '20 at 14:47