0

I have a python lambda function that is using requests to scrape metrics from Prometheus, which is http. I began by zipping the requests packages and a python function on my local machine. I uploaded and attached the zip file as a layer to the lambda function. When I test the function it times out on the GET request

import requests
import copy 
import json
print("Imported packages successfull" )

# Grabbing metadata on metrics categoriies
print("Reading URL")
URL_meta = 'http://xxx.xxx.x.xxx:9091/api/v1/metadata'
print("URL read successfull")

print("Initiating requests.get")
r = requests.get(url = URL_meta) <--------- Times out 
print("successful requests.get")

metadata = r.json()

I then made a VPC 10.0.0.0/16, a Public and Private Subnet which I attached to their respective Public and Private Route tables, an Internet Gateway attached to the VPC, a Public NAT Gateway connected to the Public Subnet, I made a Role and attached the permission policies: AWSLambdaExecute and AWSLambdaVPCAccessExecutionRole. I then went to the Lambda Function > Configuration > VPC, and added the VPC I made, attached the Private Subnet and the default VPC Security group to that. I then went to Lambda Function > Configuration > Permissions, and assigned the Execution Role.

When I run the function it times out at the same location. Here are the logs:

Function Logs
START RequestId: aa1e5ffa-ed04-4274-9a66-a5ac455b5325 Version: $LATEST
Imported packages successfull
Reading URL
URL read successfull
Initiating requests.get
END RequestId: xxxxxxxxx

I have successfully run the same requests function from a Jupyter Notebook on my local machine and have verified that the http address can be accessed from various devices.

I have increased the timeout period to 30 seconds. I have searched google, I have checked SO:

Cannot use Requests-Module on AWS Lambda

Issue with VPC attached to Lambda function

Why is my Lambda unable to access the internet?

How to make a HTTP rest call in AWS lambda using python?

Python request in AWS Lambda timing out

get request Task timed out in Lambda using Python

I've watched numerous YouTube videos, poured over the AWS Docs, read Medium articles, and I checked the "Similar Questions" that were suggested when I initiated this. I am stuck.

zelfde
  • 300
  • 2
  • 13
  • 1
    Let's start simple... If the Lambda function is _NOT_ attached to a VPC, does the GET request timeout? Also, is there a particular reason why you want to attach the Lambda function to a VPC? – John Rotenstein Aug 08 '22 at 04:14
  • Yes it times out without a VPC. I only have one attached because it timed out on the GET request @JohnRotenstein – zelfde Aug 08 '22 at 04:24
  • 2
    Do not attach a VPC unless you have a need to access resources inside the VPC. When a VPC is _not_ connected, the Lambda function has direct access to the Internet. It is possible that the website you are connecting to is blocking requests coming from AWS. If you login to an EC2 instance and try to `curl` that URL, does it work? – John Rotenstein Aug 08 '22 at 04:43
  • Now that I have not tried, an EC2 instance. – zelfde Aug 08 '22 at 04:52
  • 1
    It looks like you are calling an official Prometheus API, so they are unlikely to be blocking it. – John Rotenstein Aug 08 '22 at 04:54
  • That’s what has me stumped. – zelfde Aug 08 '22 at 04:57
  • 2
    I notice in the Prometheus documentation that the examples refer to `curl 'http://localhost:9090/api...`. This suggests that you need to install and run Prometheus on your own server. What domain name are you pointing to? Is it to your own computer? Is it accessible from the Internet? I would again suggest connecting to it from an EC2 instance (even just using `curl` to see if it responds). – John Rotenstein Aug 08 '22 at 06:45
  • It is pointing to a computer that is not accessible from the internet. But I thought the Prometheus metrics were stored on their servers and are accessible from the internet. I’m not sure of that though. I will try connecting to an ec2 instance and get back to you this week sometime. Thanks for the response @JohnRotenstein, I appreciate it – zelfde Aug 08 '22 at 12:13

0 Answers0