I am trying to use AWS Chalice to upload my Python application to a Lambda function. I need to use a number of dependencies that will exceed the 50MB limit for a Lambda app. I think I can overcome this limit by using Lambda layers for each dependency, provided each layer is below the 50MB limit. For my final app, the total unzipped deployment size is about 225 MB so it is below the 250 MB unzipped limit imposed for Lambda deployments. When I has issues with the full deployment, I decided to try increasing the size by adding dependencies. I have been successful in using Chalice to deploy an app using Lambda layers that has scipy and numpy as dependencies. The deployment size with scipy and numpy dependencies is 48.6 MB zipped and 157 MB unzipped. When I also add matplotlib the size is 66 MB zipped and 210 MB unzipped and causes an error as described below.
My requirements.txt file contains:
scipy>=1.7.0
numpy>=1.19.2
matplotlib>=3.5.0
My chalice config file contains:
{
"automatic_layer": true,
"stages": {
"dev": {
"api_gateway_stage": "api"
}
},
"version": "2.0",
"app_name": "demoapp"
}
My chalice app code looks like this:
from chalice import Chalice
app = Chalice(app_name='demoapp')
app.debug = True
@app.lambda_function()
def numpy_import(event, context):
import numpy
return {'numpy': numpy.__file__}
@app.lambda_function()
def scipy_import(event, context):
import scipy
return {'scipy': scipy.__file__}
@app.lambda_function()
def matplotlib_import(event, context):
import matplotlib
return {'matplotlib': matplotlib.__file__}
@app.route('/myapp', methods=['POST'])
def mufunction():
jsonbody = app.current_request.json_body
# extract data from json
data_array = jsonbody['data_array']
# do stuff. As a test case it is just returning what it was given.
result = data_array
# package output in json
output = {'result': result}
return output
The stack trace of the error I get is:
Creating shared layer deployment package.
Reusing existing shared layer deployment package.
Creating app deployment package.
Creating lambda layer: demoapp-dev-managed-layer
Traceback (most recent call last):
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\awsrequest.py", line 94, in _send_request
rval = super()._send_request(
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\awsrequest.py", line 123, in _send_output
self.send(msg)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\awsrequest.py", line 218, in send
return super().send(str)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 971, in send
self.sock.sendall(data)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1204, in sendall
v = self.send(byte_view[count:])
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1173, in send
return self._sslobj.write(data)
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\httpsession.py", line 455, in send
urllib_response = conn.urlopen(
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\util\retry.py", line 507, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
raise value.with_traceback(tb)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\awsrequest.py", line 94, in _send_request
rval = super()._send_request(
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\awsrequest.py", line 123, in _send_output
self.send(msg)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\awsrequest.py", line 218, in send
return super().send(str)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 971, in send
self.sock.sendall(data)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1204, in sendall
v = self.send(byte_view[count:])
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1173, in send
return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionAbortedError(10053, 'An established connection was aborted by the software in your host machine', None, 10053, None))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\chalice\cli\__init__.py", line 636, in main
return cli(obj={})
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1055, in main
rv = self.invoke(ctx)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\click\decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\chalice\cli\__init__.py", line 189, in deploy
deployed_values = d.deploy(config, chalice_stage_name=stage)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\chalice\deploy\deployer.py", line 376, in deploy
return self._deploy(config, chalice_stage_name)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\chalice\deploy\deployer.py", line 392, in _deploy
self._executor.execute(plan)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\chalice\deploy\executor.py", line 42, in execute
getattr(self, '_do_%s' % instruction.__class__.__name__.lower(),
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\chalice\deploy\executor.py", line 55, in _do_apicall
result = method(**final_kwargs)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\chalice\awsclient.py", line 334, in publish_layer
return self._client('lambda').publish_layer_version(
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\client.py", line 514, in _api_call
return self._make_api_call(operation_name, kwargs)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\client.py", line 921, in _make_api_call
http, parsed_response = self._make_request(
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\client.py", line 944, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\endpoint.py", line 119, in make_request
return self._send_request(request_dict, operation_model)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\endpoint.py", line 202, in _send_request
while self._needs_retry(
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\endpoint.py", line 354, in _needs_retry
responses = self._event_emitter.emit(
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\hooks.py", line 412, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\hooks.py", line 256, in emit
return self._emit(event_name, kwargs)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\hooks.py", line 239, in _emit
response = handler(**kwargs)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\retryhandler.py", line 207, in __call__
if self._checker(**checker_kwargs):
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\retryhandler.py", line 284, in __call__
should_retry = self._should_retry(
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\retryhandler.py", line 320, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\retryhandler.py", line 363, in __call__
checker_response = checker(
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\retryhandler.py", line 247, in __call__
return self._check_caught_exception(
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\retryhandler.py", line 416, in _check_caught_exception
raise caught_exception
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\endpoint.py", line 281, in _do_get_response
http_response = self._send(request)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\endpoint.py", line 377, in _send
return self.http_session.send(request)
File "C:\Users\Matthew Reid\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\httpsession.py", line 494, in send
raise ConnectionClosedError(
botocore.exceptions.ConnectionClosedError: Connection was closed before we received a valid response from endpoint URL: "https://lambda.ap-southeast-2.amazonaws.com/2018-10-31/layers/demoapp-dev-managed-layer/versions".
I think this error is a connection timeout issue caused by my computer. I have tried to fix this by adjusting the DEFAULT_TIMEOUT variable in botocore from 60 to 600. I also deploy with the command “chalice deploy --connection-timeout 600”.
My internet upload speed is approximately 5 MB/s so for a 66 MB file it should manage that in about 15 seconds.
If you have any suggestions about how to deploy an app that has multiple python dependancies that exceed 50MB when zipped, then I’d love to know your solution.
For reference, I have looked at these similar questions:
