3

Problem

Docker image compiles successfully, however fails when ran from Lambda because of its read only file system.

Summary

Luminati-proxy has a docker integration for their proxy manager. I copied over their docker file and appended it to my own docker file for pushing out a script to AWS Lambda. The building of the docker image was successful, but when pushed off to Lambda, it failed because of a read only file system error:

Failed to create directory /home/sbx_user1051/proxy_manager: [code=EROFS] Error: EROFS: read-only file system, mkdir '/home/sbx_user1051'
2022-02-28 19:37:22.049 FILE (8): Failed to create directory /home/sbx_user1051/proxy_manager: [code=EROFS] Error: EROFS: read-only file system, mkdir '/home/sbx_user1051'

Analysis

Upon examining the trackback, the error is focused on the proxy_manager installation and fails with directory changes (mkdir, mk_work_dir ...). These changes are made within the .js files of the GitHub which is pulled from the docker file as the proxy_manager installation. Obviously the only mutable directory on Lambda is the /tmp directory, but is there a workaround for getting this set up without resorting to putting everything under the /tmp directory as it wipes itself upon runtime? Reinstalling a proxy_manager each run is not at all ideal...

Answer?

Could this be as simple as setting environment stipulations such as:

ENV PATH=...
ENV LD_LIBRARY_PATH=...

If so, I how should they be configured? I am adding the docker file below for quick reference:

FROM node:14.18.1
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
    && apt-get update \
    && apt-get install -y google-chrome-stable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf \
      --no-install-recommends \
    && rm -rf /var/lib/apt/lists/*

USER root
RUN npm config set user root
RUN npm install -g npm@8.1.3
RUN npm install -g @luminati-io/luminati-proxy
ENV DOCKER 1
CMD ["luminati", "--help"]

I appreciate the insight!

Luke Hamilton
  • 637
  • 5
  • 19
  • This question should be reduced purely to getting `proxy_manager` to run under different directories - there is no inherent issue with AWS here and you're more likely to get answers in that case as this is 2 Qs related in 1. – Ermiya Eskandary Mar 06 '22 at 08:26
  • @ErmiyaEskandary I do realize that, I am trying to be as general as possible so that the question can get answered as well as aid others who have the same problem. I could have worded it better and expressed how docker image works locally but not on AWS because of AWS directory restrictions. That is good insight though, I appreciate it – Luke Hamilton Mar 07 '22 at 02:30
  • @LukeHamilton I don't get it RUN command is done during build, so how is `npm install` for `luminati-proxy` be the root cause? The other `npm install` would also be modifying other directories. Are you saying the error is actually from CMD, just that it is using the files pulled during install? – Register Sole Mar 11 '22 at 02:52
  • Also wouldn't the proxy try to write logs, which will again fail due to the read-only restriction? To me sounds like you have to configure the working dir in `tmp` anyway. If the `mkdir` is a one time thing, I think you can initiate this by executing the command as a `RUN luminati --help` once? – Register Sole Mar 11 '22 at 03:08
  • @RegisterSole I am confused by it as well. I have offboarded other Docker images to Lambda with no trouble which means it has to be the proxy portion of the docker file. It's hard to pinpoint exactly where the error comes from, the only thing I know is that there is a problem with the write/read file permissions which explains the error in the summary portion of my post. – Luke Hamilton Mar 11 '22 at 15:15

2 Answers2

4

TL;DR:

  • You should instead leverage an S3 bucket to store, read and modify any file. All lambdas and microservices. In general, should always be treated as stateless
  • All Luminati-proxy functionality comes prebuilt within amazon lambdas and API Gateway
  • Lambda functions are not meant to run long-running processes as they are limited to 15 minutes maximum so the design of the container that you are trying to run in lambdas has to have AWS serverless architecture considerations in its design

Explanation:

According to the documentation of AWS Lambda functions:

The container image must be able to run on a read-only file system. Your function code can access a writable /tmp directory with 512 MB of storage.

Since containers based on Linux based images are already supposed to have a folder called /tmp you should pretty much be able to access that folder any time from your code to read( remember, read-only FS)

If you are looking to store content amazon's solution for that is for you to have any content created and manage over an S3 bucket, buckets are as easy to use as if you read a file locally but will remain accessible after the lambda instance finishes the workload

Please refer to Read file from aws s3 bucket using node fs and Upload a file to Amazon S3 with NodeJS for more details on how to use an S3 bucket. There are plenty of ways to achieve it regardless of the language been used.

This is all based on a best practice promoted by AWS over their platform. Where lambdas remain stateless

lcarvajal
  • 483
  • 1
  • 3
  • 11
  • Sounds like the proxy_manager should be built and stored within S3 then I can reference it from there? – Luke Hamilton Mar 16 '22 at 14:16
  • all that Luminate-proxy is doing is functionality you have available prebuilt in AWS cloud , im going to update my answer to cover that as well – lcarvajal Mar 17 '22 at 15:09
1

AWS Lambda provides /tmp folder for users to write files on lambda, as I don't about your question context but hope this help. You can write files to AWS Lambda at /tmp folder eg. I want to create a file demo.txt at runtime/programmatically using AWS lambda, then i can write the file to /txt/demo.txt

Dharman
  • 30,962
  • 25
  • 85
  • 135
sandeep.kokane
  • 66
  • 4