Can I copy a directory from some location outside the docker area to my dockerfile?

Question

I have installed a library called fastai==1.0.59 via requirements.txt file inside my Dockerfile.

But the purpose of running the Django app is not achieved because of one error. To solve that error, I need to manually edit the files /site-packages/fastai/torch_core.py and site-packages/fastai/basic_train.py inside this library folder which I don't intend to.

Therefore I'm trying to copy the fastai folder itself from my host machine to the location inside docker image.

source location: /Users/AjayB/anaconda3/envs/MyDjangoEnv/lib/python3.6/site-packages/fastai/

destination location: ../venv/lib/python3.6/site-packages/ which is inside my docker image.

being new to docker, I tried this using COPY command like: COPY /Users/AjayB/anaconda3/envs/MyDjangoEnv/lib/python3.6/site-packages/fastai/ ../venv/lib/python3.6/site-packages/

which gave me an error:

ERROR: Service 'app' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder583041406/Users/AjayB/anaconda3/envs/MyDjangoEnv/lib/python3.6/site-packages/fastai: no such file or directory.

I tried referring this: How to include files outside of Docker's build context? but seems like it bounced off my head a bit..

Please help me tackling this. Thanks.

Dockerfile:

FROM python:3.6-slim-buster AS build
MAINTAINER model1
ENV PYTHONUNBUFFERED 1


RUN python3 -m venv /venv
RUN apt-get update && \
    apt-get upgrade -y && \
    apt-get install -y git && \
    apt-get install -y build-essential && \
    apt-get install -y awscli && \
    apt-get install -y unzip && \
    apt-get install -y nano && \
    apt-get install -y libsm6 libxext6 libxrender-dev


RUN apt-cache search mysql-server
RUN apt-cache search libmysqlclient-dev
RUN apt-get install -y libpq-dev
RUN apt-get install -y postgresql
RUN apt-cache search postgresql-server-dev-9.5
RUN apt-get install -y libglib2.0-0

RUN mkdir -p /model/
COPY . /model/
WORKDIR /model/

RUN pip install --upgrade awscli==1.14.5 s3cmd==2.0.1 python-magic
RUN pip install -r ./requirements.txt

EXPOSE 8001

RUN chmod -R 777 /model/

COPY /Users/AjayB/anaconda3/envs/MyDjangoEnv/lib/python3.6/site-packages/fastai/ ../venv/lib/python3.6/site-packages/

CMD python3 -m /venv/activate
CMD /model/my_setup.sh development
CMD export API_ENV = development
CMD cd server && \
    python manage.py migrate && \
    python manage.py runserver 0.0.0.0:8001

Any time you have to edit a file in `site-packages` is a red flag. What kind of change needs to be made? Can you post your Dockerfile so we can see what you are attempting to do? — C.Nivs, Mar 23 '20 at 14:43
@C.Nivs - The folder ```fastai``` in host machine has extra lines ```import dill``` on top and ```state = torch.load(source, map_location='cpu', pickle_module=dill) if defaults.device == torch.device('cpu') else torch.load(source, pickle_module=dill)``` on line 403 in these files., that are not in library by default. This way the error would be solved. — Ajay Bhagchandani, Mar 23 '20 at 14:47
@00 The error that I got without the ```COPY``` statement in Dockerfile is ```AttributeError: Can't get attribute '__main__' on ``` which is solved in our local machines is by editing the ```fastai``` files., which reside outside the docker scope. Therefore asked this question just to know if I could copy a directory from a remote location on the machine to a location inside docker image. — Ajay Bhagchandani, Mar 23 '20 at 14:52
@AjayBhagchandani you would have to move that file into the build context (copy from site-packages to where the dockerfile is) — C.Nivs, Mar 23 '20 at 14:56
@AjayBhagchandani Instead of modifying the source, have you tried monkeypatching `torch.load` to set the pickle arg before importing `fastai` in your own code? You can see an example of this in [fastai itself](https://github.com/fastai/fastai/blob/master/fastai/torch_core.py#L69). — tzaman, Mar 23 '20 at 15:05
@tzaman - unfortunately, didn't know about this. Thanks for helping it out. I ain't a data scientist, just a backend developer here who integrates the models in Django app. Will forward it to the concerned person and will let him know about this approach. — Ajay Bhagchandani, Mar 23 '20 at 15:57
@C.Nivs - That helped me a lot and now the server is running smooth. It didn't click me earlier. Thanks a lot. — Ajay Bhagchandani, Mar 23 '20 at 15:58

Software Engineer · Accepted Answer · 2020-03-23T15:06:58.480

Short Answer

No

Long Answer

When you run docker build the current directory and all of its contents (subdirectories and all) are copied into a staging area called the 'build context'. When you issue a COPY instruction in the Dockerfile, docker will copy from the staging area into a layer in the image's filesystem.

As you can see, this procludes copying files from directories outside the build context.

Workaround

Either download the files you want from their golden-source directly into the image during the build process (this is why you often see a lot of curl statements in Dockerfiles), or you can copy the files (dirs) you need into the build-tree and check them into source control as part of your project. Which method you choose is entirely dependent on the nature of your project and the files you need.

Notes

There are other workarounds documented for this, all of them without exception break the intent of 'portability' of your build. The only quality solutions are those documented here (though I'm happy to add to this list if I've missed any that preserve portability).

That was very helpful. Thanks a lot! – Ajay Bhagchandani Mar 23 '20 at 15:59 — Ajay Bhagchandani, Mar 23 '20 at 15:59

Can I copy a directory from some location outside the docker area to my dockerfile?

1 Answers1