1

I've got a file containing a list of paths that I need to copy by Dockerfile's COPY command on docker build.

My use case is such: I've got a python requirements.txt file, when inside I'm calling multiple other requirements files inside the project, with -r PATH.

Now, I want to docker COPY all the requirements files alone, run pip install, and then copy the rest of the project (for cache and such). So far i haven't managed to do so with docker COPY command.

No need of help on fetching the paths from the file - I've managed that - just if it is possible to be done - how?

thanks!

Peter Badida
  • 11,310
  • 10
  • 44
  • 90
drizzt13
  • 630
  • 6
  • 15
  • What's the single thing the image is going to do when you've built it; why you do have multiple sets of package dependencies you need to install? If the image has multiple applications, would it make more sense to have a separate image for each (and a single `requirements.txt` file)? – David Maze Jul 11 '21 at 23:33

2 Answers2

3

Not possible in the sense that the COPY directive allows it out of the box, however if you know the extensions you can use a wildcard for the path such as COPY folder*something*name somewhere/.

For simple requirements.txt fetching that could be:

# but you need to distinguish it somehow
# otherwise it'll overwrite the files and keep the last one
# e.g. rename package/requirements.txt to package-requirements.txt
# and it won't be an issue
COPY */requirements.txt ./
RUN for item in $(ls requirement*);do pip install -r $item;done

But if it gets a bit more complex (as in collecting only specific files, by some custom pattern etc), then, no. However for that case simply use templating either by a simple F-string, format() function or switch to Jinja, create a Dockerfile.tmpl (or whatever you'd want to name a temporary file), then collect the paths, insert into the templated Dockerfile and once ready dump to a file and execute afterwards with docker build.

Example:

# Dockerfile.tmpl
FROM alpine
{{replace}}

# organize files into coherent structures so you don't have too many COPY directives
files = {
    "pattern1": [...],
    "pattern2": [...],
    ...
}
with open("Dockerfile.tmpl", "r") as file:
    text = file.read()

insert = "\n".join([
    f"COPY {' '.join(values)} destination/{key}/"
    for key, values in files.items()
])

with open("Dockerfile", "w") as file:
    file.write(text.replace("{{replace}}", insert))
Peter Badida
  • 11,310
  • 10
  • 44
  • 90
  • 1
    This is a very flexible yet still very readable solution. And paired with multistage I can now imagine the possibilities. Thank you for this! – Jerven Clark Jul 12 '21 at 02:15
1

You might want to do this for example:

FROM ...
ARG files
COPY files

and run with

docker build -build-args items=`${cat list_of_files_to_copy.txt}`
Alberto Sinigaglia
  • 12,097
  • 2
  • 20
  • 48
  • Note that environment variables (at least on Windows) [have length limit](https://superuser.com/q/1070272/571014) and although there [doesn't seem to be one for Linux](https://stackoverflow.com/a/1078125/5994041) it's strictly dependent on how Docker retrieves the value and if Docker or the underlying OS distro implements any limitation. Also, whitespace / spaces in names can cause problems as well if just `cat`ed. – Peter Badida Jul 11 '21 at 22:59
  • it is a good option in general, but if the requirements files all are with the same name (and they are and i don't have controll over it) then they will run over each other. such is my case. thanks anyway! – drizzt13 Jul 12 '21 at 07:24