Kubernetes + Gitlab CI artefacts : What kind of volume should I use?

Question

I have a running Gitlab CI pipeline in my Kubernetes cluster.

When the tests are failing, I need to grab the app screenshots and logs from the pod where it ran so that they are available where the Gitlab Runner is expecting them.

I tried the kubectl cp <namespace>/<podname>:/in-pod-path /local/path to copy the files from a stopped pod (having the tar command installed in my Docker image), but it isn't yet supported.

Until this is available, I guess I need a volume mounted in the pod at the path where are saved my artefacts so that I can grab them from this volume after the tests execution is finished.

I'm wondering what kind of volume should I use knowing that I have 3 kube workers, I don't need that volume to be persistent over time, more to be shared across the nodes?

I'm expecting to deploy this volume before deploying the pod running my tests mounting this volume. When tests failure is detected, I would extract the artefacts to the right place and delete the pod and the volume.

Answer 1

You could try and define a PVC with access mode ReadWriteMany, in order to get a volume shared between multiple pods.
See "How to share storage between Kubernetes pods?"

It would still be a persistent volume (to support that), with all the pods scheduled to the node with that volume.

There are several volume types that are suitable for that and not tied to any cloud provider:

• NFS
• RBD (Ceph Block Device)
• CephFS
• Glusterfs
• Portworx Volumes

But:

I don't really need to share the volume between many pods, I'm fine to create a volume per pods.
I'd like to avoid installing/configuring a node shared volume service from the list you gave.
I'm looking for an ephemeral volume if that is possible?

Then an ephemeral storage is possible:

Kubernetes version 1.8 introduces a new resource, ephemeral-storage for managing local ephemeral storage. In each Kubernetes node, kubelet’s root directory (/var/lib/kubelet by default) and log directory (/var/log) are stored on the root partition of the node.
This partition is also shared and consumed by Pods via emptyDir volumes, container logs, image layers and container writable layers.

In your case, you need a runtime ephemeral storage.