2

I have been interacting with Docker for a while. For all of its great promise, tutorial and books, no one seems to be talking about how to actually use Docker in reality. I've found many examples, running Nginx with a static website. But there's never a complete workflow.

I have a docker composer system, both develop and production. It's in a git repo. I can clone it into any server, run a simple script, a docker container spins up and I have a website. Or at least a piece of one. The stack is basically Django + Gunicorn + Nginx + postgres all dockerized .

Yet, that's where docker fails for me. The container needs data, from a database and some file (images, etc) that are currently data volumes. If I spin the container down, that data is lost, necessitating loading of the data when container runs. My development consists of improving the front end and back end.

I have yet to understand how to actually push this to a real system. People talk about docker swarm and Kubernetes to deploy applications. Where is the actual data? How is that handled?. I currently have to have it in my repository, then run manual commands to load the postgres database and copy the static files over.

My goal is to actually automate things and make it easier to deploy (as close to one click push as possible). Yet every tutorial starts talking about the details of docker which everyone repeats ad nauseum (how to build an image, the scripts, etc). It's incredibly frustrating.

Is docker only used to setup the main architecture? Where is all the specific application data stored? How is that tracked and developed? Am I supposed to have an architecture where I can edit a repo and push it to the server?

I also have issues where the container takes time to spin up. How does this work in production? Do I push it to another server then change the IP addresses? Or do I put the new docker in the same server and do something? (I assume IP ports will conflict)

What about when there's a reboot of the server running the container? They don't start back up.

Thank you for any help

Here's my docker compose file. As per many recommendations online, uses volumes, environment variables, etc etc. It runs, but it's not a movable container. Even when I call down without the -v (for volumes) they go down and don't stay there when I do another up command

docker-compose -f docker-compose.prod.yml up -d --build
docker-compose -f docker-compose.prod.yml down -v

This is the compose file

version: '3.7'

services:
  web:
    build:
      context: ./app
      dockerfile: Dockerfile.prod
    command: gunicorn awebsite.wsgi:application --bind 0.0.0.0:8000
    volumes:
      - static_volume:/home/app/web/staticfiles
      - media_volume:/home/app/web/mediafiles
    expose:
      - 8000
    env_file:
      - ./.env.prod
    depends_on:
      - db
  db:
    image: postgres:13.4-alpine
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    env_file:
      - ./.env.prod.db
  nginx:
    build: ./nginx
    volumes:
      - static_volume:/home/app/web/staticfiles
      - media_volume:/home/app/web/mediafiles
    ports:
      - 80:80
    depends_on:
      - web

volumes:
  postgres_data:
  static_volume:
  media_volume:
Phil
  • 157,677
  • 23
  • 242
  • 245
odyssy3105
  • 97
  • 1
  • 8
  • _"Where is the actual data?"_... in your PostgreSQL database which could be running on a server you manage, [in a container itself](https://dzone.com/articles/the-pros-and-cons-of-running-production-databases) or as a managed service like [Amazon RDS](https://aws.amazon.com/rds/) or [Azure Database for PostgreSQL](https://azure.microsoft.com/en-us/services/postgresql/#overview) – Phil Sep 21 '21 at 03:47
  • @Phil - So let's assume the data is in another container, how is that handled? How is that moved from one server to another (my development machine to production) – odyssy3105 Sep 21 '21 at 03:48
  • Are you asking how to migrate data from one DB instance to another? If so, that doesn't really seem relevant to where / how those DB instances are running. – Phil Sep 21 '21 at 03:59
  • You developed an application in your development environment and now you need to deploy it. How can it run if it is missing critical database and other data? – odyssy3105 Sep 21 '21 at 04:02
  • 1
    Just an example: machine language workloads used docker a lot and the primary reason seems to be so that they can be very specific about the _runtime_., i.e. all the correct feral python libraries. The actual container refers to data that is permanently stored, but not in a container. The container is used as runtime/compute, never for permanent storage. Containers can of course refer to external resources. – Nick.Mc Sep 21 '21 at 04:07
  • I feel like your question is less about Docker / containerisation and more about infrastructure provisioning and / or DB migrations. Your question is also extremely broad and could have any number of solutions – Phil Sep 21 '21 at 04:07
  • Note in all of your existing experiments the focus is on getting the build and runtime right. That seems to be the purpose of containers from my perspective. Nothing about permanent storage. A database is a network service that your container connects to. The database may or may not be in a container. It seems to me that if you want to put a database in a container, you can never stop that container unless you have a strategy for preserving it's data. Perhaps this article will clarify: https://www.nocentino.com/posts/2019-09-01-persisting-sql-server-data-in-docker-containers-part-1/ – Nick.Mc Sep 21 '21 at 04:20
  • @Nick.McDermaid - I guess I need to start thinking about splitting things more, perhaps a separate db instance unrelated to docker and the static data unrelated to it as well and manage them separately. It's just docker is basically sold as a solution to move applications easily, when in fact parts of the application (the data) that is needed won't be there. – odyssy3105 Sep 21 '21 at 04:21
  • @Phil - I understand the question is broad but the purpose of containers is to improve infrastructure provisioning, etc. – odyssy3105 Sep 21 '21 at 04:21
  • The article basically suggest using the `-v` switch to define an _external_ volume. When your container stops, the external volume remains, and can be picked up again if you start a new container. – Nick.Mc Sep 21 '21 at 04:23
  • _"the purpose of containers is to improve infrastructure provisioning"_... I'd have to disagree with that one (unless I'm misunderstanding). IMO, containers are great for deploying your application. Infrastructure can be provisioned many ways (click-ops, automation tools like Terraform, etc) but I'm not sure how or where containers fit into that process – Phil Sep 21 '21 at 04:24
  • @Nick.McDermaid - Yep, my script defines volumes and gets run as separate volumes. Except once I stop the container the volumes are gone and the data is too. The official docker docs say nothing about this except the dangling volumes that may be connected later somehow, but there's no way to easily move the container. – odyssy3105 Sep 21 '21 at 04:26
  • @Phil - the containers are the infrastructure, or the most critical part of it.. No containers, -> no infrastructure -> No app – odyssy3105 Sep 21 '21 at 04:27
  • Have added specific code to show what my docker compose looks like. – odyssy3105 Sep 21 '21 at 04:32
  • 1
    All the articles say that the external volume persists, and the container refernencing can be deleted and recreated and just pick the external volume up again. I have never done it myself, that's just my understanding of how it's meant to work. "if we don’t use a Volume that data will be written into the writeable layer of the container and if we delete the container…we delete our data. We don’t want that so let’s start up a container with a Volume" – Nick.Mc Sep 21 '21 at 04:32
  • I call : docker-compose -f docker-compose.prod.yml down Then I call docker-compose -f docker-compose.prod.yml up -d --build And what happens is the original volumes and data are gone – odyssy3105 Sep 21 '21 at 04:33
  • Your volumes have no host mounts so they're not persistent – Phil Sep 21 '21 at 04:42
  • @Phil: Ok, then we've found part of the issue. Can you help me fix that? How can I move these volumes across systems too? Do I need to add external: true to them? I think I tried that at some point, not sure it worked. Maybe need to retry – odyssy3105 Sep 21 '21 at 04:43
  • I don't have enough experience to help any further. However I would check the external file after `down` and see if it's deleted or if it still exists. – Nick.Mc Sep 21 '21 at 04:45
  • 1
    Back to your original question, when you want to deploy a front end improvement I assume you just deploy the front end container. When you want to deploy a back end (database) change, you don't redeploy the container, you apply a datbase migration process (i.e. apply a drift script, apply data patch script). Again it's about the runtime. If you wanted to upgrade the postgres engine, then you would need to work out how to preserve the data files, but there are numerous articles claiming that this is done with external volumes – Nick.Mc Sep 21 '21 at 04:48
  • Does this answer your question? [Docker-compose named mounted volume](https://stackoverflow.com/questions/35841241/docker-compose-named-mounted-volume). I wouldn't attempt to move a volume across systems. Use a DB migration task to move schema and version-able data changes – Phil Sep 21 '21 at 04:48
  • @Nick.McDermaid - looks like that's the real answer. To decouple the database, the data volume, etc and treat them separately – odyssy3105 Sep 21 '21 at 04:50
  • @Phil - Looks like it explains a bit more about it but will have to test it – odyssy3105 Sep 21 '21 at 04:51

0 Answers0