1

I have been struggling with this issue for a while now. I want to do a performance test using a specific set of data. To achieve this I am using Docker-compose and I have exported a couple of .sql files.

When I first build and run the container using docker-compose build and docker-compose up the dataset is fine. Then I run my test (which also inserts) data. After running my test I want to perform it again, so I relaunch the container using docker-compose up. But this time, for some reason I don't understand, the data that was inserted the last time (by my test) is still there. I get different behaviour now.

At the moment I have the following Dockerfile:

FROM mysql:5.7

ENV MYSQL_DATABASE dev_munisense1\
    MYSQL_ROOT_PASSWORD pass

EXPOSE 3306

ADD docker/data/ /docker-entrypoint-initdb.d/

Because I read that the mysql Docker image runs everything in /docker-entrypoint-initdb.d/. The first time it works properly.

I have also tried what these posts suggested:

How do i migrate mysql data directory in docker container?

http://txt.fliglio.com/2013/11/creating-a-mysql-docker-container/

How to create populated MySQL Docker Image on build time

How to make a docker image with a populated database for automated tests?

And a couple of identical other posts.

None of them seem to work currently.

How can I make sure the dataset is exactly the same each time I launch the container? Without having to rebuild the image each time (this takes kind of long because of a large dataset).

Thanks in advance

EDIT:

I have also tried running the container with different arguments like: docker-compose up --force-recreate --build mysql but this has no success. The container is rebuilt and restarted but the db is still affected by my test. Currently the only solution to my problem is to remove the entire container and image.

Chris Snow
  • 23,813
  • 35
  • 144
  • 309
Anton S
  • 21
  • 4
  • This answer might be useful for you: http://stackoverflow.com/a/27572665/1856086 – Stas Makarov Mar 07 '17 at 14:12
  • @StasMakarov well it's the exact opposite of what I want. I do not want to persist the database. I just want to start a container from a certain point, do stuff to it, and then throw it all away, just so i can start over from the same exact beginning... – Anton S Mar 07 '17 at 14:18
  • You can try to create your own mysql image using [official Dockerfile](https://github.com/docker-library/mysql/blob/eeb0c33dfcad3db46a0dfb24c352d2a1601c7667/5.7/Dockerfile#L68) but without `VOLUME /var/lib/mysql` line (#68 at the moment). This way it shouldn't save any data after deleting container. It's probably far from the best solution of this problem, but it might work. – Stas Makarov Mar 07 '17 at 16:14
  • Also you can try to remove a volume with mysql data with `docker volume rm $volume_hash`. Or remove all volumes with `docker volume rm $(docker volume ls -q)`. Last command can delete useful stuff. – Stas Makarov Mar 07 '17 at 16:19
  • @StasMakarov When using the exact Dockerfile without that line #68, how can i fill the image so each time i spin up the container it is pre filled? I dont mind having a larger image, as long as the container creating is short – Anton S Mar 08 '17 at 09:01

2 Answers2

1

I managed to fix the issue (with mysql image) by doing the following:

  1. change the mounting point of the sql storage (This is what actually caused the problem) I used the solution suggested here: How to create populated MySQL Docker Image on build time but I did it by running a sed command: RUN sed -i 's|/var/lib/mysql|/var/lib/mysql2|g' /etc/mysql/my.cnf

  2. add my scripts to a folder inside the container

  3. run import.sh script that inserts data using the daemon (using the wait-for-it.sh script)
  4. remove the sql scripts
  5. expose port like regular

The docker file looks like this (the variables are used to select different SQL files, I wanted multiple versions of the image):

FROM mysql:5.5.54

ADD https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh /utils/wait-for-it.sh
COPY docker/import.sh /usr/local/bin/

RUN sed -i 's|/var/lib/mysql|/var/lib/mysql2|g' /etc/mysql/my.cnf

ARG MYSQL_DATABASE
ARG MYSQL_USER
ARG MYSQL_PASSWORD
ARG MYSQL_ROOT_PASSWORD
ARG MYSQL_ALLOW_EMPTY_PASSWORD
ARG DEVICE_INFORMATION
ARG LAST_NODE_STATUS
ARG V_NODE
ARG NETWORKSTATUS_EVENTS

ENV MYSQL_DATABASE=$MYSQL_DATABASE \
    MYSQL_USER=$MYSQL_USER \
    MYSQL_PASSWORD=$MYSQL_PASSWORD \
    MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD \
    DEVICE_INFORMATION=$DEVICE_INFORMATION \
    LAST_NODE_STATUS=$LAST_NODE_STATUS \
    V_NODE=$V_NODE \
    MYSQL_ALLOW_EMPTY_PASSWORD=$MYSQL_ALLOW_EMPTY_PASSWORD

#Set up tables
COPY docker/data/$DEVICE_INFORMATION.sql /usr/local/bin/device_information.sql
COPY docker/data/$NETWORKSTATUS_EVENTS.sql /usr/local/bin/networkstatus_events.sql
COPY docker/data/$LAST_NODE_STATUS.sql /usr/local/bin/last_node_status.sql
COPY docker/data/$V_NODE.sql /usr/local/bin/v_node.sql


RUN chmod 777 /usr/local/bin/import.sh && chmod 777 /utils/wait-for-it.sh && \
    /bin/bash /entrypoint.sh mysqld --user='root' & /bin/bash /utils/wait-for-it.sh -t 0 localhost:3306 -- /usr/local/bin/import.sh; exit
RUN rm -f /usr/local/bin/*.sql

ENTRYPOINT ["docker-entrypoint.sh"]

EXPOSE 3306
CMD ["mysqld"]

the script looks like this:

#!/bin/bash
echo "Going to insert the device information"
mysql  --user=$MYSQL_USER --password=$MYSQL_PASSWORD --database=$MYSQL_DATABASE < /usr/local/bin/device_information.sql
echo "Going to insert the last_node_status"
mysql  --user=$MYSQL_USER --password=$MYSQL_PASSWORD --database=$MYSQL_DATABASE < /usr/local/bin/last_node_status.sql
echo "Going to insert the v_node"
mysql  --user=$MYSQL_USER --password=$MYSQL_PASSWORD --database=$MYSQL_DATABASE < /usr/local/bin/v_node.sql
echo "Going to insert the networkstatus_events"
mysql  --user=$MYSQL_USER --password=$MYSQL_PASSWORD --database=$MYSQL_DATABASE < /usr/local/bin/networkstatus_events.sql
echo "Database now has the following tables"
mysql --user=$MYSQL_USER --password=$MYSQL_PASSWORD --database=$MYSQL_DATABASE --execute="SHOW TABLES;"

So now all I have to do to start my performance tests is 

#!/usr/bin/env bash
echo "Shutting down previous containers"
docker-compose -f docker-compose.yml down
docker-compose -f docker-compose-test-10k-half.yml down
docker-compose -f docker-compose-test-100k-half.yml down
docker-compose -f docker-compose-test-500k-half.yml down
docker-compose -f docker-compose-test-1m-half.yml down

echo "Launching rabbitmq container"
docker-compose up -d rabbitmq & sh wait-for-it.sh -t 0 -h localhost -p 5672 -- sleep 5;

echo "Going to execute 10k test"
docker-compose -f docker-compose-test-10k-half.yml up -d mysql_10k & sh wait-for-it.sh -t 0 -h localhost -p 3306 -- sleep 5 && ./networkstatus-event-service --env=performance-test --run-once=true;
docker-compose -f docker-compose-test-10k-half.yml stop mysql_10k

couple of more of these lines (slightly different, cause different container names)

Community
  • 1
  • 1
Anton S
  • 21
  • 4
0

Running docker-compose down after your tests will destroy everything associated with your docker-compose.yml

Docker Compose is a container life cycle manager and by default it tries to keep everything across multiple runs. As Stas Makarov mentions, there is a VOLUME defined in the mysql image that persists the data outside of the container.

Matt
  • 68,711
  • 7
  • 155
  • 158
  • This "down" command does seem to destroy the container, but now each time I run `docker-compose up` it has to re-insert all the test data. – Anton S Mar 08 '17 at 08:01
  • @antonsteenvoorden You could add the initial data to an image rather than after startup so the next run contains some initial state. – Matt May 04 '17 at 07:13
  • I managed to fix the issue (with mysql image) by doing the following: 1. change the mounting point of the sql storage (This is what actually caused the problem) I used the solution suggested here: http://stackoverflow.com/questions/32482780/how-to-create-populated-mysql-docker-image-on-build-time but I did it by running a sed command: `RUN sed -i 's|/var/lib/mysql|/var/lib/mysql2|g' /etc/mysql/my.cnf` 2. add my scripts to a folder inside the container 3. run import.sh script that inserts data using the daemon 4. remove the sql scripts I just make multiple images with varying data – Anton S May 04 '17 at 08:16