Output of pv on docker-compose startup not working as expected

Question

I am trying to output the progress of an import of an .sql file inside a mariadb docker container.

I have the following file/directory setup:

│-  docker-compose.yml
│-  Dockerfile
│-  import.sh
└── sql
    -  test.sql (rather big: ~ 1GB)

My docker-compose.yml is as simple as...

services:
  db:
    build: ./
    environment:
      MYSQL_ROOT_PASSWORD: root
    volumes:
      - ./:/docker-entrypoint-initdb.d

...with the following Dockerfile to install pv (pipe viewer). pv should give me a progress bar how far the import is currently...

FROM mariadb
RUN apt-get update && apt-get install -y pv

The import.sh will be executed through the mapped volume in /docker-entrypoint-initdb.d as described here.

#!/bin/bash
# create db
mysql -uroot -proot <<-EOF
  CREATE DATABASE test;
EOF
# import sql file and output progress with pv
echo "importing test.sql..."
pv --force "/docker-entrypoint-initdb.d/sql/test.sql" | mysql -uroot -proot "test"

---

Now, if I run docker-compose up it only outputs the 100% pv output at the end of the import:

importing test.sql...
953MiB 0:01:24 [11.2MiB/s] [================================>] 100%    0:05:42

If I execute the same command inside the container it works and it gives me a moving progress bar:

pv --force "/docker-entrypoint-initdb.d/sql/test.sql" | mysql -uroot -proot "test"
60.4MiB 0:00:14 [5.79MiB/s] [=>                              ]  6%     0:04:53

How can I get this progress bar on docker-compose up instead of the loong wait and the 100% output?

Answer 1

Background

First let's understand how pv is able to render a moving progress bar on a text-only output to the terminal: pv actually just prints plain text to its stdout with each progress update:

"[==>           ] 25%\r"
"[======>       ] 50%\r"
"[=========>    ] 76%\r"
"[============>] 100%\n"

Each line here represents a single progress update for which pv outputs the text within the quotes (so no quotes).

But this will not print in multiple lines to the terminal: \r is a carriage return character which will move the cursor back to the beginning of the line without starting a new line. So the next progress output will override the previous text resulting in the progress bar animation.

Only after the last update pv will print the new line character \n resulting in a final line break after the output.

Now to the problem with docker-compose: starting an app with docker-compose up will start all services, attach to their output and log it to its own output - prefixed with the respective service name:

app_1  | starting App...
db_1   | initializing database
....

To do this docker-compose will read each output line from each container and prefix it with the service name before printing it.

But as we saw before pv actually only prints a single line! This is why docker-compose will buffer the output until the end before finally printing it!

---

Solutions

I see two possible solutions here:

1. use docker-compose run db to initialize the database: this will run the container with its output directly attached to the console and print the output without any buffering or post-processing.

In this case you then can even omit the --force flag.

2. replace \r with \n to force every progress update to be printed on a new line, e.g. using tr. Additionally to be sure to disable any output buffering you may use stdbuf with it (see turn off buffering in pipe):

(pv --force -p "/docker-entrypoint-initdb.d/sql/test.sql" | mysql -uroot -proot "test") 2>&1 | stdbuf -o0 tr '\r' '\n'

will log

db_1   | [==>           ] 25%
db_1   | [======>       ] 50%
db_1   | [=========>    ] 76%
db_1   | [============>] 100%

---

Demo

Here is a small demo of the above:

# Dockerfile
FROM alpine
RUN apk add pv

# docker-compose.yml
services:
  app:
    build: .
    command: sh -c "pv --force -p -Ss 1024 -L 100 /dev/urandom 2>&1 > /dev/null | tr '\r' '\n'"

Addendum

As per comments the above demo does not work with an ubuntu based image. It seems in such images tr will buffer its output and only print everything once it exits.

The output buffer can however be disabled using stdbuf (see also turn off buffering in pipe):

# Dockerfile
FROM ubuntu
RUN apt-get update && apt-get install -y pv

# docker-compose.yml
services:
  app:
    build: .
    command: sh -c "pv --force -p -Ss 1024 -L 100 /dev/urandom 2>&1 > /dev/null | stdbuf -o0 tr '\r' '\n'"

Answer 2

I found a somewhat okayish solution:

Adding the --numeric flag to my pv command.

From the man page of pv:

-n, --numeric

Numeric output. Instead of giving a visual indication of progress, pv will give an integer percentage, one per line, on standard error, suitable for piping (via convoluted redirection) into dialog(1). Note that -f is not required if -n is being used.

So the command in my import.sh would be:

pv --numeric "/docker-entrypoint-initdb.d/sql/test.sql" | mysql -uroot -proot "test"

This gives me the following output:

importing test.sql...
36
53
80
100

Since this is not as nice as the typical output of pv this answer is not perfect...

---

I also tried a little nicer output without the newlines like:

importing test.sql...
36% 53% 80% 100%

with appending an awk command:

(pv --numeric "/docker-entrypoint-initdb.d/sql/test.sql" | mysql -uroot -proot "test") 2>&1 | awk '{printf "%s% ",$0}'

But again, same problem: this did only work inside the container and not on the output of docker-compose up.