Time in Docker container out of sync with host machine

Question

I'm trying to connect to CosmosDB through my SpringBoot app. I have all of this working if I run the app with Spring or via Intellij. But, when I run the app in Docker I get the following error message:

com.azure.data.cosmos.CosmosClientException: The authorization token is not valid at the current time.
Please create another token and retry
(token start time: Thu, 26 Mar 2020 04:32:10 GMT, 
token expiry time: Thu, 26 Mar 2020 04:47:10 GMT, current server time: Tue, 31 Mar 2020 20:12:42 GMT).

Note that in the above error message the current server time is correct but the other times are 5 days behind.

What I find interesting is that I only ever receive this in the docker container.

FROM {copy of zulu-jdk11}

ARG JAR_FILE

#.crt file in the same folder as your Dockerfile
ARG CERT="cosmos.cer"
ARG ALIAS="cosmos2"

#import cert into java
COPY $CERT /
RUN chmod +x /$CERT
WORKDIR $JAVA_HOME/lib/security
RUN keytool -importcert -file /$CERT -alias $ALIAS -cacerts -storepass changeit -noprompt

WORKDIR /
COPY /target/${JAR_FILE} app.jar
COPY run-java.sh /
RUN chmod +x /run-java.sh

ENV JAVA_OPTIONS "-Duser.timezone=UTC"
ENV JAVA_APP_JAR "/app.jar"

# run as non-root to mitigate some security risks
RUN addgroup -S pcc && adduser -S nonroot -G nonroot
USER nonroot:nonroot

ENTRYPOINT ["/run-java.sh"]

One thing to note is ENV JAVA_OPTIONS "-Duser.timezone=UTC" but removing this didn't help me at all

I basically run the same step from IntelliJ and I have no issues with it but in docker the expiry date seems to be 5 days behind.

version: "3.7"
services:
  orchestration-agent:
    image: {image-name}
    ports:
      - "8080:8080"
    network_mode: host
    environment:
      - COSMOSDB_URI=https://host.docker.internal:8081/
      - COSMOSDB_KEY={key}
      - COSMOSDB_DATABASE={database}
      - COSMOSDB_POPULATEQUERYMETRICS=true
      - COSMOSDB_ITEMLEVELTTL=60

I think it should also be mentioned that I changed the network_mode to host. And I also changed the CosmosDB URI from https://localhost:8081 to https://host.docker.internal:8081/

I would also like to mention that I built my dockerfile with the help of:

Importing self-signed cert into Docker's JRE cacert is not recognized by the service

How to add a SSL self-signed cert to Jenkins for LDAPS within Dockerfile?

It feels as though your container (image) is not correctly obtaining|updating the time. If it has a shell, can you `docker run --interactive --tty {image-name} /bin/sh -c 'date'`. If your client is dependent on the image for its time and the image time is incorrect, this could explain the problem. I assume the `current server time` is being return by the CosmosDB service. — DazWilkin, Mar 31 '20 at 21:27
@DazWilkin what am I supposed to be seeing as the ouput, I'm seeing that same errors as I usually do — Archmede, Mar 31 '20 at 21:35
Since I don't turn my work computer off, it sleeps over the weekend, and when the computer goes to sleep it makes docker go out of sync. Restarting my computer fixed this issue — Archmede, Mar 31 '20 at 21:53
Ah, apologies you probably need to adjust the entrypoint instead: `docker run --interactive --tty --entrypoint=/bin/sh {image-name} date` to get the date according to the container. Evidently, this datetime matches your host's which was wrong. — DazWilkin, Mar 31 '20 at 23:22
@DazWilkin I'm getting "/bin/sh: can't open 'date': No such file or directory" with your solution. I'm on windows — Archmede, Apr 01 '20 at 12:40
It appears there's no shell in the image. IIUC you've identified the underlying issue that your host's time was wrong and thus the client used an outdated time to construct the token. — DazWilkin, Apr 01 '20 at 15:10
@DazWilkin yes, I will be rewording the question and posting an answer later today. Thanks for your help — Archmede, Apr 01 '20 at 15:24
Related: https://stackoverflow.com/questions/24551592/how-to-make-sure-dockers-time-syncs-with-that-of-the-host — Vadzim, May 28 '21 at 07:47

BMitch · Accepted Answer · 2020-04-02T15:07:50.807

Docker containers don't maintain a separate clock, it's identical to the Linux host since time is not a namespaced value. This is also why Docker removes the permission to change the time inside the container, since that would impact the host and other containers, breaking the isolation model.

However, on Docker Desktop, docker runs inside of a VM (allowing you to run Linux containers on non-Linux desktops), and that VM's time can get out of sync when the laptop is suspended. This is currently being tracked in an issue over on github which you can follow to see the progress: https://github.com/docker/for-win/issues/4526

Potential solutions include restarting your computer, restarting docker's VM, running NTP as a privileged container, or resetting the time sync in the windows VM with the following PowerShell:

Get-VMIntegrationService -VMName DockerDesktopVM -Name "Time Synchronization" | Disable-VMIntegrationService
Get-VMIntegrationService -VMName DockerDesktopVM -Name "Time Synchronization" | Enable-VMIntegrationService

With WSL 2, restarting the VM involves:

wsl --shutdown
wsl

Vadzim · Answer 2 · 2021-05-28T07:04:02.220

5

There is recent known problem with WSL 2 time shift after sleep which has been fixed in 5.10.16.3 WSL 2 Linux kernel which is still not included in Windows 10 version 21H1 update but can be installed manually.

How to check WSL kernel version:

> wsl uname -r

Temporal workaround for the old kernel that helps until next sleep:

> wsl hwclock -s

edited May 28 '21 at 07:04

answered May 27 '21 at 20:54

Vadzim

24,954
11
143
151

Sean McCarthy · Answer 3 · 2021-02-11T18:17:17.027

Here's an alternative that worked for me on WSL2 with Docker Desktop on Windows:

Since it's not possible to set the date inside a Docker container, I just opened Ubuntu in WSL2 and ran the following command to synchronize the clock:

sudo date -s "$(wget -qSO- --max-redirect=0 google.com 2>&1 | grep Date: | cut -d' ' -f5-8)Z"

It worked well, so I added the following line in my root user's crontab:

# Edit root user's crontab
sudo crontab -e

# Add the following line to run it every minute of every day:
* * * * * sudo date -s "$(wget -qSO- --max-redirect=0 google.com 2>&1 | grep Date: | cut -d' ' -f5-8)Z"

After that, I just restarted my Docker containers, and the dates were correct since they seemed to use the WSL2 Ubuntu dates.

Date before (incorrect):

date
Thu Feb  4 21:50:35 UTC 2021

Date after (correct):

date
Fri Feb  5 19:01:05 UTC 2021

Time in Docker container out of sync with host machine

3 Answers3

Linked

Related