I have a jenkins configuration as follows:
Master (ubuntu)
~$ date
Tue Mar 7 08:35:06 UTC 2017
slave (redhat)
# date
Tue Mar 7 08:36:10 PST 2017
In jenkins system information the master shows (should show pacific):
user.timezone GMT
Even though I have placed these lines from jenkins wiki:
JENKINS_JAVA_OPTIONS="-Duser.timezone=America/Los_Angeles"
JAVA_ARGS="-Djava.awt.headless=true -Dorg.apache.commons.jelly.tags.fmt.timeZone=America/Los_Angeles"
And the strangest part is when I check the node configuaration in jenkins it shows my redhat slave node is 8 hours ahead.
My redhat slaves time ideally shouldn't be changed because it would screw with DB writes that I will be doing in testing. However I am completely stumped so any information would be helpful.
Where did you placed the JAVA env strings? Could you ensure that they are being used:
su jenkins
echo $JAVA_ARGS
echo $JENKINS_JAVA_OPTIONS
My problem was my UTC time was off. When I did timedatectl it showed NTP was working and UTC was in sync but it was lying. My labs firewall blocks the NTP port and I ended up finding this magic command to sync my clock:
sudo date -s "$(wget -S "http://www.google.com/" 2>&1 | grep -E '^[[:space:]]*[dD]ate:' | sed 's/^[[:space:]]*[dD]ate:[[:space:]]*//' | head -1l | awk '{print $1, $3, $2, $5 ,"GMT", $4 }' | sed 's/,//')"
It works well I just have to
Related
I'm trying to perform some user operation(change admin-user), after Neo4j container boots up. But my background script doesn't wait for the neo4j to come up and dies before Neo4j comes online.
entrypoint.sh is something like
if [some condition]
my_function &
fi
if [${cmd}" == "neo4j" ]; then
exec neo4j console
fi
helper_file.sh has my_function
function my_function {
echo "Checking to see if Neo4j has started at http://${DB_HOST}:${DB_PORT}..."
curl --retry-connrefused --retry 5 --retry-max-time 300 http://${DB_HOST}:${DB_PORT}
if [ $? -ne 0 ]; then
echo "Curl failed with error $?. Exiting.."
return 1
fi
migrate_users <--- another function
}
the problem that I'm facing is Neo4j doesn't bootup till curl is doing the retries.
Tue Sep 20 12:46:35 UTC 2022 Checking to see if Neo4j has started at http://localhost:7474...
Tue Sep 20 12:46:35 UTC 2022 % Total % Received % Xferd Average Speed Time Time Time Current
Tue Sep 20 12:46:35 UTC 2022 Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
Tue Sep 20 12:46:35 UTC 2022 curl: (7) Failed to connect to localhost port 7474: Connection refused
Tue Sep 20 12:46:35 UTC 2022 Curl failed with error 0. Exiting..
user: vmanage; command: neo4j
Directories in use:
How can I ensure that migrate_users function gets called after Neo4j has come up online completely?
Edit:
thank you for providing the suggestion.
If I go with the background process approach, I'm seeing that Neo4j doesn't boots up, till curl queries have finished
Tue Sep 20 18:57:34 UTC 2022 Checking to see if Neo4j has started
at http://localhost:7474...
Tue Sep 20 18:57:34 UTC 2022 Neo4j not ready
Tue Sep 20 18:57:34 UTC 2022 Connection refused
Tue Sep 20 18:57:34 UTC 2022 config-db is not up, try to setup password again
user: vmanage; command: neo4j
Directories in use:
home: /var/lib/neo4j
config: /var/lib/neo4j/conf
logs: /log
plugins: /var/lib/neo4j/plugins
import: /var/lib/neo4j
data: /data
certificates: /var/lib/neo4j/certificates
licenses: /var/lib/neo4j/licenses
run: /var/lib/neo4j/run
Starting Neo4j.
Going to try this : https://github.com/neo4j/docker-neo4j/issues/166#issuecomment-486890785
You can add a loop inside your script to check the health of neo4j container. If the health check get pass only proceeed further in you script otherwise loop untill it pass.
You can use docker-compose with the depends_on + condition to do that.
Even docker-compose documentation recommends to implement some kind of script to wait until the service is up. Take a look to the following links docker-compose and stackoverflow
But it could be something like:
version: "2"
services:
neo4j-admin:
build: .
depends_on:
- "neo4j"
command: ["./wait-for-it.sh","--", "sh", "change_admin_passwd.sh"]
neo4j:
image: neo4j
Your function named my_function could use until to keep waiting for neo4j to start, for example:
function my_function {
let RETRIES=0
declare SUCCESS=0
until [[ $SUCCESS -eq 1 ]] || [[ $RETRIES -eq 50 ]]; do
echo "Checking to see if Neo4j has started at
http://${DB_HOST}:${DB_PORT}..."
STATUS_CODE=$(curl -w %{http_code} -o /dev/null -s http://${DB_HOST}:${DB_PORT})
if [[ $STATUS_CODE -eq 200 ]]; then
echo "Neo4j is up and running" && SUCCESS=1 && exit 0
else
echo "Neo4j not ready" && let RETRIES+=1 && sleep 10
fi
done
migrate_users
}
Using macos Catalina and docker desktop.
The time of the conteiners perfectly syncs with the time in Vm Docker Desktop.
But I need to test one conteiner with date in the future.
I dont want to advance the clock of my mac because of iCloud services.
So I can achieve this just changing the hour in VM docker-desktop
I run:
docker run --privileged --rm alpine date -s "2023-02-19 11:27"
It changes the time ok. But it last just some seconds. Clearly there is some type of "syncronizer" that keeps changing back the time.
How do I disable this "syncronizer"?
There's only one time in Linux, it's not namespaced, so when Docker runs ntp on the VM to keep it synchronized (in the past it would get out of sync, especially after the parent laptop was put to sleep), that sync applies to the Linux kernel, which applies to every container since it's the same kernel value for everything. Therefore it's impossible to set this on just one container in the Linux kernel.
Instead, I'd recommend going with something like libfaketime that can be used to alter the response applications see when the query that time value. It basically sits as a layer between the kernel and application, and injects an offset based on an environment variable you set.
FROM debian
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update \
&& apt-get install -y libfaketime \
&& rm -rf /var/lib/apt/lists*
ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/faketime/libfaketime.so.1
And then to run it, set FAKETIME:
$ docker run --rm test-faketime date
Thu Feb 17 14:59:48 UTC 2022
$ docker run -e FAKETIME="+7d" --rm test-faketime date
Thu Feb 24 14:59:55 UTC 2022
$ date
Thu 17 Feb 2022 09:59:57 AM EST
I found that you can kill the NTP service which syncs the VM time to the host's time. Details of how service works.
First, use this guide to get a shell inside the VM.
Then, find the sntpc service:
/ # ps a | grep sntpc
1356 root 0:00 /usr/bin/containerd-shim-runc-v2 -namespace services.linuxkit -id sntpc -address /run/containerd/containerd.sock
1425 root 0:00 /usr/sbin/sntpc -v -i 30 127.0.0.1
3465 root 0:00 grep sntpc
Take the number at the beginning of the /usr/sbin/sntpc line, and use kill to stop the process.
/ # kill 1425
I have found that Docker Desktop does not seem to restart this process if it dies, and you can change the VM time without SNTPC changing it back.
I'm trying to connect to CosmosDB through my SpringBoot app. I have all of this working if I run the app with Spring or via Intellij. But, when I run the app in Docker I get the following error message:
com.azure.data.cosmos.CosmosClientException: The authorization token is not valid at the current time.
Please create another token and retry
(token start time: Thu, 26 Mar 2020 04:32:10 GMT,
token expiry time: Thu, 26 Mar 2020 04:47:10 GMT, current server time: Tue, 31 Mar 2020 20:12:42 GMT).
Note that in the above error message the current server time is correct but the other times are 5 days behind.
What I find interesting is that I only ever receive this in the docker container.
FROM {copy of zulu-jdk11}
ARG JAR_FILE
#.crt file in the same folder as your Dockerfile
ARG CERT="cosmos.cer"
ARG ALIAS="cosmos2"
#import cert into java
COPY $CERT /
RUN chmod +x /$CERT
WORKDIR $JAVA_HOME/lib/security
RUN keytool -importcert -file /$CERT -alias $ALIAS -cacerts -storepass changeit -noprompt
WORKDIR /
COPY /target/${JAR_FILE} app.jar
COPY run-java.sh /
RUN chmod +x /run-java.sh
ENV JAVA_OPTIONS "-Duser.timezone=UTC"
ENV JAVA_APP_JAR "/app.jar"
# run as non-root to mitigate some security risks
RUN addgroup -S pcc && adduser -S nonroot -G nonroot
USER nonroot:nonroot
ENTRYPOINT ["/run-java.sh"]
One thing to note is ENV JAVA_OPTIONS "-Duser.timezone=UTC" but removing this didn't help me at all
I basically run the same step from IntelliJ and I have no issues with it but in docker the expiry date seems to be 5 days behind.
version: "3.7"
services:
orchestration-agent:
image: {image-name}
ports:
- "8080:8080"
network_mode: host
environment:
- COSMOSDB_URI=https://host.docker.internal:8081/
- COSMOSDB_KEY={key}
- COSMOSDB_DATABASE={database}
- COSMOSDB_POPULATEQUERYMETRICS=true
- COSMOSDB_ITEMLEVELTTL=60
I think it should also be mentioned that I changed the network_mode to host. And I also changed the CosmosDB URI from https://localhost:8081 to https://host.docker.internal:8081/
I would also like to mention that I built my dockerfile with the help of:
Importing self-signed cert into Docker's JRE cacert is not recognized by the service
How to add a SSL self-signed cert to Jenkins for LDAPS within Dockerfile?
Docker containers don't maintain a separate clock, it's identical to the Linux host since time is not a namespaced value. This is also why Docker removes the permission to change the time inside the container, since that would impact the host and other containers, breaking the isolation model.
However, on Docker Desktop, docker runs inside of a VM (allowing you to run Linux containers on non-Linux desktops), and that VM's time can get out of sync when the laptop is suspended. This is currently being tracked in an issue over on github which you can follow to see the progress: https://github.com/docker/for-win/issues/4526
Potential solutions include restarting your computer, restarting docker's VM, running NTP as a privileged container, or resetting the time sync in the windows VM with the following PowerShell:
Get-VMIntegrationService -VMName DockerDesktopVM -Name "Time Synchronization" | Disable-VMIntegrationService
Get-VMIntegrationService -VMName DockerDesktopVM -Name "Time Synchronization" | Enable-VMIntegrationService
With WSL 2, restarting the VM involves:
wsl --shutdown
wsl
There is recent known problem with WSL 2 time shift after sleep which has been fixed in 5.10.16.3 WSL 2 Linux kernel which is still not included in Windows 10 version 21H1 update but can be installed manually.
How to check WSL kernel version:
> wsl uname -r
Temporal workaround for the old kernel that helps until next sleep:
> wsl hwclock -s
Here's an alternative that worked for me on WSL2 with Docker Desktop on Windows:
Since it's not possible to set the date inside a Docker container, I just opened Ubuntu in WSL2 and ran the following command to synchronize the clock:
sudo date -s "$(wget -qSO- --max-redirect=0 google.com 2>&1 | grep Date: | cut -d' ' -f5-8)Z"
It worked well, so I added the following line in my root user's crontab:
# Edit root user's crontab
sudo crontab -e
# Add the following line to run it every minute of every day:
* * * * * sudo date -s "$(wget -qSO- --max-redirect=0 google.com 2>&1 | grep Date: | cut -d' ' -f5-8)Z"
After that, I just restarted my Docker containers, and the dates were correct since they seemed to use the WSL2 Ubuntu dates.
Date before (incorrect):
date
Thu Feb 4 21:50:35 UTC 2021
Date after (correct):
date
Fri Feb 5 19:01:05 UTC 2021
I have 3 nodes that I want to play with managers and workers. My first one (Debian) I made it into a swarm manager:
root#debiancli:~# docker swarm init --advertise-addr 192.168.182.129
Swarm initialized: current node (mkg6ecl3x28uyyqx7gvzz0ja3) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join --token SWMTKN-1-47h52q7mpdkhbi4dsqyjt7pnjqgvm4oxxfh87k6e2hoj8f4op0-2p1zkg309owyophvk95bw7rj0 192.168.182.129:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
On my second soon-to-be node (CentOS), I tried to join it to the cluster:
[root#centostraining ~]# docker swarm join --token SWMTKN-1-47h52q7mpdkhbi4dsqyjt7pnjqgvm4oxxfh87k6e2hoj8f4op0-2p1zkg309owyophvk95bw7rj0 192.168.182.129:2377
Error response from daemon: error while validating Root CA Certificate: x509: certificate has expired or is not yet valid
but it said, as shown above, that the certificate is not valid (date issue). I checked the date on my Debian and it is fine
root#debiancli:~# date
Tue Aug 14 22:02:29 EDT 2018
I also checked the date in my CentOS:
[root#centostraining ~]# date
Ter Ago 14 22:05:05 -03 2018
Now, I checked my swarm manager CA cert date:
root#debiancli:~# docker swarm ca | openssl x509 -noout -text | grep -E "Before|After"
Not Before: Aug 15 01:58:00 2018 GMT
Not After : Aug 10 01:58:00 2038 GMT
So, weirdly enough, my certificate was generated to start the day after it was generated?
Then on my future node (CentOS), if I change the date:
[root#centostraining ~]# date +%Y%m%d -s "20180816"
20180816
[root#centostraining ~]# date
Qui Ago 16 00:00:01 -03 2018
[root#centostraining ~]# docker swarm join --token SWMTKN-1-47h52q7mpdkhbi4dsqyjt7pnjqgvm4oxxfh87k6e2hoj8f4op0-2p1zkg309owyophvk95bw7rj0 192.168.182.129:2377
This node joined a swarm as a worker.
Voilá, it now works as expected. Can anyone explain why my swarm ca cert is "in the future"?
Docker's certificate is not in the future. Instead, your timezones are setup incorrectly. On one, you are showing EDT, or -4. And on the other node, it's running an hour behind since it shows the same local time but in a -3 timezone. The output from the certificate is GMT or +0.
You should be using a tool like NTP to keep your clocks in sync, which would then show the -3 machine running an hour ahead of the other in local time, but the same time if you compare the same timezone.
I want to run a lot of docker containers in foreground mode like docker run johndoe/example doSomething.
I don't want to waste disk space on exited containers, so I have to remove them. But it takes about 9 seconds to remove an exited container, so I should remove the container after the command execution.
That's why I have two concurrent processes: one runs containers, the other removes them.
The problem is that docker rm appears to be blocking - it doesn't run while removing.
Here is a minimal working example. Following command runs a docker container every second, then print current date to the console:
while true; do docker run ubuntu ls > /dev/null; sleep 1; date; done
And the output is like:
Tue Sep 30 14:25:18 MSK 2014
Tue Sep 30 14:25:20 MSK 2014
Tue Sep 30 14:25:22 MSK 2014
Tue Sep 30 14:25:24 MSK 2014
But when I run docker rm some_id in a separate console, then I see that the time span increases like:
Tue Sep 30 14:26:53 MSK 2014
Tue Sep 30 14:26:55 MSK 2014
Tue Sep 30 14:27:03 MSK 2014
Tue Sep 30 14:27:10 MSK 2014
Am I getting something wrong? Why is it so? How can I deal with it?
Have you tried removing multiple containers in one rm command? This may reduce the number of hiccups. Perhaps you could do that once a day at a time where a hiccup is not important to you?
Also, have you measured how much disk space gets consumed, it may not matter for a long time thanks to the nature of the overlay filesystems. Of course this depends on what your doSomething actually does...
As in the docker-user mailing list - can you please post the output from
docker version ; docker info
as this may well be storage driver and setup specific
(though perhaps you just have so many containers or images that your system is going to take time to process them.