Run timer exactly every second with systemd - devops

How to run some service exactly every second? Overlapping allowed.
Here is my timer unit for systemd:
[Unit]
Description=Send jobs every second
[Timer]
OnBootSec=1min
OnUnitActiveSec=1s
Unit=app.jobs-send.service
[Install]
WantedBy=multi-user.target
But app.jobs-send.service service doesn't executed every second:
May 07 17:57:38 app.deployer.org systemd[1]: Started Send jobs to queue.
May 07 17:57:48 app.deployer.org systemd[1]: Starting Send jobs to queue...
May 07 17:57:48 app.deployer.org systemd[1]: Started Send jobs to queue.
May 07 17:57:58 app.deployer.org systemd[1]: Starting Send jobs to queue...
May 07 17:57:58 app.deployer.org systemd[1]: Started Send jobs to queue.
May 07 17:58:08 app.deployer.org systemd[1]: Starting Send jobs to queue...
May 07 17:58:08 app.deployer.org systemd[1]: Started Send jobs to queue.
There is a gap about 17:57:58 - 17:57:48 = 10 seconds.
How to exec task every second?

As documented in man systemd.time, the accuracy of OnUnitActiveSec= is subject to the AccuracySec= setting, also documented there. Set to AccuracySec=1us for best resolution.

Related

Jupyterhub (TLJH): Failure setting up Development Environment: "Failed to start jupyterhub.service."

since last week, I am unable to set up a TLJH Development Environment as described here:
https://tljh.jupyter.org/en/latest/contributing/dev-setup.html.
Up until January 23 2023 the process went smoothly, but after rebuilding the image last Friday (27 January), after running bootstrap.py in the container, the server doesn't spawn. Browsing to my localhost:12000 I just see the message "404 page not found".
I am running TLJH in a container in an Ubuntu 22.04 host, running Docker version 20.10.21.
On the same host, when using the image I build before January 23 with the same method I can spawn the server ok. There were no new commits on the TLJH repo since, so the issue should not be there.
Here is the output of journalctl -u jupyterhub -f ran inside the container:
-- Logs begin at Mon 2023-01-30 12:44:34 UTC. --
Jan 30 12:44:41 87f857896c91 python3[64]: raise exc.ArgumentError(msg, code=code) from err
Jan 30 12:44:41 87f857896c91 python3[64]: sqlalchemy.exc.ArgumentError: Column expression, FROM clause, or other columns clause element expected, got [1]. Did you mean to say select(1)?
Jan 30 12:44:41 87f857896c91 python3[64]:
Jan 30 12:44:41 87f857896c91 systemd[1]: jupyterhub.service: Main process exited, code=exited, status=1/FAILURE
Jan 30 12:44:41 87f857896c91 systemd[1]: jupyterhub.service: Failed with result 'exit-code'.
Jan 30 12:44:41 87f857896c91 systemd[1]: jupyterhub.service: Scheduled restart job, restart counter is at 5.
Jan 30 12:44:41 87f857896c91 systemd[1]: Stopped jupyterhub.service.
Jan 30 12:44:41 87f857896c91 systemd[1]: jupyterhub.service: Start request repeated too quickly.
Jan 30 12:44:41 87f857896c91 systemd[1]: jupyterhub.service: Failed with result 'exit-code'.
Jan 30 12:44:41 87f857896c91 systemd[1]: Failed to start jupyterhub.service.
Since the logs show an error in sqlalchemy, I tried dowongrading sqlalchemy from 2.0.0 to 1.4.46, and then reloaded the hub, but that didn't solve the problem
Is anyone else experiancing the same issue? Any ideas for further troubleshooting?
Thanks!
EDIT: I applied the solution described here and it worked.
Commands to run:
pip3 install --upgrade 'SQLAlchemy<2.0.0'
tljh-config reload hub
I applied the solution described here and it worked.
The reason seems to be an issue introduced in SQLAlchemy 2.0.0, is resolved by downgrading SQLAlchemy.
Commands to run:
pip3 install --upgrade 'SQLAlchemy<2.0.0'
tljh-config reload hub

After restoring from server backup, Jenkins stopped working

in the logs the following information
The command journalctl -xe issues failrul
-- Support: http://www.ubuntu.com/support
--
-- Automatic restarting of the unit jenkins.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Dec 05 04:11:12 vwkdenicdn systemd[1]: Stopped Jenkins Continuous Integration Server.
-- Subject: A stop job for unit jenkins.service has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A stop job for unit jenkins.service has finished.
--
-- The job identifier is 10799 and the job result is done.
Dec 05 04:11:12 vwkdenicdn systemd[1]: jenkins.service: Start request repeated too quickly.
Dec 05 04:11:12 vwkdenicdn systemd[1]: jenkins.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit jenkins.service has entered the 'failed' state with result 'exit-code'.
Dec 05 04:11:12 vwkdenicdn systemd[1]: Failed to start Jenkins Continuous Integration Server.
-- Subject: A start job for unit jenkins.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit jenkins.service has finished with a failure.
--
-- The job identifier is 10799 and the job result is failed.
and
systemctl status jenkins.service
● jenkins.service - Jenkins Continuous Integration Server
Loaded: loaded (/lib/systemd/system/jenkins.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2022-12-05 04:11:12 UTC; 1min 41s ago
Process: 137379 ExecStart=/usr/bin/jenkins (code=exited, status=1/FAILURE)
Main PID: 137379 (code=exited, status=1/FAILURE)
Dec 05 04:11:12 vwkdenicdn systemd[1]: jenkins.service: Scheduled restart job, restart counter is at 5.
Dec 05 04:11:12 vwkdenicdn systemd[1]: Stopped Jenkins Continuous Integration Server.
Dec 05 04:11:12 vwkdenicdn systemd[1]: jenkins.service: Start request repeated too quickly.
Dec 05 04:11:12 vwkdenicdn systemd[1]: jenkins.service: Failed with result 'exit-code'.
Dec 05 04:11:12 vwkdenicdn systemd[1]: Failed to start Jenkins Continuous Integration Server.
Tried to update Jenkins as after the same error the first time it worked
and I also tried changing the Jenkins config and setting the version:
openjdk version "1.8.0_352"
OpenJDK Runtime Environment (build 1.8.0_352-8u352-ga-1~20.04-b08)
OpenJDK 64-Bit Server VM (build 25.352-b08, mixed mode)
the problem was that the new version of django does not support version 8 of java
this command caused a problem when starting jenkins:
journalctl -f --unit=jenkins
next, I'm just install java11:
sudo apt install openjdk-11-jdk
and change default java,
sudo update-alternatives --config java
after this command choice java version by number
There are 2 choices for the alternative java (providing /usr/bin/java).
Selection Path Priority Status
0 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 auto mode
1 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 manual mode
2 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1081 manual mode
Press to keep the current choice[*], or type selection number: 0
next, start jenkins
systemctl start jenkins.service

Docker flooding syslog with run-docker-runtime logs

I've google this, but so far no way to fix it. My syslog under /var/log is being flooded every second with messages like this;
Aug 27 20:58:27 mail-server systemd[1]: run-docker-runtime\x2drunc-moby-e4bfb13118b141bf232cf981fe9b535706243c47ae0659466b8e6667bd4feceb-runc.YHoxmJ.mount: Succeeded.
Aug 27 20:58:27 mail-server systemd[1083]: run-docker-runtime\x2drunc-moby-e4bfb13118b141bf232cf981fe9b535706243c47ae0659466b8e6667bd4feceb-runc.YHoxmJ.mount: Succeeded.
Aug 27 20:58:27 mail-server systemd[8395]: run-docker-runtime\x2drunc-moby-e4bfb13118b141bf232cf981fe9b535706243c47ae0659466b8e6667bd4feceb-runc.YHoxmJ.mount: Succeeded.
Aug 27 20:58:28 mail-server systemd[1]: run-docker-runtime\x2drunc-moby-5dc4f4e0b3cbd5e5bfbcc88b8d22f92575706b7c3603847ccb2fd4e56f188f99-runc.gt51Ek.mount: Succeeded.
Aug 27 20:58:28 mail-server systemd[1083]: run-docker-runtime\x2drunc-moby-5dc4f4e0b3cbd5e5bfbcc88b8d22f92575706b7c3603847ccb2fd4e56f188f99-runc.gt51Ek.mount: Succeeded.
Aug 27 20:58:28 mail-server systemd[8395]: run-docker-runtime\x2drunc-moby-5dc4f4e0b3cbd5e5bfbcc88b8d22f92575706b7c3603847ccb2fd4e56f188f99-runc.gt51Ek.mount: Succeeded.
I am running Ubuntu 20.04 and dockerd is run by systemd.
Could anyone help me to find the cause if this? It seems that every single container is generating this.
Best,
Francis
Those messages are from systemd itself about the mount. This is addressed in systemd v249; see https://github.com/systemd/systemd/issues/6432 for more information.
In a nutshell, that version of systemd allows controlling of that mount via its unit file using the following:
[Mount]
LogLevelMax=0
The LogLevelMax setting applies not just to the unit but also to systemd's log messages itself about the unit. That is the change introduced in v249.

Getting "neo4j.service: Failed with result 'start-limit-hit'." on installation of Neo4j Server 3.5.7

Trying to upgrade local Neo4j instances and getting errors on start.
Ubuntu 16.04: Trying to upgrade local instances of Neo4j databases which are currently at 3.3.1.
Installed Neo4j Desktop thinking I could do it with that and found the lowest version it upgrades from is 3.4. I now can't remove that. Ubuntu software fails (I just click remove, restart and Neo4j Desktop is still there). dpkg --list doesn't list neo4j desktop, so I can't use "apt-get remove" to remove it.
Uninstalled 3.3.1 and installed 3.3.9 (latest version of 3.3.x). Started Neo4j and ran fine, updating the database stores to 3.3.9.
Uninstalled 3.3.9 and installed 3.5.7.
Expecting Neo4j to start normally with "sudo service neo4j start", but now getting the following:
neo4j.service - Neo4j Graph Database
Loaded: loaded (/lib/systemd/system/neo4j.service; disabled; vendor preset: enabled)
Active: failed (Result: start-limit-hit) since Tue 2019-07-09 14:00:22 BST; 58s ago
Process: 1417 ExecStart=/usr/share/neo4j/bin/neo4j console (code=exited, status=1/FAILURE)
Main PID: 1417 (code=exited, status=1/FAILURE)
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Main process exited, code=exited, status=1/FAILURE
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Unit entered failed state.
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Failed with result 'exit-code'.
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Service hold-off time over, scheduling restart.
Jul 09 14:00:22 doug-ubuntu systemd[1]: Stopped Neo4j Graph Database.
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Start request repeated too quickly.
Jul 09 14:00:22 doug-ubuntu systemd[1]: Failed to start Neo4j Graph Database.
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Unit entered failed state.
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Failed with result 'start-limit-hit'.
Checked that I am running JVE 1.8:
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (build 1.8.0_212-8u212-b03-0ubuntu1.16.04.1-b03)
OpenJDK 64-Bit Server VM (build 25.212-b03, mixed mode)
Unsure how to proceed. Any ideas welcome please. Thank you!
Found the answer by running "journalctl -e -u neo4j" to view the full error log. Turns out that it was an APOC jar that was installed for 3.3.x but not compatible with 3.5.x. Removing the jar file solved it.

Strange behavior of neo4j-service on Debian 8.1

I have installed neo4j on Debian 8.1 thanks to these instructions : http://debian.neo4j.org/
Now if, as root, I start neo4j with neo4j-service like this
service neo4j-service start
Sometimes it will works correctly but most of the the time, the neo4j-service will timeout. But the interesting fact is that neo4j is indeed started, I can go the the browser and make some queries. But the neo4j-service tells me that it failed :
root#ns***:~# service neo4j-service start
Job for neo4j-service.service failed. See 'systemctl status neo4j-service.service' and 'journalctl -xn' for details.
root#ns***:~# systemctl status neo4j-service.service
● neo4j-service.service - LSB: Neo4j Graph Database server
Loaded: loaded (/etc/init.d/neo4j-service)
Active: failed (Result: timeout) since Fri 2015-10-16 19:03:08 CEST; 6min ago
Process: 24556 ExecStop=/etc/init.d/neo4j-service stop (code=exited, status=0/SUCCESS)
Process: 29730 ExecStart=/etc/init.d/neo4j-service start (code=killed, signal=TERM)
Oct 16 18:58:08 ns***.ip-91-***-***.eu neo4j-service[29730]: WARNING: Max 1024 open files allowed, minimum of 40 000 recommended. See the Neo4j manual.
Oct 16 18:58:08 ns***.ip-91-***-***.eu neo4j-service[29730]: Starting Neo4j Server...WARNING: not changing user
Oct 16 19:03:08 ns***.ip-91-***-***.eu systemd[1]: neo4j-service.service start operation timed out. Terminating.
Oct 16 19:03:08 ns***.ip-91-***-***.eu systemd[1]: Failed to start LSB: Neo4j Graph Database server.
Oct 16 19:03:08 ns***.ip-91-***-***.eu systemd[1]: Unit neo4j-service.service entered failed state.
And sometimes it will tell me that the service started correctly but will not manage to stop it.
Most of the time, I have to kill the process myself to "reset everything" correctly.
Do you know why this is happening ?
Are you aware of any issues with the neo4j-service on Debian 8.1 ?
This approach to running Neo4j is deprecated and you should use neo4j command.
Or you can write your own service wrapper and for that I suggest to you use http://supervisord.org/

Resources