Systemd and Puma in an infinite startup loop - ruby-on-rails

I'm banging my head against the wall with puma and systemd. I used foreman to set up my systemd files, but can't get puma out of its restart loop. Ubuntu 16.
Jun 19 02:48:12 ip-172-31-28-225 systemd[1]: Stopped rajlogviewer-web.service.
Jun 19 02:48:12 ip-172-31-28-225 systemd[1]: Started rajlogviewer-web.service.
Jun 19 02:48:12 ip-172-31-28-225 rajlogviewer-web.service[8954]: APP_DIR = /home/ubuntu/rajlogviewer, SHARED_DIR /home/ubuntu/rajlogviewer/shared
Jun 19 02:48:12 ip-172-31-28-225 rajlogviewer-web.service[8954]: [8954] Puma starting in cluster mode...
Jun 19 02:48:12 ip-172-31-28-225 rajlogviewer-web.service[8954]: [8954] * Version 3.6.0 (ruby 2.3.3-p222), codename: Sleepy Sunday Serenity
Jun 19 02:48:12 ip-172-31-28-225 rajlogviewer-web.service[8954]: [8954] * Min threads: 1, max threads: 6
Jun 19 02:48:12 ip-172-31-28-225 rajlogviewer-web.service[8954]: [8954] * Environment: production
Jun 19 02:48:12 ip-172-31-28-225 rajlogviewer-web.service[8954]: [8954] * Process workers: 2
Jun 19 02:48:12 ip-172-31-28-225 rajlogviewer-web.service[8954]: [8954] * Preloading application
Jun 19 02:48:14 ip-172-31-28-225 rajlogviewer-web.service[8954]: [8954] * Listening on tcp://0.0.0.0:3000
Jun 19 02:48:14 ip-172-31-28-225 rajlogviewer-web.service[8954]: [8954] * Listening on unix:///home/ubuntu/rajlogviewer/shared/tmp/sockets/puma.sock
Jun 19 02:48:14 ip-172-31-28-225 rajlogviewer-web.service[8954]: [8954] * Daemonizing...
Jun 19 02:48:24 ip-172-31-28-225 systemd[1]: rajlogviewer-web.service: Service hold-off time over, scheduling restart.
Jun 19 02:48:24 ip-172-31-28-225 systemd[1]: Stopped rajlogviewer-web.service.
Jun 19 02:48:24 ip-172-31-28-225 systemd[1]: Started rajlogviewer-web.service.
It just keeps restarting indefinitely. Here is my systemd init file
/etc/systemd/system/rajlogviewer-web.service
[Service]
User=ubuntu
WorkingDirectory=/home/ubuntu/rajlogviewer/current
#Environment=PATH=/home/ubuntu/.rvm/gems/ruby-2.3.3#rajlogsViewer/bin:$PATH
WorkingDirectory=/home/ubuntu/rajlogviewer/current/
ExecStart=/bin/bash -lc 'PATH=/home/ubuntu/.rvm/gems/ruby-2.3.3#rajlogsViewer/bin:$PATH exec /home/ubuntu/.rvm/bin/rvm ruby-2.3.3 do bundle exec puma -C /home/ubuntu/rajlogviewer
/shared/config/puma.rb --daemon'
Restart=no
RestartSec=10
StandardInput=null
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=%n
KillMode=mixed
TimeoutStopSec=5
The app usually boots in 5 seconds when booting with 'cap production puma:start' which works, so RestartSec=10 ought to work. Making it 20 seconds makes no difference.
Nothing interesting in the puma.stderr.log and puma.stdout.log .
Any ideas?

You should remove --daemon option from the systemd, there are several types of unit:
Configures the unit process startup type that affects the
functionality of ExecStart and related options. One of:
simple – The
default value. The process started with ExecStart is the main process
of the service.
forking – The process started with ExecStart spawns a
child process that becomes the main process of the service. The parent
process exits when the startup is complete.
oneshot – This type is
similar to simple, but the process exits before starting consequent
units. dbus – This type is similar to simple, but consequent units are
started only after the main process gains a D-Bus name.
notify – This
type is similar to simple, but consequent units are started only after
a notification message is sent via the sd_notify() function.
idle –
similar to simple, the actual execution of the service binary is
delayed until all jobs are finished, which avoids mixing the status
output with shell output of services.
The default value is simple, for the sake of puma configuration you use --daemon options which contradict with the systemd configuration.

Related

Jupyterhub (TLJH): Failure setting up Development Environment: "Failed to start jupyterhub.service."

since last week, I am unable to set up a TLJH Development Environment as described here:
https://tljh.jupyter.org/en/latest/contributing/dev-setup.html.
Up until January 23 2023 the process went smoothly, but after rebuilding the image last Friday (27 January), after running bootstrap.py in the container, the server doesn't spawn. Browsing to my localhost:12000 I just see the message "404 page not found".
I am running TLJH in a container in an Ubuntu 22.04 host, running Docker version 20.10.21.
On the same host, when using the image I build before January 23 with the same method I can spawn the server ok. There were no new commits on the TLJH repo since, so the issue should not be there.
Here is the output of journalctl -u jupyterhub -f ran inside the container:
-- Logs begin at Mon 2023-01-30 12:44:34 UTC. --
Jan 30 12:44:41 87f857896c91 python3[64]: raise exc.ArgumentError(msg, code=code) from err
Jan 30 12:44:41 87f857896c91 python3[64]: sqlalchemy.exc.ArgumentError: Column expression, FROM clause, or other columns clause element expected, got [1]. Did you mean to say select(1)?
Jan 30 12:44:41 87f857896c91 python3[64]:
Jan 30 12:44:41 87f857896c91 systemd[1]: jupyterhub.service: Main process exited, code=exited, status=1/FAILURE
Jan 30 12:44:41 87f857896c91 systemd[1]: jupyterhub.service: Failed with result 'exit-code'.
Jan 30 12:44:41 87f857896c91 systemd[1]: jupyterhub.service: Scheduled restart job, restart counter is at 5.
Jan 30 12:44:41 87f857896c91 systemd[1]: Stopped jupyterhub.service.
Jan 30 12:44:41 87f857896c91 systemd[1]: jupyterhub.service: Start request repeated too quickly.
Jan 30 12:44:41 87f857896c91 systemd[1]: jupyterhub.service: Failed with result 'exit-code'.
Jan 30 12:44:41 87f857896c91 systemd[1]: Failed to start jupyterhub.service.
Since the logs show an error in sqlalchemy, I tried dowongrading sqlalchemy from 2.0.0 to 1.4.46, and then reloaded the hub, but that didn't solve the problem
Is anyone else experiancing the same issue? Any ideas for further troubleshooting?
Thanks!
EDIT: I applied the solution described here and it worked.
Commands to run:
pip3 install --upgrade 'SQLAlchemy<2.0.0'
tljh-config reload hub
I applied the solution described here and it worked.
The reason seems to be an issue introduced in SQLAlchemy 2.0.0, is resolved by downgrading SQLAlchemy.
Commands to run:
pip3 install --upgrade 'SQLAlchemy<2.0.0'
tljh-config reload hub

redis json (redislabs/rejson:latest) on Heroku

I cannot figure this out. Hope I can get some help
I have a hobby tier in Heroku running django. To this I would like to attach a Redis service. However, I would like to use the recent rejson (redislabs/rejson:latest) docker image (instead of redistogo or heroku-redis)because it has json support. This works great on my local env. I was able to push the docker image into the container registry and actually start the redis server
2021-07-23T00:14:48.576294+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.576 # Current maximum open files is 10000. maxclients has been reduced to 9968 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
2021-07-23T00:14:48.577054+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 * Running mode=standalone, port=6379.
2021-07-23T00:14:48.577124+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 # Server initialized
2021-07-23T00:14:48.577184+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2021-07-23T00:14:48.577671+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 # <ReJSON> JSON data type for Redis v1.0.7 [encver 0]
2021-07-23T00:14:48.577789+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 * Module 'ReJSON' loaded from /usr/lib/redis/modules/rejson.so
2021-07-23T00:14:48.578232+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.578 * Ready to accept connections
Unfortunately, django is unable to connect to it
ConnectionError: Error -2 connecting to redis://localhost:6379. Name or service not known
There are no ENV variable exposed, so am unable to set any (I mean I can set them, but doubt if they will be relevant)
I did experiment with installing redistogo add on and am able to connect to it (this required setting the connection based on the REDIS_URL env variable that gets exposed when redistogo is added)
At my wits end...any help appreciated...I guess the question really boils do to:

Getting "neo4j.service: Failed with result 'start-limit-hit'." on installation of Neo4j Server 3.5.7

Trying to upgrade local Neo4j instances and getting errors on start.
Ubuntu 16.04: Trying to upgrade local instances of Neo4j databases which are currently at 3.3.1.
Installed Neo4j Desktop thinking I could do it with that and found the lowest version it upgrades from is 3.4. I now can't remove that. Ubuntu software fails (I just click remove, restart and Neo4j Desktop is still there). dpkg --list doesn't list neo4j desktop, so I can't use "apt-get remove" to remove it.
Uninstalled 3.3.1 and installed 3.3.9 (latest version of 3.3.x). Started Neo4j and ran fine, updating the database stores to 3.3.9.
Uninstalled 3.3.9 and installed 3.5.7.
Expecting Neo4j to start normally with "sudo service neo4j start", but now getting the following:
neo4j.service - Neo4j Graph Database
Loaded: loaded (/lib/systemd/system/neo4j.service; disabled; vendor preset: enabled)
Active: failed (Result: start-limit-hit) since Tue 2019-07-09 14:00:22 BST; 58s ago
Process: 1417 ExecStart=/usr/share/neo4j/bin/neo4j console (code=exited, status=1/FAILURE)
Main PID: 1417 (code=exited, status=1/FAILURE)
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Main process exited, code=exited, status=1/FAILURE
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Unit entered failed state.
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Failed with result 'exit-code'.
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Service hold-off time over, scheduling restart.
Jul 09 14:00:22 doug-ubuntu systemd[1]: Stopped Neo4j Graph Database.
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Start request repeated too quickly.
Jul 09 14:00:22 doug-ubuntu systemd[1]: Failed to start Neo4j Graph Database.
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Unit entered failed state.
Jul 09 14:00:22 doug-ubuntu systemd[1]: neo4j.service: Failed with result 'start-limit-hit'.
Checked that I am running JVE 1.8:
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (build 1.8.0_212-8u212-b03-0ubuntu1.16.04.1-b03)
OpenJDK 64-Bit Server VM (build 25.212-b03, mixed mode)
Unsure how to proceed. Any ideas welcome please. Thank you!
Found the answer by running "journalctl -e -u neo4j" to view the full error log. Turns out that it was an APOC jar that was installed for 3.3.x but not compatible with 3.5.x. Removing the jar file solved it.

Gitlab always exited automatically

I am running the gitlab with docker, but it always exits after a period of time
==> /var/log/gitlab/redis/current <==
2019-06-21_18:00:08.72435 459:signal-handler (1561140008) Received SIGTERM scheduling shutdown...
2019-06-21_18:00:08.81864 459:M 21 Jun 18:00:08.817 # User requested shutdown...
2019-06-21_18:00:08.81866 459:M 21 Jun 18:00:08.817 * Saving the final RDB snapshot before exiting.
2019-06-21_18:00:08.83736 459:M 21 Jun 18:00:08.837 * DB saved on disk
2019-06-21_18:00:08.83741 459:M 21 Jun 18:00:08.837 * Removing the pid file.
2019-06-21_18:00:08.83817 459:M 21 Jun 18:00:08.838 * Removing the unix socket file.
2019-06-21_18:00:08.83935 459:M 21 Jun 18:00:08.839 # Redis is now ready to exit, bye bye...
ok: down: redis-exporter: 0s, normally up
==> /var/log/gitlab/gitlab-rails/sidekiq.log <==
2019-06-21_18:00:09.57615 2019-06-21T18:00:09.576Z 807 TID-oviw2sgmf INFO: Shutting down
2019-06-21_18:00:09.57625 2019-06-21T18:00:09.576Z 807 TID-ovivo05i7 INFO: Scheduler exiting...
2019-06-21_18:00:09.57655 2019-06-21T18:00:09.576Z 807 TID-oviw2sgmf INFO: Terminating quiet workers
This was reported in gitlab-org/omnibus-gitlab issue 4137: "runsv send SIGTERM to redis in docker version"
runsv sends SIGTERM to redis every 60 secs
gitlab-org/omnibus-gitlab issue 1611 suggests a docker restart first.
But the general issue is not conclusively resolved yet.

Strange behavior of neo4j-service on Debian 8.1

I have installed neo4j on Debian 8.1 thanks to these instructions : http://debian.neo4j.org/
Now if, as root, I start neo4j with neo4j-service like this
service neo4j-service start
Sometimes it will works correctly but most of the the time, the neo4j-service will timeout. But the interesting fact is that neo4j is indeed started, I can go the the browser and make some queries. But the neo4j-service tells me that it failed :
root#ns***:~# service neo4j-service start
Job for neo4j-service.service failed. See 'systemctl status neo4j-service.service' and 'journalctl -xn' for details.
root#ns***:~# systemctl status neo4j-service.service
● neo4j-service.service - LSB: Neo4j Graph Database server
Loaded: loaded (/etc/init.d/neo4j-service)
Active: failed (Result: timeout) since Fri 2015-10-16 19:03:08 CEST; 6min ago
Process: 24556 ExecStop=/etc/init.d/neo4j-service stop (code=exited, status=0/SUCCESS)
Process: 29730 ExecStart=/etc/init.d/neo4j-service start (code=killed, signal=TERM)
Oct 16 18:58:08 ns***.ip-91-***-***.eu neo4j-service[29730]: WARNING: Max 1024 open files allowed, minimum of 40 000 recommended. See the Neo4j manual.
Oct 16 18:58:08 ns***.ip-91-***-***.eu neo4j-service[29730]: Starting Neo4j Server...WARNING: not changing user
Oct 16 19:03:08 ns***.ip-91-***-***.eu systemd[1]: neo4j-service.service start operation timed out. Terminating.
Oct 16 19:03:08 ns***.ip-91-***-***.eu systemd[1]: Failed to start LSB: Neo4j Graph Database server.
Oct 16 19:03:08 ns***.ip-91-***-***.eu systemd[1]: Unit neo4j-service.service entered failed state.
And sometimes it will tell me that the service started correctly but will not manage to stop it.
Most of the time, I have to kill the process myself to "reset everything" correctly.
Do you know why this is happening ?
Are you aware of any issues with the neo4j-service on Debian 8.1 ?
This approach to running Neo4j is deprecated and you should use neo4j command.
Or you can write your own service wrapper and for that I suggest to you use http://supervisord.org/

Resources