systemd init script for resque worker - ruby-on-rails

i habe an nginx webserver running on an Ubuntu 16.04 Server.
Now i am trying to build an init script for resque worker and scheduler for a Rails app.
I created a file resque-worker.service in "/etc/systemd/system/" and it looks like this:
[Unit]
Description=resque-worker for pageflow
[Service]
Type=forking
ExecStart=/home/pageflow/pageflow_daad/rake resque:scheduler QUEUE=* RAILS_ENV=production > /home/pageflow/pageflow_daad/log/resqueschedule.log &
[Install]
WantedBy=multi-user.target
For some reason after executing "systemctl daemon-reload" and "systemctl start name.service" i get this error:
$ systemctl status resque-worker.service
● resque-worker.service - resque-worker for pageflow
Loaded: loaded (/etc/systemd/system/resque-worker.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2017-05-17 15:52:38 CEST; 15s ago
Process: 28096 ExecStart=/home/pageflow/pageflow_daad/rake resque:scheduler QUEUE=* RAILS_ENV=production > /home/pageflow/pageflow_daad/log/resqueschedule.log & (code=exited, status=203/EXEC)
May 17 15:52:38 ostheim systemd[1]: Starting resque-worker for pageflow...
May 17 15:52:38 ostheim systemd[1]: resque-worker.service: Control process exited, code=exited status=203
May 17 15:52:38 ostheim systemd[1]: Failed to start resque-worker for pageflow.
May 17 15:52:38 ostheim systemd[1]: resque-worker.service: Unit entered failed state.
May 17 15:52:38 ostheim systemd[1]: resque-worker.service: Failed with result 'exit-code'.
In this case i used the root path of my Rails app for "/home/pageflow/pageflow_daad/rake".
The times before where i tried the path of the rake binary i got the error:May 17 15:30:26 ostheim rake[26846]: rake aborted!
May 17 15:30:26 ostheim rake[26846]: ArgumentError: couldn't find HOME environment -- expanding~'`
I hope someone with more experience in this can help me out.
Thanks in advance and best regards,
Ronald

After studying the several docu sites and the documentation itself i found a way to get this running. Just wanted to post this if someone finds this to be helpfull:
[Unit]
Description=resque-scheduler for pageflow
[Service]
User=yourUser
WorkingDirectory=/path/to/rails/app
ExecStart=/path/to/executeable/rake resque:scheduler &
Environment=QUEUE=*
Environment=RAILS_ENV=production
[Install]
WantedBy=multi-user.target
With this script, a
systemctl daemon-reload
&
systemctl start example.service
The Service startet running and runs like a charm.

I think there 2 types of errors inside the systemd unit. Here are some advices :
1) Let systemd capture your application logs, then use journalctl to look at it
journalctl -u resque-worker.service
2) Use env the systemd way :
[Unit]
Description=resque-worker for pageflow
[Service]
Type=forking
ExecStart=/home/pageflow/pageflow_daad/rake resque:scheduler
Environement=QUEUE=*
Enrironement=RAILS_ENV=production
[Install]
WantedBy=multi-user.target
Then,
systemctl daemon-reload
systemctl restart resque-worker.service

Related

Job for sidekiq.service failed in ubuntu 20.04 deployment server

I have tired to setup sidekiq on ubuntu
This is my sidekiq.service file (wrote by this example)
[Unit]
Description=sidekiq
After=syslog.target network.target
[Service]
Type=notify
WatchdogSec=10
WorkingDirectory=/var/www/document-draft
# WorkingDirectory=/var/www/document-draft/current -> I also tried this
ExecStart=/bundle exec sidekiq -e production
# I have also tried these commands:
# ExecStart=/sudo bundle exec sidekiq -e production
# ExecStart=bundle exec sidekiq -e production
# ExecStart=/home/deploy/.rvm/gems/ruby-2.7.1/wrappers/bundle exec sidekiq -e production
# ExecStart=/home/deploy/.rvm/bin/rvm in /opt/myapp/current do bundle exec sidekiq -e production
Environment=MALLOC_ARENA_MAX=2
RestartSec=1
Restart=always
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=sidekiq
[Install]
WantedBy=multi-user.target
I'm using ruby 2.7 and when I start sidekiq service
$ systemctl enable sidekiq
$ systemctl start sidekiq
I get this error
Job for sidekiq.service failed because the control process exited with error code.
See "systemctl status sidekiq.service" and "journalctl -xe" for details.
when I check logs I see this
● sidekiq.service - sidekiq
Loaded: loaded (/lib/systemd/system/sidekiq.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2022-04-08 05:04:21 UTC; 9s ago
Process: 150072 ExecStart=/sudo bundle exec sidekiq -e production (code=exited, status=200/CHDIR)
Main PID: 150072 (code=exited, status=200/CHDIR)
Apr 08 05:04:19 ip-172-31-29-35 systemd[1]: sidekiq.service: Main process exited, code=exited, status=200/CHDIR
Apr 08 05:04:19 ip-172-31-29-35 systemd[1]: sidekiq.service: Failed with result 'exit-code'.
Apr 08 05:04:19 ip-172-31-29-35 systemd[1]: Failed to start sidekiq.
Apr 08 05:04:21 ip-172-31-29-35 systemd[1]: sidekiq.service: Scheduled restart job, restart counter is at 5.
Apr 08 05:04:21 ip-172-31-29-35 systemd[1]: Stopped sidekiq.
Apr 08 05:04:21 ip-172-31-29-35 systemd[1]: sidekiq.service: Start request repeated too quickly.
Apr 08 05:04:21 ip-172-31-29-35 systemd[1]: sidekiq.service: Failed with result 'exit-code'.
Apr 08 05:04:21 ip-172-31-29-35 systemd[1]: Failed to start sidekiq.
I'm confused on why I cannot start sidekiq because my Gemfile has sidekiq gem and i can successfully start it manually using any of the commands I'm using in service file.
But my motive is to start is as a background service so it may not shut down.
I was able to make it run by writing my sidekiq.service job in /lib/systemd/system using sudo vim /lib/systemd/system/sidekiq.service
# start us only once the network and logging subsystems are available,
# consider adding redis-server.service if Redis is local and systemd-managed.
After=syslog.target network.target
[Service]
# You may want to use
# Type=notify
# to ensure service is not marked as started before it actually did.
# Include sd_notify gem to send a message on sidekiq startup like
# Sidekiq.configure_server do |config|
# config.on(:startup) { SdNotify.ready }
# end
# to let systemd know when the service is actually started.
Type=simple
WorkingDirectory=/var/www/document-draft
# If you use rbenv:
ExecStart=/bin/bash -lc 'exec /home/ubuntu/.rbenv/shims/bundle exec sidekiq -e production'
# If you use the system's ruby:
# ExecStart=/bin/bash -lc 'exec /home/deploy/.rvm/wrappers/ruby-2.6.5#brentmark-portal/bundle exec sidekiq -e production'
# use `systemctl reload sidekiq` to send the quiet signal to Sidekiq
# at the start of your deploy process.
ExecReload=/usr/bin/kill -TSTP $MAINPID
User=ubuntu
Group=ubuntu
UMask=0002
# Greatly reduce Ruby memory fragmentation and heap usage
# https://www.mikeperham.com/2018/04/25/taming-rails-memory-bloat/
Environment=MALLOC_ARENA_MAX=2
# if we crash, restart
RestartSec=1
Restart=on-failure
# output goes to /var/log/syslog
# StandardOutput=syslog
# StandardError=syslog
# ERROR: Logfile redirection was removed in Sidekiq 6.0, Sidekiq will only log to STDOUT
# StandardOutput=/var/www/sites/document-draft/log/sidekiq.log
# StandardError=/var/www/sites/document-draft/log/sidekiq.log
# This will default to "bundler" if we don't specify it
SyslogIdentifier=sidekiq
[Install]
WantedBy=multi-user.target
After that systemctl start sidekiq to start this service

Docker daemon cannot be started for some (hidden) reason

I am trying to push a docker image and noticed that my docker daemon actually is probably not running.
If for example I run:
docker run hello-world
docker: Cannot connect to the Docker daemon at
unix:///var/run/docker.sock. Is the docker daemon running?.
If I try to restart the daemon using:
systemctl start docker
Job for docker.service failed because the control process exited with
error code. See "systemctl status docker.service" and "journalctl -xe"
for details.
Continuing running:
systemctl status docker.service
docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor
preset: enabled)
Active: failed (Result: start-limit-hit) since Wed 2021-05-12 14:45:09
EEST; 43s ago
Docs: https://docs.docker.com
Process: 4810 ExecStart=/usr/bin/dockerd -H fd://
--containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
Main PID: 4810 (code=exited, status=1/FAILURE)
May 12 14:45:07 iti-554 systemd[1]: docker.service: Unit entered
failed state.
May 12 14:45:07 iti-554 systemd[1]: docker.service: Failed with result
'exit-code'.
May 12 14:45:09 iti-554 systemd[1]: docker.service: Service hold-off
time over, scheduling restart.
May 12 14:45:09 iti-554 systemd[1]: Stopped Docker Application
Container Engine.
May 12 14:45:09 iti-554 systemd[1]: docker.service: Start request
repeated too quickly.
May 12 14:45:09 iti-554 systemd[1]: Failed to start Docker Application
Container Engine.
May 12 14:45:09 iti-554 systemd[1]: docker.service: Unit entered
failed state.
May 12 14:45:09 iti-554 systemd[1]: docker.service: Failed with result
'start-limit-hit'.
which as I understand it it means docker daemon is not loaded (it's in a failed state) and the last reason for this is the start-limit-hit has been reached. This on this side probably means another reason exists for this to happen.
SO, how do I find out which is the actual reason for my docker daemon refusing to start?
If I run to reset the failed attemps counter with:
systemctl reset-failed docker.service
it return without error so I assume it succeeds. And indeed when I check the status it has become:
Active: inactive (dead) since Wed 2021-05-12 14:45:09 EEST; 14min ago
Of course if I run docker daemon again it fails.
Can someone provide any workaround about this issue? I even tried to invoke the commands after restarting (didn't help).
Edit
Well, to my case the problem was a rather stupid one. I had added a daemon.json file with minimal content in it. Just this:
cat /etc/docker/daemon.json
{
"insecure-registries": [
"docker-server.com:10022",
"docker-server.com:10023"
],
}
The problem was that the dangling comma before } made docker search for another parameter. The relevant message shown using journalctl -u docker was:
unable to configure the Docker daemon with file
/etc/docker/daemon.json: invalid character '}' looking for beginning
of object key string
is quite obvious but the previous ones did not help much.
journalctl -u docker gives you docker daemon logs. Maybe u can find something there.
The unix:///var/run/docker.sock requires the correct permission to work. This a security feature for Docker.
Try sudo chmod 755 /var/run/docker.sock and re-run Docker command.
Note the permission number given here may not be suitable for everyone.

docker.socket: Failed with result 'service-start-limit-hit' after protecting docker daemon socket

I followed the steps provided in the documentation here to add tls security for docker api. Certificates are located in ~/.docker/ as well as /etc/docker/ssl/ folders. I added override.conf to /etc/systemd/system/docker.service.d/ with content
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 --tlsverify --tlscacert=ca.pem --tlscert=server-cert.pem --tlskey=server-key.pem
Then, I used daemon-reload and docker start
$ systemctl daemon-reload
$ service docker start
The errors in journalctl -xe is:
-- Unit docker.socket has finished starting up.
--
-- The start-up result is RESULT.
Jan 15 21:43:24 cynicalplyaground systemd[1]: docker.service: Start request repeated too quickly.
Jan 15 21:43:24 cynicalplyaground systemd[1]: docker.service: Failed with result 'exit-code'.
Jan 15 21:43:24 cynicalplyaground systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit docker.service has failed.
--
-- The result is RESULT.
Jan 15 21:43:24 cynicalplyaground systemd[1]: docker.socket: Failed with result 'service-start-limit-hit'.
Jan 15 21:45:01 cynicalplyaground CRON[12768]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan 15 21:45:01 cynicalplyaground CRON[12769]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jan 15 21:45:01 cynicalplyaground CRON[12768]: pam_unix(cron:session): session closed for user root
How can I sort this issue?
In the present case the same error occured after the latest manjaro update (2020-01-20).
Tried to change the systemd docker service, as adviced in other cases, but I reverted those changes and finally this was solved with:
a reboot of the system
(like advised here: https://www.reddit.com/r/archlinux/comments/7ya4ug/installing_docker_on_arch_linux/)
Getting to the root of the problem;
systemctl status docker.service
has this:
/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Trying to run that command, it complains about
unable to configure the Docker daemon with file /etc/docker/daemon.json: EOF
ls -l /etc/docker/daemon.json
-rw-r--r-- 1 root root 0 Jul 30 10:32 /etc/docker/daemon.json
NOTE that the JSON file is empty. Delete it.
For me it was because the docker installer uses iptables for nat. Unfortunately Debian uses nftables. You can convert the entries over to nftables or just setup Debian to use the legacy iptables.
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
dockerd, should start fine after switching to iptables-legacy.
I have the same issue and just modify the "/usr/bin/dockerd" to "/usr/sbin/dockerd", then it works.
You can check the dockerd path first.
in my case... the host was part of a docker swarm...but the IPv6 was no longer reachable or automatically assigned to the host...
I manually add the old_IPv6
ip -6 address add 28xx:xxxx:x:x:xx:ebff:fe14:xxx dev ens3x
the journalctl -u docker.service mention:
level=fatal msg="Error starting cluster component: could not find local IP address: dial udp [2xxx:xxx:xxxx:xxx]:2377: connect: network is unreachable"
after add manually the IPv6 I was able to start docker so with docker running I leave the "swarm" and reboot
docker swarm leave --force
after reboot the docker services run as usual
For me it was missing disk space. Reboot also helped, but I was stillnot able to build any container.
After pruning some outdated stuff from the docker volumes I was able to continue.
I faced a similar issue on Ubuntu because I added the hosts option to /etc/docker/daemon.json file. That's ok, but for systems that use systemd it may cause conflict with the arguments passed to dockerd on start.
The solution was to delete the /etc/docker/daemon.json's hosts entry and set this config on file /etc/systemd/system/docker.service.d/options.conf.
$ cat /etc/systemd/system/docker.service.d/options.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix://
After that, restart the service.
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker
You may check that your changes has been applied by running docker info. Also, you may note on the docker service status that Drop-In field is using the options.conf created, and dockerd was executed with the specified host list.
$ systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset>
Drop-In: /etc/systemd/system/docker.service.d
└─options.conf
Active: active (running) since Fri 2022-11-18 01:02:18 EST; 1h 50min ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 1111 (dockerd)
Tasks: 18
Memory: 58.5M
CPU: 1.294s
CGroup: /system.slice/docker.service
└─1111 /usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix://
References:
Daemon configuration file
Control Docker with systemd
I had a similar issue on nixOS installed in a btrfs filesystem.
For me the solution was to add virtualisation.docker.storageDriver = "btrfs"; to my /etc/nixos/configuration.nix
Which according to the docker docs should equate to adding the following to /etc/docker/daemon.json in most other distros:
{
"storage-driver": "btrfs"
}
I was able to solve the problem by disabling the firewalld
systemctl disable firewalld
systemctl stop firewalld

Docker containers shut down after systemd start

For some reason when using systemd unit files my docker containers start but get shut down instantly. I have tried finding logs but can not see any indication on why this is happening. Is there someone that knows how to solve this / find the logs that show what is happening?
Note: When starting them manually after boot with docker start containername then it works (also when using systemctl start nginx)
After some more digging I found this error: could not find udev device: No such device it could have something to do with this?
Unit Service file:
[Unit]
Description=nginx-container
Requires=docker.service
After=docker.service
[Service]
Restart=always
RestartSec=2
StartLimitInterval=3600
StartLimitBurst=5
TimeoutStartSec=5
ExecStartPre=-/usr/bin/docker kill nginx
ExecStartPre=-/usr/bin/docker rm nginx
ExecStart=/usr/bin/docker run -i -d -t --restart=no --name nginx -p 80:80 -v /projects/frontend/data/nginx/:/var/www -v /projects/frontend: nginx
ExecStop=/usr/bin/docker stop -t 2 nginx
[Install]
WantedBy=multi-user.target
Journalctl output:
May 28 11:18:15 frontend dockerd[462]: time="2015-05-28T11:18:15Z" level=info msg="-job start(d757f83d4a13f876140ae008da943e8c5c3a0765c1fe5bc4a4e2599b70c30626) = OK (0)"
May 28 11:18:15 frontend dockerd[462]: time="2015-05-28T11:18:15Z" level=info msg="POST /v1.18/containers/nginx/stop?t=2"
May 28 11:18:15 frontend dockerd[462]: time="2015-05-28T11:18:15Z" level=info msg="+job stop(nginx)"
Docker logs: empty (docker logs nginx)
Systemctl output: (systemctl status nginx, nginx.service)
● nginx.service - nginx-container
Loaded: loaded (/etc/systemd/system/multi-user.target.wants/nginx.service)
Active: failed (Result: start-limit) since Thu 2015-05-28 11:18:20 UTC; 12min ago
Process: 3378 ExecStop=/usr/bin/docker stop -t 2 nginx (code=exited, status=0/SUCCESS)
Process: 3281 ExecStart=/usr/bin/docker run -i -d -t --restart=no --name nginx -p 80:80 -v /projects/frontend/data/nginx/:/var/www -v /projects/frontend:/nginx (code=exited, status=0/SUCCESS)
Process: 3258 ExecStartPre=/usr/bin/docker rm nginx (code=exited, status=0/SUCCESS)
Process: 3246 ExecStartPre=/usr/bin/docker kill nginx (code=exited, status=0/SUCCESS)
Main PID: 3281 (code=exited, status=0/SUCCESS)
May 28 11:18:20,frontend systemd[1]: nginx.service holdoff time over, scheduling restart.
May 28 11:18:20 frontend systemd[1]: start request repeated too quickly for nginx.service
May 28 11:18:20 frontend systemd[1]: Failed to start nginx-container.
May 28 11:18:20 frontend systemd[1]: Unit nginx.service entered failed state.
May 28 11:18:20 frontend systemd[1]: nginx.service failed.
Because you have not specified a Type in your systemd unit file, systemd is using the default, simple. From systemd.service:
If set to simple (the default if neither Type= nor BusName=, but
ExecStart= are specified), it is expected that the process
configured with ExecStart= is the main process of the service.
This means that if the process started by ExecStart exits, systemd
will assume your service has exited and will clean everything up.
Because you are running the docker client with -d, it exits
immediately...thus, systemd cleans up the service.
Typically, when starting containers with systemd, you would not use
the -d flag. This means that the client will continue running, and
will allow systemd to collect any output produced by your application.
That said, there are fundamental problems in starting Docker containers with systemd. Because of the way Docker operates, there really is no way for systemd to monitor the status of your container. All it can really do is track the status of the docker client, which is not the same thing (the client can exit/crash/etc without impacting your container). This isn't just relevant to systemd; any sort of process supervisor (upstart, runit, supervisor, etc) will have the same problem.

run puma server as a service at centos 7 - no ruby found

There are many things I do not understand, so my question may be silly.
I want to run a puma ror server as a systemd service at centos 7. Use ruby installed using rvm.
My puma_test.service file is:
[Unit]
Description=Puma application server
After=network.target
[Service]
WorkingDirectory=/var/www/test_app
Environment=RAILS_ENV=development
PIDFile=/var/www/shared/pids/puma.pid
ExecStart=/usr/local/rvm/gems/ruby-2.2.1/gems/bundler-1.9.4/bin/bundle exec puma -e development -b unix:///var/www/shared/pids/puma.sock --pidfile /var/www/shared/pids/puma.pid
[Install]
WantedBy=multi-user.target
but when I run it, it does not work. I get error (from journalctl):
kwi 18 22:56:15 vps150852.ovh.net systemd[1]: Starting Puma application server...
kwi 18 22:56:15 vps150852.ovh.net systemd[1]: Started Puma application server.
kwi 18 22:56:15 vps150852.ovh.net bundle[2072]: /usr/bin/env: ruby: No such file or directory
kwi 18 22:56:15 vps150852.ovh.net systemd[1]: puma_test.service: main process exited, code=exited, status=127/n/a
kwi 18 22:56:15 vps150852.ovh.net systemd[1]: Unit puma_test.service entered failed state.
when I run i /usr/www/test_app
/usr/local/rvm/gems/ruby-2.2.1/gems/bundler-1.9.4/bin/bundle exec puma -e development -b unix:///var/www/shared/pids/puma.sock --pidfile /var/www/shared/pids/puma.pid
everything works fine, but I am probably doing something wrong
Looks like you need to load rvm when you run your task. systemd run in shell, not in bash, your bashrc will not be loaded

Resources