I cannot figure this out. Hope I can get some help
I have a hobby tier in Heroku running django. To this I would like to attach a Redis service. However, I would like to use the recent rejson (redislabs/rejson:latest) docker image (instead of redistogo or heroku-redis)because it has json support. This works great on my local env. I was able to push the docker image into the container registry and actually start the redis server
2021-07-23T00:14:48.576294+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.576 # Current maximum open files is 10000. maxclients has been reduced to 9968 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
2021-07-23T00:14:48.577054+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 * Running mode=standalone, port=6379.
2021-07-23T00:14:48.577124+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 # Server initialized
2021-07-23T00:14:48.577184+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2021-07-23T00:14:48.577671+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 # <ReJSON> JSON data type for Redis v1.0.7 [encver 0]
2021-07-23T00:14:48.577789+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 * Module 'ReJSON' loaded from /usr/lib/redis/modules/rejson.so
2021-07-23T00:14:48.578232+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.578 * Ready to accept connections
Unfortunately, django is unable to connect to it
ConnectionError: Error -2 connecting to redis://localhost:6379. Name or service not known
There are no ENV variable exposed, so am unable to set any (I mean I can set them, but doubt if they will be relevant)
I did experiment with installing redistogo add on and am able to connect to it (this required setting the connection based on the REDIS_URL env variable that gets exposed when redistogo is added)
At my wits end...any help appreciated...I guess the question really boils do to:
Related
I've google this, but so far no way to fix it. My syslog under /var/log is being flooded every second with messages like this;
Aug 27 20:58:27 mail-server systemd[1]: run-docker-runtime\x2drunc-moby-e4bfb13118b141bf232cf981fe9b535706243c47ae0659466b8e6667bd4feceb-runc.YHoxmJ.mount: Succeeded.
Aug 27 20:58:27 mail-server systemd[1083]: run-docker-runtime\x2drunc-moby-e4bfb13118b141bf232cf981fe9b535706243c47ae0659466b8e6667bd4feceb-runc.YHoxmJ.mount: Succeeded.
Aug 27 20:58:27 mail-server systemd[8395]: run-docker-runtime\x2drunc-moby-e4bfb13118b141bf232cf981fe9b535706243c47ae0659466b8e6667bd4feceb-runc.YHoxmJ.mount: Succeeded.
Aug 27 20:58:28 mail-server systemd[1]: run-docker-runtime\x2drunc-moby-5dc4f4e0b3cbd5e5bfbcc88b8d22f92575706b7c3603847ccb2fd4e56f188f99-runc.gt51Ek.mount: Succeeded.
Aug 27 20:58:28 mail-server systemd[1083]: run-docker-runtime\x2drunc-moby-5dc4f4e0b3cbd5e5bfbcc88b8d22f92575706b7c3603847ccb2fd4e56f188f99-runc.gt51Ek.mount: Succeeded.
Aug 27 20:58:28 mail-server systemd[8395]: run-docker-runtime\x2drunc-moby-5dc4f4e0b3cbd5e5bfbcc88b8d22f92575706b7c3603847ccb2fd4e56f188f99-runc.gt51Ek.mount: Succeeded.
I am running Ubuntu 20.04 and dockerd is run by systemd.
Could anyone help me to find the cause if this? It seems that every single container is generating this.
Best,
Francis
Those messages are from systemd itself about the mount. This is addressed in systemd v249; see https://github.com/systemd/systemd/issues/6432 for more information.
In a nutshell, that version of systemd allows controlling of that mount via its unit file using the following:
[Mount]
LogLevelMax=0
The LogLevelMax setting applies not just to the unit but also to systemd's log messages itself about the unit. That is the change introduced in v249.
I am running the gitlab with docker, but it always exits after a period of time
==> /var/log/gitlab/redis/current <==
2019-06-21_18:00:08.72435 459:signal-handler (1561140008) Received SIGTERM scheduling shutdown...
2019-06-21_18:00:08.81864 459:M 21 Jun 18:00:08.817 # User requested shutdown...
2019-06-21_18:00:08.81866 459:M 21 Jun 18:00:08.817 * Saving the final RDB snapshot before exiting.
2019-06-21_18:00:08.83736 459:M 21 Jun 18:00:08.837 * DB saved on disk
2019-06-21_18:00:08.83741 459:M 21 Jun 18:00:08.837 * Removing the pid file.
2019-06-21_18:00:08.83817 459:M 21 Jun 18:00:08.838 * Removing the unix socket file.
2019-06-21_18:00:08.83935 459:M 21 Jun 18:00:08.839 # Redis is now ready to exit, bye bye...
ok: down: redis-exporter: 0s, normally up
==> /var/log/gitlab/gitlab-rails/sidekiq.log <==
2019-06-21_18:00:09.57615 2019-06-21T18:00:09.576Z 807 TID-oviw2sgmf INFO: Shutting down
2019-06-21_18:00:09.57625 2019-06-21T18:00:09.576Z 807 TID-ovivo05i7 INFO: Scheduler exiting...
2019-06-21_18:00:09.57655 2019-06-21T18:00:09.576Z 807 TID-oviw2sgmf INFO: Terminating quiet workers
This was reported in gitlab-org/omnibus-gitlab issue 4137: "runsv send SIGTERM to redis in docker version"
runsv sends SIGTERM to redis every 60 secs
gitlab-org/omnibus-gitlab issue 1611 suggests a docker restart first.
But the general issue is not conclusively resolved yet.
I'm banging my head against the wall with puma and systemd. I used foreman to set up my systemd files, but can't get puma out of its restart loop. Ubuntu 16.
Jun 19 02:48:12 ip-172-31-28-225 systemd[1]: Stopped rajlogviewer-web.service.
Jun 19 02:48:12 ip-172-31-28-225 systemd[1]: Started rajlogviewer-web.service.
Jun 19 02:48:12 ip-172-31-28-225 rajlogviewer-web.service[8954]: APP_DIR = /home/ubuntu/rajlogviewer, SHARED_DIR /home/ubuntu/rajlogviewer/shared
Jun 19 02:48:12 ip-172-31-28-225 rajlogviewer-web.service[8954]: [8954] Puma starting in cluster mode...
Jun 19 02:48:12 ip-172-31-28-225 rajlogviewer-web.service[8954]: [8954] * Version 3.6.0 (ruby 2.3.3-p222), codename: Sleepy Sunday Serenity
Jun 19 02:48:12 ip-172-31-28-225 rajlogviewer-web.service[8954]: [8954] * Min threads: 1, max threads: 6
Jun 19 02:48:12 ip-172-31-28-225 rajlogviewer-web.service[8954]: [8954] * Environment: production
Jun 19 02:48:12 ip-172-31-28-225 rajlogviewer-web.service[8954]: [8954] * Process workers: 2
Jun 19 02:48:12 ip-172-31-28-225 rajlogviewer-web.service[8954]: [8954] * Preloading application
Jun 19 02:48:14 ip-172-31-28-225 rajlogviewer-web.service[8954]: [8954] * Listening on tcp://0.0.0.0:3000
Jun 19 02:48:14 ip-172-31-28-225 rajlogviewer-web.service[8954]: [8954] * Listening on unix:///home/ubuntu/rajlogviewer/shared/tmp/sockets/puma.sock
Jun 19 02:48:14 ip-172-31-28-225 rajlogviewer-web.service[8954]: [8954] * Daemonizing...
Jun 19 02:48:24 ip-172-31-28-225 systemd[1]: rajlogviewer-web.service: Service hold-off time over, scheduling restart.
Jun 19 02:48:24 ip-172-31-28-225 systemd[1]: Stopped rajlogviewer-web.service.
Jun 19 02:48:24 ip-172-31-28-225 systemd[1]: Started rajlogviewer-web.service.
It just keeps restarting indefinitely. Here is my systemd init file
/etc/systemd/system/rajlogviewer-web.service
[Service]
User=ubuntu
WorkingDirectory=/home/ubuntu/rajlogviewer/current
#Environment=PATH=/home/ubuntu/.rvm/gems/ruby-2.3.3#rajlogsViewer/bin:$PATH
WorkingDirectory=/home/ubuntu/rajlogviewer/current/
ExecStart=/bin/bash -lc 'PATH=/home/ubuntu/.rvm/gems/ruby-2.3.3#rajlogsViewer/bin:$PATH exec /home/ubuntu/.rvm/bin/rvm ruby-2.3.3 do bundle exec puma -C /home/ubuntu/rajlogviewer
/shared/config/puma.rb --daemon'
Restart=no
RestartSec=10
StandardInput=null
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=%n
KillMode=mixed
TimeoutStopSec=5
The app usually boots in 5 seconds when booting with 'cap production puma:start' which works, so RestartSec=10 ought to work. Making it 20 seconds makes no difference.
Nothing interesting in the puma.stderr.log and puma.stdout.log .
Any ideas?
You should remove --daemon option from the systemd, there are several types of unit:
Configures the unit process startup type that affects the
functionality of ExecStart and related options. One of:
simple – The
default value. The process started with ExecStart is the main process
of the service.
forking – The process started with ExecStart spawns a
child process that becomes the main process of the service. The parent
process exits when the startup is complete.
oneshot – This type is
similar to simple, but the process exits before starting consequent
units. dbus – This type is similar to simple, but consequent units are
started only after the main process gains a D-Bus name.
notify – This
type is similar to simple, but consequent units are started only after
a notification message is sent via the sd_notify() function.
idle –
similar to simple, the actual execution of the service binary is
delayed until all jobs are finished, which avoids mixing the status
output with shell output of services.
The default value is simple, for the sake of puma configuration you use --daemon options which contradict with the systemd configuration.
I have an app setup with Rails 3.1, Mongo 1.4.0, Mongoid 2.2.4.
What I am experiencing is this:
Mongo::ConnectionFailure: Failed to connect to a master node at localhost:27017
I've had this problem before, but it went away on a computer restart... this time it does not.
I don't understand, I didn't do anything. I just put my computer in sleep mode, went home and woke it up, then there it was.
Here is the output of sudo mongod
Fri Nov 25 21:47:14 [initandlisten] MongoDB starting : pid=1963 port=27017 dbpath=/data/db/ 64-bit host=xxx.local
Fri Nov 25 21:47:14 [initandlisten] db version v2.0.0, pdfile version 4.5
Fri Nov 25 21:47:14 [initandlisten] git version: 695c67dff0ffc361b8568a13366f027caa406222
Fri Nov 25 21:47:14 [initandlisten] build info: Darwin erh2.10gen.cc 9.6.0 Darwin Kernel Version 9.6.0: Mon Nov 24 17:37:00 PST 2008; root:xnu-1228.9.59~1/RELEASE_I386 i386 BOOST_LIB_VERSION=1_40
Fri Nov 25 21:47:14 [initandlisten] options: {}
Fri Nov 25 21:47:14 [initandlisten] journal dir=/data/db/journal
Fri Nov 25 21:47:14 [initandlisten] recover : no journal files present, no recovery needed
Fri Nov 25 21:47:15 [websvr] admin web console waiting for connections on port 28017
Fri Nov 25 21:47:15 [initandlisten] waiting for connections on port 27017
And I am able to connect with mongoin terminal.
After 2 hours of Googling I hope the competence of SOs community are able to figure this out.
Please, if you need more information about my app-setup just ask.
Thanks!
What you see is that the connection times out... that happens either after a long period of inactivity, or if you put your computer to sleep.
You can change / increase the timeout value, but this way you can't get rid of the connection timing out eventually.
Some MongoDB drivers allow to set :timeout => false , but Mongoid seems to still have problems with that
(see last 3 links in the list)
Hope this helps.
See also:
Mongodb server goes down, how to prevent Rails app from timing out?
MongoDB: What is connection pooling and timeout?
https://github.com/mongodb/mongo-ruby-driver
How can I query mongodb using mongoid/rails without timing out?
http://groups.google.com/group/mongoid/browse_thread/thread/b5c94e7047b42f8a
https://github.com/mongoid/mongoid/issues/455
Try to change localhost to 127.0.0.1!
I am in the process of setting up a server to run a Ruby on Rails application on Fedora 12, using Passenger.
I am at the stage where I've installed Passenger, set it up as prescribed, but get the following errors when I restart Apache:
[Wed Jan 13 15:41:38 2010] [notice] caught SIGTERM, shutting down
[Wed Jan 13 15:41:40 2010] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:httpd_t:s0
[Wed Jan 13 15:41:40 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Jan 13 15:41:40 2010] [error] *** Passenger could not be initialized because of this error: Cannot create FIFO file /tmp/passenger.25235/.guard: Permission denied (13)
[Wed Jan 13 15:41:40 2010] [notice] Digest: generating secret for digest authentication ...
[Wed Jan 13 15:41:40 2010] [notice] Digest: done
[Wed Jan 13 15:41:40 2010] [error] *** Passenger could not be initialized because of this error: Cannot create FIFO file /tmp/passenger.25235/.guard: Permission denied (13)
[Wed Jan 13 15:41:40 2010] [error] python_init: Python version mismatch, expected '2.6', found '2.6.2'.
[Wed Jan 13 15:41:40 2010] [error] python_init: Python executable found '/usr/bin/python'.
[Wed Jan 13 15:41:40 2010] [error] python_init: Python path being used '/usr/lib/python26.zip:/usr/lib/python2.6/:/usr/lib/python2.6/plat-linux2:/usr/lib/python2.6/lib-tk:/usr/lib/python2.6/lib-old:/usr/lib/python2.6/lib-dynload'.
[Wed Jan 13 15:41:40 2010] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Wed Jan 13 15:41:40 2010] [notice] mod_python: using mutex_directory /tmp
[Wed Jan 13 15:41:40 2010] [notice] Apache/2.2.14 (Unix) DAV/2 Phusion_Passenger/2.2.9 PHP/5.3.0 mod_python/3.3.1 Python/2.6.2 mod_ssl/2.2.14 OpenSSL/1.0.0-fips-beta3 mod_perl/2.0.4 Perl/v5.10.0 configured -- resuming normal operations
As you can see, there is a permissions problem when Passenger is trying to initialize:
[Wed Jan 13 15:41:40 2010] [error] *** Passenger could not be initialized because of this error: Cannot create FIFO file /tmp/passenger.25235/.guard: Permission denied (13)
When Apache is starts, it does create a file in /tmp:
d-ws--x--x. 2 root root 4096 2010-01-13 16:04 passenger.26117
If instead I run the app by firing up mongrel directly with mongrel_rails start -e production, I see the following:
ActiveRecord::StatementInvalid (Mysql::Error: Can't create/write to file '/tmp/#sql_5d3_0.MYI' (Errcode: 13): SHOW FIELDS FROM `users`):
Again the error points to permission issues with the /tmp directory.
I am at a loss as to what the solution is. I'm not sure if it is related to simply directory permissions or Fedora's SELinux security.
Any help would be appreciated. Thanks.
I did the same as Fred, except that instead of doing it one error at a time:
Go into permissive mode by running setenforce 0
Restart apache, and hit your site and use it for a while as normal
Run grep httpd /var/log/audit/audit.log | audit2allow -M passenger
semodule -i passenger.pp
Go back to enforcing mode by running setenforce 1
Restart apache and test your site - hopefully it should all be working as before!
Note that this is basically a specific example of the procedure on the Centos SELinux help - check it out.
I'm having the same issue in CentOS 5.4, SELinux getting in the way of Passenger.
Setting PassengerTempDir to /var/run/passenger simply gives you the same permission errors in the new directory instead of /tmp :
[Mon Feb 22 11:42:40 2010] [error] *** Passenger could not be initialized because of this error: Cannot create directory '/var/run/passenger/passenger.3686'
I can then change the security context of /var/run/passenger to get past this error:
chcon -R -h -t httpd_sys_content_t /var/run/passenger/
...and that lets Passenger create the temp directory, but not files within that directory:
[Mon Feb 22 12:07:06 2010] [error] *** Passenger could not be initialized because of this error: Cannot create FIFO file /var/run/passenger/passenger.3686/.guard: Permission denied (13)
Oddly, re-running the recursive chcon again doesn't get past this error, it keeps dying at this point, and this is where my SELinux knowledge gets murky.
The Phusion Passenger guide sections 6.3.5 and 6.3.7 have some useful thoughts, but they don't seem to completely resolve the problem.
You need more than just the httpd_sys_content_t permission. I use the following technique to get things started:
start a tail on the audit log: tail -f /var/log/audit/audit.log
reload apache: apachectl restart
Go to the /tmp/directory: cd /tmp
If just 1 line is added use the command: tail -1 /var/log/audit/audit.log | audit2allow -M httpdfifo
Note that the name 'httpdfifo' is just a name chosen to reflect the kind of error that has been observed.
This will create a file named 'httpdfifo.pp'. To allow apache to create a FIFO from here on after you have to issue the command: semodule -i httpdfifo.pp
Continue to do this until all audit errors have been resolved (It took 4 different kind of permissions on my system running Centos 5.4)
Running setenforce 0 before starting will let you test if it's SELinux. Don't forget to run setenforce 1 afterwards.
I tried what Dan Sketcher and Fred Appleman suggested, i.e. repeat the following:
yum install setroubleshoot
echo > /var/log/audit/audit.log # clear irrelevant errors
cd ~
service httpd restart # try booting passenger -- audit.log now shows the relevant permission errors
tail -f /var/log/httpd/error_log # check that passenger is still failing due to permission errors
sealert -a /var/log/audit/audit.log > selinux-diag.txt # translate the permission errors
# read and check that you are happy with selinux-diag.txt
# and either follow its specific advice, or if it just wants you to grep into audit2allow, then:
cat /var/log/audit/audit.log | audit2allow -M mypol # grant everything just denied
semodule -i mypol.p # commit new permissions
But after doing this 5 or 6 times, I kept coming up against new errors, and some of the same errors came up even after I had tried to permit them with "audit2allow".
In the end I just turned off SELinux, with:
echo 0 >/selinux/enforce