Today, I have noticed that unicorn was not running on the production environment. Unicorn error log is as follows (it seems unicorn stopped due to SIGKILL signal):
D, [2017-10-11T12:25:49.521336 #9375] DEBUG -- : waiting 16.0s after suspend/hibernation
E, [2017-10-11T12:26:08.051349 #9375] ERROR -- : reaped #<Process::Status: pid 9535 SIGKILL (signal 9)> worker=1
I, [2017-10-11T12:26:21.708135 #9375] INFO -- : worker=1 spawning...
It seeems when the unicorn was killed, we had faced a out of memory problem. In the /var/log/syslog.1 I have following logs:
Oct 11 12:26:02 analiz kernel: [4486147.309450] [<ffffffff81190979>] out_of_memory+0x219/0x460
Oct 11 12:26:02 analiz kernel: [4486147.309677] Out of memory: Kill process 9375 (ruby2.3) score 81 or sacrifice child
Now my questions are;
How I can make sure that unicorn will be restarted automatically even if it gets killed by kernel? (eg, restart automatically after 30secs)
In the normal operation, memory utilization is around %50-%60. I would like to understand the root cause and take measures to prevent a similar problem in future. According to the syslog messages, out of memory issue is started with postgres. I have no clue what caused a high memory utilization by postgres or other processes. Any kind of support will be appreciated.
Oct 11 12:26:02 analiz kernel: [4486147.309419] postgres invoked oom-killer: gfp_mask=0x24200ca, order=0, oom_score_adj=0
Oct 11 12:26:02 analiz kernel: [4486147.309422] postgres cpuset=/ mems_allowed=0
Oct 11 12:26:02 analiz kernel: [4486147.309427] CPU: 0 PID: 9570 Comm: postgres Not tainted 4.4.0-1031-aws #40-Ubuntu
Oct 11 12:26:02 analiz kernel: [4486147.309428] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017
Oct 11 12:26:02 analiz kernel: [4486147.309429] 0000000000000286 08956ec91b465ba3 ffff88000918f9e8 ffffffff813f6ec3
Oct 11 12:26:02 analiz kernel: [4486147.309432] ffff88000918fba0 ffff88003cfdb700 ffff88000918fa58 ffffffff81208b3a
Oct 11 12:26:02 analiz kernel: [4486147.309433] ffffffff81cc3661 0000000000000000 ffffffff81e667a0 0000000000000206
Thanks in advance.
Related
I am running the gitlab with docker, but it always exits after a period of time
==> /var/log/gitlab/redis/current <==
2019-06-21_18:00:08.72435 459:signal-handler (1561140008) Received SIGTERM scheduling shutdown...
2019-06-21_18:00:08.81864 459:M 21 Jun 18:00:08.817 # User requested shutdown...
2019-06-21_18:00:08.81866 459:M 21 Jun 18:00:08.817 * Saving the final RDB snapshot before exiting.
2019-06-21_18:00:08.83736 459:M 21 Jun 18:00:08.837 * DB saved on disk
2019-06-21_18:00:08.83741 459:M 21 Jun 18:00:08.837 * Removing the pid file.
2019-06-21_18:00:08.83817 459:M 21 Jun 18:00:08.838 * Removing the unix socket file.
2019-06-21_18:00:08.83935 459:M 21 Jun 18:00:08.839 # Redis is now ready to exit, bye bye...
ok: down: redis-exporter: 0s, normally up
==> /var/log/gitlab/gitlab-rails/sidekiq.log <==
2019-06-21_18:00:09.57615 2019-06-21T18:00:09.576Z 807 TID-oviw2sgmf INFO: Shutting down
2019-06-21_18:00:09.57625 2019-06-21T18:00:09.576Z 807 TID-ovivo05i7 INFO: Scheduler exiting...
2019-06-21_18:00:09.57655 2019-06-21T18:00:09.576Z 807 TID-oviw2sgmf INFO: Terminating quiet workers
This was reported in gitlab-org/omnibus-gitlab issue 4137: "runsv send SIGTERM to redis in docker version"
runsv sends SIGTERM to redis every 60 secs
gitlab-org/omnibus-gitlab issue 1611 suggests a docker restart first.
But the general issue is not conclusively resolved yet.
Answer with repeated occurrence is like below
[root#centos6 bin]# grep -A 2 -B 2 "SampleTextFile.txt" /var/log/messages
Nov 10 00:14:54 centos6 kernel: Kernel logging (proc) stopped.
Nov 10 00:14:54 centos6 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1513" x-info="http://www.rsyslog.com"] exiting on signal 15.
File SampleTextFile.txt
First Line after original SampleTextFile.txt
Nov 12 18:13:48 centos6 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Nov 12 18:13:48 centos6 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1519" x-info="http://www.rsyslog.com"] start
--
Nov 12 18:14:19 centos6 polkitd[2382]: started daemon version 0.96 using authority implementation `local' version `0.96'
Nov 12 18:14:19 centos6 rtkit-daemon[2388]: Sucessfully made thread 2386 of process 2386 (/usr/bin/pulseaudio) owned by '42' high priority at nice level -11.
File SampleTextFile.txt
Second Line after original SampleTextFile.txt
Nov 12 18:14:19 centos6 pulseaudio[2386]: alsa-util.c: Disabling timer-based scheduling because running inside a VM.
Nov 12 18:14:19 centos6 rtkit-daemon[2388]: Sucessfully made thread 2392 of process 2386 (/usr/bin/pulseaudio) owned by '42' RT at priority 5.
[root#centos6 bin]#
If I use -m switch then it doe not show "AFTER" lines...
[root#centos6 bin]# grep -m 1 -A 2 -B 2 "SampleTextFile.txt" /var/log/messages
Nov 10 00:14:54 centos6 kernel: Kernel logging (proc) stopped.
Nov 10 00:14:54 centos6 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1513" x-info="http://www.rsyslog.com"] exiting on signal 15.
File SampleTextFile.txt
[root#centos6 bin]#
My grep version is below:
[root#centos6 bin]# grep -V
grep (GNU grep) 2.20
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Mike Haertel and others, see <http://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.
[root#centos6 bin]#
I would sincerely appreciate any help to solve this issue.
Thank you in advance.
--Edit:
Expected result
[root#centos6 bin]# grep -A 2 -B 2 "SampleTextFile.txt" /var/log/messages
Nov 10 00:14:54 centos6 kernel: Kernel logging (proc) stopped.
Nov 10 00:14:54 centos6 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1513" x-info="http://www.rsyslog.com"] exiting on signal 15.
File SampleTextFile.txt
First Line after original SampleTextFile.txt
Nov 12 18:13:48 centos6 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Nov 12 18:13:48 centos6 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1519" x-info="http://www.rsyslog.com"] start
My web application running on phusion passenger is getting down frequently but i am not able to trace out the reason.
Here is the error i got in /var/log/httpd/error.log file
Premature end of script headers:
The backend application (process 5417) did not send a valid HTTP response; instead, it sent nothing at all. It is possible that it has crashed; please check whether there are crashing bugs in this application.
Last 50 line of log:
[Thu Oct 11 05:29:42 2012] [error] [client 11.992.88.255] Premature end of script headers:
[ pid=22252 thr=1 file=ext/apache2/Hooks.cpp:817 time=2012-10-11 05:29:42.409 ]: The backend application (process 5488) did not send a valid HTTP response; instead, it sent nothing at all. It is possible that it has crashed; please check whether there are crashing bugs in this application.
[Thu Oct 11 05:35:16 2012] [notice] child pid 27200 exit signal Segmentation fault (11)
[Thu Oct 11 05:35:19 2012] [notice] child pid 27196 exit signal Segmentation fault (11)
[Thu Oct 11 05:35:19 2012] [notice] child pid 27199 exit signal Segmentation fault (11)
[Thu Oct 11 05:35:20 2012] [notice] child pid 27203 exit signal Segmentation fault (11)
[Thu Oct 11 05:35:20 2012] [notice] child pid 27195 exit signal Segmentation fault (11)
[Thu Oct 11 05:35:20 2012] [notice] child pid 27205 exit signal Segmentation fault (11)
[Thu Oct 11 05:35:28 2012] [notice] child pid 27194 exit signal Segmentation fault (11)
[Thu Oct 11 05:35:28 2012] [notice] child pid 27206 exit signal Segmentation fault (11)
[Thu Oct 11 05:35:35 2012] [notice] child pid 27193 exit signal Segmentation fault (11)
[Thu Oct 11 05:35:38 2012] [notice] child pid 27204 exit signal Segmentation fault (11)
[Thu Oct 11 05:35:38 2012] [notice] child pid 27192 exit signal Segmentation fault (11)
[Thu Oct 11 05:35:40 2012] [notice] child pid 27202 exit signal Segmentation fault (11)
[Thu Oct 11 05:35:42 2012] [notice] child pid 27197 exit signal Segmentation fault (11)
[Thu Oct 11 05:35:43 2012] [notice] child pid 27198 exit signal Segmentation fault (11)
[Thu Oct 11 05:37:31 2012] [notice] child pid 27207 exit signal Segmentation fault (11)
[Thu Oct 11 05:39:02 2012] [notice] child pid 9225 exit signal Segmentation fault (11)
[Thu Oct 11 05:39:02 2012] [notice] child pid 9224 exit signal Segmentation fault (11)
[Thu Oct 11 05:39:03 2012] [notice] child pid 9229 exit signal Segmentation fault (11)
[Thu Oct 11 05:39:04 2012] [notice] child pid 9230 exit signal Segmentation fault (11)
[Thu Oct 11 05:39:05 2012] [notice] child pid 9231 exit signal Segmentation fault (11)
[Thu Oct 11 05:39:06 2012] [notice] child pid 9392 exit signal Segmentation fault (11)
[Thu Oct 11 05:39:07 2012] [notice] child pid 9395 exit signal Segmentation fault (11)
[Thu Oct 11 05:39:08 2012] [notice] child pid 9396 exit signal Segmentation fault (11)
[Thu Oct 11 05:39:09 2012] [notice] child pid 27201 exit signal Segmentation fault (11)
[Thu Oct 11 05:41:55 2012] [notice] child pid 9220 exit signal Segmentation fault (11)
[Thu Oct 11 05:45:25 2012] [notice] child pid 9223 exit signal Segmentation fault (11)
[Thu Oct 11 05:52:40 2012] [notice] child pid 9517 exit signal Segmentation fault (11)
[Thu Oct 11 05:54:16 2012] [notice] child pid 9525 exit signal Segmentation fault (11)
[Thu Oct 11 05:54:17 2012] [notice] child pid 9529 exit signal Segmentation fault (11)
[Thu Oct 11 05:54:18 2012] [notice] child pid 9531 exit signal Segmentation fault (11)
[Thu Oct 11 05:54:19 2012] [notice] child pid 10414 exit signal Segmentation fault (11)
/home/john/dips/apps/production/releases/20121005650832/vendor/gems/ruby/1.9/gems/needle-1.3.0/lib/needle/definition-context.rb:36: warning: undefining `initialize' may cause serious problems
/home/john/dips/apps/production/releases/20121005650832/vendor/gems/ruby/1.9/gems/needle-1.3.0/lib/needle/definition-context.rb:36: warning: undefining `object_id' may cause serious problems
/home/john/apps/production/releases/20121005650832/vendor/gems/ruby/1.9/gems/needle-1.3.0/lib/needle/definition-context.rb:36: warning: undefining `__send__' may cause serious problems
[DEPRECATED] By requiring 'spreadsheet/excel' you are loading a Compatibility
layer which provides a drop-in replacement for Spreadsheet::Excel
versions <= 0.3.5.1. This code will be removed in Spreadsheet
version 1.0.0
/home/john/dips/apps/production/releases/20121005650832/vendor/gems/ruby/1.9/gems/needle-1.3.0/lib/needle/definition-context.rb:36: warning: undefining `initialize' may cause serious problems
/home/john/dips/apps/production/releases/20121005650832/vendor/gems/ruby/1.9/gems/needle-1.3.0/lib/needle/definition-context.rb:36: warning: undefining `object_id' may cause serious problems
/home/john/dips/apps/production/releases/20121005650832/vendor/gems/ruby/1.9/gems/needle-1.3.0/lib/needle/definition-context.rb:36: warning: undefining `__send__' may cause serious problems
[DEPRECATED] By requiring 'spreadsheet/excel' you are loading a Compatibility
layer which provides a drop-in replacement for Spreadsheet::Excel
versions <= 0.3.5.1. This code will be removed in Spreadsheet
version 1.0.0
enter code here
I have developed a sample iPad application in which I open the one file of type ppt in UIWebView. When I run this application on iPad having O.S version 5.0(iPad2) & 4.3.1(iPad1) the application gets crash with following error(Memory Warning):
Apr 11 11:01:23 unknown WebViewTest[345] <Warning>: Received memory warning.
Apr 11 11:01:24 unknown UserEventAgent[12] <Notice>: jetsam: kernel termination snapshot being created
Apr 11 11:01:24 unknown com.apple.launchd[1] <Notice>: (UIKitApplication:com.apple.Preferences[0x3072]) Exited: Killed: 9
Apr 11 11:01:24 unknown com.apple.launchd[1] <Notice>: (UIKitApplication:com.extentia.DMG[0x6ba7]) Exited: Killed: 9
Apr 11 11:01:24 unknown SpringBoard[15] <Warning>: Application 'Settings' exited abnormally with signal 9: Killed: 9
Apr 11 11:01:24 unknown SpringBoard[15] <Warning>: Application 'DMG' exited abnormally with signal 9: Killed: 9
Apr 11 11:01:25 unknown ReportCrash[347] <Error>: Saved crashreport to /Library/Logs/CrashReporter/LowMemory-2012-04-11-110125.plist using uid: 0 gid: 0, synthetic_euid: 0 egid: 0
Apr 11 11:01:25 unknown WebViewTest[345] <Warning>: Received memory warning.
Apr 11 11:01:26 unknown SpringBoard[15] <Warning>: Received memory warning.
Apr 11 11:01:26 unknown com.apple.launchd[1] <Notice>: (UIKitApplication:com.extentia.WebViewTest[0x99f4]) Exited: Killed: 9
Apr 11 11:01:26 unknown UserEventAgent[12] <Notice>: jetsam: kernel termination snapshot being created
Apr 11 11:01:26 unknown SpringBoard[15] <Warning>: Application 'WebViewTest' exited abnormally with signal 9: Killed: 9
Apr 11 11:01:35 unknown ReportCrash[348] <Error>: Saved crashreport to /Library/Logs/CrashReporter/LowMemory-2012-04-11-110135.plist using uid: 0 gid: 0, synthetic_euid: 0 egid: 0
But when I run same application on iPad having O.S version 4.3.3(iPad2) it runs perfectly. What is the reason behind this strange behavior & any solution to solve this. Thanks in advance.
your app is crashing because it ran out of memory.
Try using PDF format or PPS (power point slide show) format. That may take some lesser space.
I'm doing a app like photo browser just like the "Photo" on iTOuch. And I use a third-party lib : https://github.com/kirbyt/KTPhotoBrowser
When I scroll the photos several times, the app crashes.
The log shows that it has receive memory warning.
Then I use instrument to check the memory (I choose Memory-Allocation).
I track the Live Bytes and Overall Bytes for "All Allocations".
However, the Live Bytes is never more than 5 MB, and the Overall Bytes is beyond 100 MB .
When the app carshes, the Live Bytes is 3.97MB and the Overall is 125.48MB.
I think Live Bytes is the memory currently used, and it seems it costs a little.
Why the app still crashes?
Xcode 4.2 4D199,Lion 10.7.2(11c74), iTouch ios5.0.1 beta2.
Below is the screenshot, and the instrument configuration is the default.
Any idea?
the crash log is like:
Nov 8 16:54:42 unknown SpringBoard[15] : Received memory warning.
Nov 8 16:54:43 unknown com.apple.launchd[1] : (UIKitApplication:com.apple.mobilephone[0xcb31]) Exited: Killed: 9
Nov 8 16:54:43 unknown com.apple.launchd[1] : (UIKitApplication:com.freebit.minimumcloud.ios[0xf787]) Exited: Killed: 9
Nov 8 16:54:43 unknown UserEventAgent[12] : jetsam: kernel termination snapshot being created
Nov 8 16:54:43 unknown SpringBoard[15] : Application 'Min.Cloud' exited abnormally with signal 9: Killed: 9
Nov 8 16:54:44 unknown SpringBoard[15] : Application 'FaceTime' exited abnormally with signal 9: Killed: 9
Nov 8 16:54:45 unknown kernel[0] : launchd[481] Builtin profile: container (sandbox)
Nov 8 16:54:45 unknown kernel[0] : launchd[481] Container: /private/var/mobile/Applications/301E2A00-F1B6-4E88-ACF4-8EBEFB92B9DA [69] (sandbox)
Nov 8 16:54:45 unknown librariand[459] : client connection is invalid: Connection invalid
Nov 8 16:54:45 unknown ReportCrash[480] : Saved crashreport to /Library/Logs/CrashReporter/LowMemory-2011-11-08-165445.plist using uid: 0 gid: 0, synthetic_euid: 0 egid: 0
Nov 8 16:54:49 unknown sandboxd[483] : MinimumCloud(481) deny file-write-data /private/var/mobile/Library/Mobile Documents/*********
Nov 8 16:54:50 unknown MinimumCloud[481] : file://localhost/private/var/mobile/Library/Mobile%20Documents/*******/Documents/
Nov 8 16:54:50 unknown sandboxd[483] : MinimumCloud(481) deny file-write-create /file:
Nov 8 16:54:50 unknown MinimumCloud[481] : ubiquityContainerURL=/private/var/mobile/Library/Mobile Documents/******/Documents
Nov 8 16:54:50 unknown sandboxd[483] : MinimumCloud(481) deny file-write-create /file:
Nov 8 16:54:50 unknown MinimumCloud[481] : mystorageSymLink=/var/mobile/Applications/301E2A00-F1B6-4E88-ACF4-8EBEFB92B9DA/Library/htdocs/MyStorage
Nov 8 16:54:51 unknown wifid[27] : WiFi:[342435291.518957]: Disable WoW requested by "spd"
It is my own fault!
when I use NSURLConnection to detect the MIME-TYPE of the file, the memory of the connection is never released and it becomes larger and larger(As I detect the type of more than 100 files).
I reconstruct the code.