As we know mediaserverd daemon process can cause app foreground OOM and get killed, is there way to get how many memory used by these daemon process or get other daemon process will cause app foreground OOM?
Related
I am running an opencpu based image on openshift, every time the pod starts, after just a few seconds, it crashes with the error:
command terminated with non-zero exit code: Error executing in Docker Container: 137
Event tab shows only below three events and terminal logs does not show anything as well.
Back-off restarting the failed container
Pod sandbox changed, it will be killed and re-created.
Killing container with id docker://opencpu-test-temp:Need to kill Pod
I am really not getting any clue on why container gets restarted in every few seconds. This image runs just fine locally.
Does anyone give me a clue on how to debug this issue ?
Error 137 is often memory related in a docker context.
The actual error is from the process that is isolated in the docker container. It means that the process could not be killed with a SIGKILL. Source
From bobcares.com:
Error 137 in Docker denotes that the container was ‘KILL’ed by
‘oom-killer’ (Out of Memory). This happens when there isn’t enough
memory in the container for running the process.
‘OOM killer’ is a proactive process that jumps in to save the system
when its memory level goes too low, by killing the resource-abusive
processes to free up memory for the system.
Try checking your memory config of the container? And available memory on the host that is launching the pod? Is there nothing the the opencpu container log?
Check the seting rlimit.as in the config file /etc/opencpu/server.conf, inside the image. This limit is the "per request" memory limit for your opencpu instance (I realize that your problem is at startup, so this is perhaps not too likely to be the case).
When stopping a Docker container in native Docker environment, by default it sends the SIGTERM signal to the container's init process (PID 1) which should be the actual application, which should then handle the shutdown properly. However when running container in the Jelastic, this does not seem to be case, and instead of gracefully terminating the SQL server, it seems that the server crashes every time.
I did try writing and enabling a Systemd service that gets the SQL PID and then send SIGTERM to it, but it doesn't seem to run, and judging from the logs there's no service shutdown messages at all, just startup messages.
So what changes would be required to the container or the environment to get the server to get the SIGTERM signal and have enough time, maybe few seconds, to do the graceful shutdown?
thank you for reporting the issue, we tried to reproduce the problem on our test lab and were able to get exactly same result. We agree that issue is really serious so we are going to fix it with highest priority now. Please accept our apologies for that inconvenience. I want to notice that due to our primary design we also expect the process to be terminated first with "sigterm" signal and only after not receiving a termination result for some period of time the system had to send "sigkill", only after considering that process cannot be terminated gracefully. Our engineers will work on this to explore the issue deeper and will deliver a fix shortly.
Thank you!
I want to start monit process in docker.
Since its getting daemonized, the container is getting completed once monit starts. What is the best way to run it as a foreground process?
From https://wiki.gentoo.org/wiki/Monit:
Running monit in the foreground
To run monit in the foreground and provide feedback on everything it is detecting, use the -Ivv option:
root #monit -Ivv
I have got strange errors
2016-07-15T14:34:09.334Z 16484 TID-33ld0 WARN: Terminating 1 busy worker threads
2016-07-15T14:34:09.334Z 16484 TID-33ld0 WARN: Work still in progress [#<struct Sidekiq::BasicFetch::UnitOfWork queue="queue:load_xml", job="{\"class\":\"GuitarmaniaWorker\",\"args\":[],\"retry\":false,\"queue\":\"load_xml\",\"jid\":\"56c01b371c3ee077c2ccf440\",\"created_at\":1468590072.35382,\"enqueued_at\":1468590072.3539252}">]
2016-07-15T14:34:09.334Z 16484 TID-33ld0 DEBUG: Re-queueing terminated jobs
2016-07-15T14:34:09.335Z 16484 TID-33ld0 INFO: Pushed 1 jobs back to Redis
2016-07-15T14:34:09.336Z 16484 TID-33ld0 INFO: Bye!
What can cause it?
Locally all work great but after deployment on production server this errors appeared.
Any suggestions.
This means the Sidekiq is being shut down. In the event of a normal "kill" operation (TERM signal), Sidekiq server will attempt to shutdown gracefully by waiting 8 seconds for jobs to be complete. It will then stop the job execution and re-queue for the next time the server starts.
That begs the question, why is your Sidekiq shutting down. Possible reasons are: you on command-line or a script killed the process; you or your data host shut down the machine; your OS ran out of memory. The last reason is the most likely if you're not sure of the cause.
If the memory leak occurs soon after startup, you may be running too many Sidekiq processes and/or your machine may not have enough memory to load the app. If the message happens after some time, it's possible you have a memory leak - run free periodically after starting Sidekiq to see if memory usage scales up, which would indicate a leak. Sometimes a memory leak is due to a library, sometimes it's your own application. More about tracking down leaks in Ruby here.
Is it possible to run a script through a launch daemon for an indefinite amount of time on jailbroken iOS 9? Would iOS 9 eventually kill a launch daemon that runs a script indefinitely, or would it just let the script keep running? Would a launch daemon be a feasible way of running said script on an iPhone?
Launchd doesn't do anything special if you didn't ask it to. It will parse your plist, launch the binary and that's it. The daemon can run for as long as it wants. You can check out Cydia auto-install script at /Library/LaunchDaemons/com.saurik.Cydia.Startup.plist. Using that plist as a reference you can launch your script that will run indefinitely. Launchd will not do anything to it.
There're other components that can kill your process but there're ways to prevent that. For example, if memory starts running low the kernel will start killing processes and your daemon might be killed as well. That kernel component is called jetsam. All processes have some jetsam priority and memory limit associated with them and depending on that they will or will not be killed when memory runs low. You can read about that here. You can also just tell launchd to relaunch your process automatically if that fits your case.