Docker Scrapinghub/splash exited with 139 - docker

I'm using Scrapy to do some crawling with Splash using the Scrapinghub/splash docker container however the container exit after a while by itself with exit code 139, I'm running the scraper on an AWS EC2 instance with 1GB swap assigned.
i also tried to run it in background and view the logs later but nothing indicates an error it just exit.
From what i understand 139 is for Segmentation Fault errors in UNIX, is there anyway to check or log what part of memory being accessed or code being executed to debug this?
Or can i increase the container memory or swap size to avoid this?

Related

Docker Container exited with code 247 when getting data from GCP

I'm running a Flask API inside a Docker Container.
My application has to download files from Google Cloud, and sometimes, after some minutes of execution, my container exits with the following message:
pm-api exited with code 247
I am not sure, but I think it might be related to the size of the data I'm trying to download from GCP, becuse I'm using a query and I don't seem to have any problem whem limiting the number of rows I get from it.
Could be data-size related? And if so, how can I configure my docker container to not break when downloading/saving large files?
In order to solve this problem I had to increase the size of my docker container. This is found in settings resources. Increasing the memory removed this issue.
Yes, possibly due to the large file size.
You may have to define FILE_UPLOAD_MAX_MEMORY_SIZE larger than your downloading file.What is code 247
Also refer
max-memory-vs-data-upload-max-memory
I have a similar issue. docker container (python code processing some data) exited
command terminated with exit code 247
my docker container is running on k8s.
the issue was caused because I set the k8s resources memory limit to 4GB, but the python container needs to use >4GB memory. after I increase the memory limit to 8GB, the issue is sovled.
but about why the exit code is 247, I don't find the answer. from the docker exit code, it doesn't have code 247.

Openshift command terminated with non-zero exit code: Error executing in Docker Container: 137

I am running an opencpu based image on openshift, every time the pod starts, after just a few seconds, it crashes with the error:
command terminated with non-zero exit code: Error executing in Docker Container: 137
Event tab shows only below three events and terminal logs does not show anything as well.
Back-off restarting the failed container
Pod sandbox changed, it will be killed and re-created.
Killing container with id docker://opencpu-test-temp:Need to kill Pod
I am really not getting any clue on why container gets restarted in every few seconds. This image runs just fine locally.
Does anyone give me a clue on how to debug this issue ?
Error 137 is often memory related in a docker context.
The actual error is from the process that is isolated in the docker container. It means that the process could not be killed with a SIGKILL. Source
From bobcares.com:
Error 137 in Docker denotes that the container was ‘KILL’ed by
‘oom-killer’ (Out of Memory). This happens when there isn’t enough
memory in the container for running the process.
‘OOM killer’ is a proactive process that jumps in to save the system
when its memory level goes too low, by killing the resource-abusive
processes to free up memory for the system.
Try checking your memory config of the container? And available memory on the host that is launching the pod? Is there nothing the the opencpu container log?
Check the seting rlimit.as in the config file /etc/opencpu/server.conf, inside the image. This limit is the "per request" memory limit for your opencpu instance (I realize that your problem is at startup, so this is perhaps not too likely to be the case).

Docker Logs filling up on running container

I've got a Docker container currently running in production on a CentOS 7 VM. We have encountered a problem where the logs of the container are filling up the host drive (the log files found at /var/lib/docker/{continer_name}) over time and causing the container to become unresponsive forcing us to clear logs on the host in order to enable it to continue processing.
We can't take the container down, meaning I can't just bring it back up using the --log-opt flag to set up some log rotation options.
We've tried using logrotate, but the nature of the container means the logs are being written to regularly and what we find is often the logs are rotated, but the original file does not decrease in size due to being written to whilst the rotation is underway.
I'm trying to find a solution to this problem where we can set up some kind of task that will clear the logs down to a specific file size. Any help is greatly appreciated.
i would suggest mounting the containers logs directory to a host directory, and there you can schedule whatever task to zip/move/delete the log files...

Docker image fails to create netlink handle

Can anyone help me make sense of the below error and others like it? I've Googled around, but nothing makes sense for my context. I download my Docker Image, but the container refuses to start. The namespace referenced is not always 26, but could be anything from 20-29. I am launching my Docker container onto an EC2 instance and pulling the image from AWS ECR. The error is persistent no matter if I re-launch the instance completely or restart docker.
docker: Error response from daemon: oci runtime error:
container_linux.go:247: starting container process caused
"process_linux.go:334: running prestart hook 0 caused \"error running
hook: exit status 1, stdout: , stderr: time=\\\"2017-05-
11T21:00:18Z\\\" level=fatal msg=\\\"failed to create a netlink handle:
failed to set into network namespace 26 while creating netlink socket:
invalid argument\\\" \\n\"".
Update from my Github issue: https://github.com/moby/moby/issues/33656
It seems like the DeepSecurity agent (ds_agent) running on a container with Docker can cause this issue invariably. A number of other users reported this problem, causing me to investigate. I previously installed ds_agent on these boxes, before replacing it with other software as a business decision, which is when the problem went away. If you are having this problem, might be worthwhile to check if you are running the ds_agent process, or other similar services that could be causing a conflict using 'htop' as the user in the issue above did.
Did you try running it with the --privileged option?
If it still doesn't run, try adding --security-opts seccomp=unconfined and either --security-opts apparmor=unconfined or --security-opts selinux=unconfined depending whether you're running Ubuntu or a distribution with SELinux enabled, respectively.
If it works, try substituting the --privileged option with --cap-add=NET_ADMIN` instead, as running containers in privileged mode is discouraged for security reasons.

Docker container stops without any errors while runnning sbt/play application

I'm running into an issue where my docker container will exit with exit code 137 after ~a day of running. The logs for the container contains no information indicating that an error code has occurred. Additionally, attempts to restart the container returns an error that the PID already exists for the application.
The container is built using the sbt docker plugin, sbt docker:publishLocal and is then run using
docker run --name=the_app --net=the_app_nw -d the_app:1.0-SNAPSHOT.
I'm also running 3 other docker containers which all together do use 90% of the available memory, but its only ever that particular container which exits.
Looking for any advice on to where to look next.
The error code 137 (128+9) means that it was killed (like kill -9 yourApp) by something. That something can be a lot of things (maybe it was killed because was using too much resources by docker or something else, maybe it got out of memory, etc)
Regarding the pid problem, you can add to your build.sbt this
javaOptions in Universal ++= Seq(
"-Dpidfile.path=/dev/null"
)
Basically this should instruct Play to not create a RUNNING_PID file. If it does not work you can try to pass that option directly in Docker using the JAVA_OPTS env variable.

Resources