I am hosting my Rails app on Ubuntu 12.04 VPS, Nginx + Unicorn, after deployment everything is fine, but few hours later, when I ssh to the VPS I get this message
-bash: fork: Cannot allocate memory
-bash: wait_for: No record of process 4201
-bash: wait_for: No record of process 4201
If I run any command, it would just return
-bash: fork: Cannot allocate memory.
Seems you have run out of memory. Many VPS servers are setup with no swap, so when you run out of memory, it will kill things off in a seemingly random manner.
The easiest way to fix it is to get more memory provisioned to your VPS, likely costing more money. The next best way (other than running less stuff and memory optimizing everything running) would be to add a swap partition or swap file.
For a 1GB swap file (as root):
dd if=/dev/zero of=/swapfile bs=1M count=1024
mkswap /swapfile
swapon /swapfile
Be sure to add it to /etc/fstab too as:
/swapfile none swap defaults 0 0
That will make it come back after reboot.
To get out of this condition without rebooting, you can trigger the OOM killer manually as follows:
echo 1 > /proc/sys/kernel/sysrq
echo f > /proc/sysrq-trigger
echo 0 > /proc/sys/kernel/sysrq
Having done so you can inspect dmesg to find the process responsible for taking all your memory.
Related
I have a Ruby on Rails web application that allow users to upload images which then automatically get resized as small thumbnails using libvips and the ImageProcessing ruby gem. Sometimes users legitimately need to upload 100MP+ images. These large images break our server that only has 1GB of RAM. If it's relevant, these images are almost always JPEGs.
What I'm hoping is to use libvips to first scale down these images to a size that my server can handle--maybe like under 8,000x8,000 pixels--without using lots of RAM. Then I would use that image to do the other things we already do, like change the colorspace to sRGB and resize and strip metadata, etc.
Is this possible? If so can you give an example of a vips or vipsthumbnail linux CLI command?
I found a feature in Imagemagick that should theoretically solve this issue, mentioned in the two links below. But I don't want to have to switch the whole system to Imagemagick just for this.
https://legacy.imagemagick.org/Usage/formats/#jpg_read
https://github.com/janko/image_processing/wiki/Improving-ImageMagick-performance
P.S.: I'm using Heroku so if the RAM usage peaks at up to 2GB the action should still work.
(I've always been confused about why image processing seems to always require loading the entire image in RAM at once...)
UPDATE:
I'm providing more context because jcupitt's command is still failing for me.
This is the main software that is installed on the Docker container that is running libvips, as defined in the Dockerfile:
FROM ruby:3.1.2
RUN apt-get update -qq && apt-get install -y postgresql-client
# uglifier requires nodejs -- `apt-get install nodejs` only installs older version by default
RUN apt-get install -y curl
RUN curl -sL https://deb.nodesource.com/setup_14.x | bash -
RUN apt-get install -y nodejs
RUN apt-get install -y libvips libvips-dev libvips-tools
# install pdftotext
RUN apt-get install -y xpdf
I am limiting the memory usage of the sidekiq container to 500MB to be more similar to production server. (I also tried this when limiting memory and reserved memory to 1GB and the same thing happens.) This is the config as specified in docker-compose.yml
sidekiq:
depends_on:
- db
- redis
build: .
command: sidekiq -c 1 -v -q mailers -q default -q low -q searchkick
volumes:
- '.:/myapp'
env_file:
- '.env'
deploy:
resources:
limits:
memory: 500M
reservations:
memory: 500M
This is the exact command I'm trying, based on the command that jcupitt suggested:
first I run docker stats --all to see the sidekiq container's memory usage after booting up, not running libvips:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
4d7e9ff9c7c7 sidekiq_1 0.48% 210.2MiB / 500MiB 42.03% 282kB / 635kB 133MB / 0B 7
I also check docker-compose exec sidekiq top and get a higher RAM limit, which I think is normal for Docker
top - 18:39:48 up 1 day, 3:21, 0 users, load average: 0.01, 0.08, 0.21
Tasks: 3 total, 1 running, 2 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.2 us, 1.5 sy, 0.0 ni, 97.1 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 3929.7 total, 267.4 free, 1844.1 used, 1818.1 buff/cache
MiB Swap: 980.0 total, 61.7 free, 918.3 used. 1756.6 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 607688 190620 12848 S 0.3 4.7 0:10.31 ruby
54 root 20 0 6984 3260 2772 R 0.3 0.1 0:00.05 top
39 root 20 0 4092 3256 2732 S 0.0 0.1 0:00.03 bash
then I run the command
docker-compose exec sidekiq bash
root#4d7e9ff9c7c7:/myapp# vipsheader /tmp/shrine20220728-1-8yqju5.jpeg
/tmp/shrine20220728-1-8yqju5.jpeg: 23400x15600 uchar, 3 bands, srgb, jpegload
VIPS_CONCURRENCY=1 vipsthumbnail /tmp/shrine20220728-1-8yqju5.jpeg --size 500x500
Then in another Terminal window I check docker stats --all again
In maybe 0.5s the memory usage quickly shoots to 500MB and the vipsthumbnail process dies and just returns "Killed".
libvips will almost always stream images rather than loading them in memory, so you should not see high memory use.
For example:
$ vipsheader st-francis.jpg
st-francis.jpg: 30000x26319 uchar, 3 bands, srgb, jpegload
$ ls -l st-francis.jpg
-rw-rw-r-- 1 john john 227612475 Sep 17 2020 st-francis.jpg
$ /usr/bin/time -f %M:%e vipsthumbnail st-francis.jpg --size 500x500
87412:2.57
So 87MB of memory and 2.5s. The image is around 3gb uncompressed. You should get the same performance with ActiveRecord.
In fact there's not much useful concurrency for this sort of operation, so you can run libvips with a small threadpool.
$ VIPS_CONCURRENCY=1 /usr/bin/time -f %M:%e vipsthumbnail st-francis.jpg --size 500x500
52624:2.49
So with one thread in the threadpool it's about the same speed, but memory use is down to 50MB.
There are a few cases when this will fail. One is with interlaced (also called progressive) images.
These represent the image as a series of passes of increasingly higher detail. This can help when displaying an image to the user (the image appears in slowly increasing detail, rather than as a line moving down the screen), but unfortunately this also means that you don't get the final value of the first pixel until the entire image has been decompressed. This means you have to decompress the whole image into memory, and makes this type of file extremely unsuitable for large images of the sort you are handling.
You can detect an interlaced image in ruby-vips with:
if image.get_typeof("interlaced") != 0
error "argh! can't handle this"
end
I would do that test early on in your application and block upload of this type of file.
I am using redis-benchmark on Ubuntu by using docker container.
I am experimenting performance of memory and CPU by limiting resources by Docker.
For example:
docker run --cpus="2" -m=4M -it [imageID]
My image is based on ubntu:latest and installed redis-server.
So, when I get into a container I run redis-server on background and use redis-benchmark:
10000 changes in 60 seconds. Saving...
Background saving started by pid 17
DB saved on disk
RDB: 0MB of memory used by copy-on-write
Background saving terminated with success
Error: Server closed the connection
[1]+ Killed redis-server
An error shows up.
My redis-benchmark command to make key/value size 100000:
redis-benchmark -q -r 100000 -c 1
If this is because of limitation of memory size, I think this command:
redis-benchamrk -c 1 -r 100000 -n 400
should also shut down the redis-server.
I'm using kubernetes on my cluster with several rails / node docker images. Most of them have :3000/healtz health check that simply returns status 200 with OK in body.
Now I'm trying to discover the best way how this health check can be performed on docker image running sidekiq. How I can verify that the worker is running?
If your image is unix like, you can check if the proccess is running with
$ ps aux | grep '[s]idekiq'
But this don't guarantee that everything is working inside sidekiq and redis.
A better approach is described/developed in this sidekiq plugin https://github.com/arturictus/sidekiq_alive
I'm facing problems with livenessProbe for k8s and trying to solve without using this lib but not successful yet.
Sidekiq 6.0 ships with a new sidekiqmon, which you could use to validate that a process is running on your current machine with redis:
REDIS_URL=redis://redis.example.com:6380/5 sidekiqmon | grep $$(hostname)
Documentation: https://github.com/mperham/sidekiq/wiki/Monitoring#sidekiqmon
You have different approaches depending mostly on how you deployed Sidekiq inside the Kubernetes pod.
Here some of them.
SIDEKIQMON
Since Sidekiq v6 we have sidekiqmon
You can try with something like:
bundle exec sidekiqmon processes | grep $(hostname)
To make it work you must be able to grep the correct running process related to current pod/container.
PS AUX (I used in the past)
Something like this could work:
ps aux | grep '[s]idekiq 6'
Note the 6 (sidekiq version) at the end to differentiate from the other process that you could have.
Anyway here the idea is to just check your processes with ps aux and try to "grep" the sidekiq one you need to monitor.
SIDEKIQ_ALIVE
Another approach is to use sidekiq_alive
But it requires more effort to be configured and managed and I never really tried it. This is the only one that really check the full functionality with also the Redis status.
SYSTEMCTL
If you deployed sidekiq using systemd you can just do:
systemctl status sidekiq | grep '[r]unning'"
So at the end my YAML manifest for Sidekiq deployment looks like this:
livenessProbe:
exec:
command: ["/bin/bash", "-l", "-c", "bundle exec sidekiqmon processes | grep $(hostname)"]
initialDelaySeconds: 120
periodSeconds: 30
successThreshold: 1
failureThreshold: 3
timeoutSeconds: 30
Note that only the sidekiq_alive approach is going to perform a full end to end test, checking also the redis status. Maybe this is too much, you could have other monitoring system for Redis so maybe here, with the kubernetes LivenessProbes, you don't need to check the whole flow but just check if the sidekiq process is still alive or crashed due to a memory leak. This is up to you, based on your needs.
"Why you need this kind of LivenessProbes?"
Because if you are launching Sidekiq via systemd or similar, you will have the main process (PID 1) as /usr/bin/python3 /usr/bin/systemctl start sidekiq and the related process launched with ExecStart as sidekiq 6.1.3 your_app_name [0 of 5 busy]. If you have a memory leak, the last one will crash and not the PID 1.
For this reason you need a sort of livenessProbes for Sidekiq.
Instead if you are not using systemd and your PID 1 is sidekiq 6.1.3 your_app_name [0 of 5 busy] because you are launching it with bundle exec sidekiq you don't need it.
I would like to run a docker container that requires a lot of memory on a machine that doesn't have much RAM. I have been trying to increase the swap space available for the container to no avail. Here is the last command I tried:
docker run -d -m 1000M --memory-swap=10000M --name=my_container my_image
Following these tips on how to check memory metrics I found the following:
$ boot2docker ssh
docker#boot2docker:~$ cat /sys/fs/cgroup/memory/docker/35af5a072751c7af80ce7a255a01ab3c14b3ee0e3f15341f7bb22a777091c67b/memory.stat
cache 454656
rss 65015808
rss_huge 29360128
mapped_file 208896
writeback 0
swap 0
pgpgin 31532
pgpgout 22702
pgfault 49372
pgmajfault 0
inactive_anon 28672
active_anon 65183744
inactive_file 241664
active_file 16384
unevictable 0
hierarchical_memory_limit 1048576000
hierarchical_memsw_limit 10485760000
total_cache 454656
total_rss 65015808
total_rss_huge 29360128
total_mapped_file 208896
total_writeback 0
total_swap 0
total_pgpgin 31532
total_pgpgout 22702
total_pgfault 49372
total_pgmajfault 0
total_inactive_anon 28672
total_active_anon 65183744
total_inactive_file 241664
total_active_file 16384
total_unevictable 0
Is it possible to run a container that requires 5G of memory on a machine that only has 4G of physical memory?
This GitHub issue was very helpful in figuring out how to increase the swap space available in the boot2docker-vm. Adapting it to my situation I used the following commands to ssh into the boot2docker-vm and set up a new swapfile:
boot2docker ssh
export SWAPFILE=/mnt/sda1/swapfile
sudo dd if=/dev/zero of=$SWAPFILE bs=1024 count=4194304
sudo mkswap $SWAPFILE
sudo chmod 600 $SWAPFILE
sudo swapon $SWAPFILE
exit
I am trying to run ruby on rails under passenger with apache2 under fedora 19 and I got this error in log:
[Tue Feb 25 09:37:52.367683 2014] [passenger:error] [pid 2779] ***
Passenger could not be initialized because of this error: Unable to
start the Phusion Passenger watchdog because it encountered the
following error during startup: Cannot change the directory
'/tmp/passenger.1.0.2779/generation-1/buffered_uploads' its UID to 48
and GID to 48: Operation not permitted (errno=1)
That directory (/tmp/passenger.1.0.2779) doesn't even exist. I think that problem is with selinux. I tried to solve it about 4 hours. Httpd is running under user apache and group apache, I tried:
cat /var/log/audit/audit.log | grep passenger | audit2allow -M
passenger semodule -i passenger.pp
but still nothing.
In your case, you should switch SELinux into Permissive mode at first, then try to capture the audit log from starting Apache to run your application.1
Once you got the home page of your application, you can build your custom policy with the logs.
Switch SELinux into Permissive mode and clean audit.log
]# setenforce 0
]# rm /var/log/audit/audit.log
]# service auditd restart
Restart Apache
]# service httpd restart
Try to open your application with a web browser
It might give more information about what is happenning when you application is running.
Make a custom policy module to allow these actions
]# mkdir work
]# cd work
]# grep httpd /var/log/audit/audit.log | audit2allow -M passenger
]# ls
passenger.pp passenger.te
Load postgrey policy module using the 'semodule' command into the current SELinux policy:
]# semodule -i passenger.pp
]# setenforce 1
Restart Apache
]# service httpd restart
References:
http://wiki.centos.org/HowTos/SELinux#head-faa96b3fdd922004cdb988c1989e56191c257c01
I ran into a similar error, with a startup error about being unable to create a directory that did not exist. (logs, not tmp, but same sort of thing) I, too, battled with it for an hour and couldn't make sense of it. I created/deleted/chmod the directory many ways without success.
The fix for me was to change the parameters to passenger-start. Initially, my Docker container started passenger with:
exec bundle exec passenger start --auto --disable-security-update-check --min-instances 20 --max-pool-size 20 --max-request-queue-size 500
I removed all parameters, leaving just this:
exec bundle exec passenger start
At this point, passenger could create the log folder and file, and all was well. I could have restored the params at this point, but we decided they were not needed for the development environment so left them out moving ahead.
In hindsight, I have a hunch that I deleted the log directory while a file in it was still open, and the file system persisted that condition in some way. But that's just a hunch. Perhaps simply rebooting my Mac would have fixed it...