Can someone describe me how sidekiq workers relate to pids in OS?
I have ubuntu 14.04 on aws and I configured my QA environment in sidekiq.yml to use 2 workers. As it shown here.
But what I see in my OS is next:
So 2 workers equals 6 processes. But is correct. How do I limit processes for sidekiq? Or what should I do to reduce memory usage.
Sometimes my server runs out of memory. So shutdown occurs.
Any help appreciated.
Those are threads in the htop listing, not processes.
Related
I ran htop in my production server to see what was eating my RAM. A lot of sidekiq process is running, is this normal?
Press Shift-H. htop shows individual threads as processes by default. There is only one actual sidekiq process.
You seem to have configured it to have 25 workers.
By default, one sidekiq process creates 25 threads.
If that's crushing your machine with I/O, you can adjust it down:
sidekiq -c 10
https://github.com/mperham/sidekiq/wiki/Advanced-Options
If you are not using JRuby then it's likely these all are seperate processes that consume memory.
I'm running a rails app (dev server) with passenger in Amazon AWS with t2.micro instance. But i'm getting -bash: fork: Cannot allocate memory error constantly.
I'm running redis server on it with 50 sidekiq concurrency. Normally sites runs fine but when i start 2-3 sidekiq process simultaneously do do some batch process. The site take take time to redirect and eventylly crash with
502 Bad Gateway
nginx/1.10.0
Then i have to nginx restart every to to get the site running again. This is my dev server so i don't want to put more financial resources for upgrading to t2.small (as of now, this is our last option) as this is dev servre and will be using twice in 15 days. Is there any way i can solve this otherwise? Previously i had same 120 concurrency as production but then i changed to 50. That help a bit but still memory problems.
here are few stats with htop
This stats are while the server is idle. But when i run few task with sidekiq it crashing with 502.
I check few post suggesting swap memory but not sure this is preferable with t2.micro. Is this advisable for this server setup. Here in pic you can see i don't swap memory. Is it okay yo add swap memory to tackle this issue or there is any other better option.
Your server has a lack of memory, to fix it:
or: buy more operative memory
or: mount a swap
Then try again
In my case, redis used 2.5G memory, the server has 4.5G in total, and 3G is used.
1.5G free.
and Redis kept throwing this error.
solution:
add vm.overcommit_memory=1 to file: /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf
refer to: redis bgsave failed because fork Cannot allocate memory
I'm getting my slave processes killed when running more than 300 clients per node.
The machines are hosted in Rackspace and the configuration of each one is:
512MB RAM
1 CPU
Should I increase the memory of the machines?
It's totally dependent on the code of your test scripts. Generally, I would say that it's more common to be limited by CPU before RAM, however in your case it sounds like it might be the Linux OOM Killer killing the processes. The reason for this could be that you have a memory leak in the test code, but it's really hard to tell without more info.
Also, on machines with only a single CPU core, you shouldn't run more than one Locust slave process.
I need to batch process a large set of files (millions of database records) as quickly as possible. For this purpose, I split the files into 3 directories and set up Sidekiq with the standard configuration (no config file).
I then started 3 Heroku workers and called 3 methods, which started 3 Sidekiq workers, all with the "default" queue. Initially, Sidekiq used 2 Heroku workers and after a while it decided to use only 1 worker.
How can I force Sidekiq to use all 3 workers to get the job done asap?
Thanks
I found the solution at the bottom of this page: http://manuelvanrijn.nl/blog/2012/11/13/sidekiq-on-heroku-with-redistogo-nano/
# app/config/sidekiq.yml
:concurrency: 1
# Procfile
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec sidekiq -e production -C config/sidekiq.yml
Also, if you have many workers and a free / cheap Redis instance, make sure you limit the number of connections from each worker to the Redis server:
# app/config/initializers/sidekiq.rb
require 'sidekiq'
Sidekiq.configure_client do |config|
config.redis = { :size => 1 }
end
Sidekiq.configure_server do |config|
config.redis = { :size => 2 }
end
You can calculate the maximum of connections here: http://manuelvanrijn.nl/sidekiq-heroku-redis-calc/
I wanted to clarify a few things about your question. Your question reads "Tell Sidekiq to use all available Heroku workers". In fact, for each Dyno, a sidekiq process will execute using a command like bundle exec sidekiq -e production -C config/sidekiq.yml. Each one of these Sidekiq processes can handle multiple threads as specified in the config/sidekiq.yml file with a line like: :concurrency: 3 which is what the Sidekiq docs recommend for a Heroku standard-2x dyno (read here for more details https://github.com/mperham/sidekiq/wiki/Heroku), which only has 1gb of memory.
But technically you don't need to tell Sidekiq to use all available Heroku processes. There is another key element of this, which is the Redis server. Our main app will publish messages to a Redis server. Each Sidekiq process running on a given Dyno can be configured with the same Queue and thus all are subscribed to that Redis queue and pull the messages from the Queue. This is clearly stated by the creator of Sidekiq from the Sidekiq github page: https://github.com/mperham/sidekiq/issues/3603.
There are a couple of key points to share the load. First restrict the concurrency of a given Sidekiq process to a number like I mentioned above. Second, you can also limit the connections to the Redis server from within Sidekiq.configure_client. Finally, think of the Heroku load balancing somewhat different from how ALB in AWS works. ALB is a round robin that distributes traffic to instances in Target Groups based on certain metrics defined in Launch Templates and Auto Scaling Groups, such as vCPU utilization, memory utilization and read/write io. Rather the load balancing here is more like a publish-subscribe system where Sidekiq instances take work when they are able to and based on their restrictions both on concurrency and connection limits to the Redis server.
Finally, Heroku discourages long running jobs. The longer your job runs the higher the amount of memory it will eat up. Heroku dynos are expensive. The standard-2x is 4x the cost of a t3.micro in AWS for the same vCPU and Memory (1gb). Furthermore, in AWS you can create a spot fleet, where you purchase compute for 10 percent of the cost of its on-demand price and then execute these spot instances as batch jobs. In fact, AWS also has a service called AWS Batch. The spot fleet option doesn't exist in Heroku. Therefore, it's important to keep price in mind and thus how long the job runs. Read this article here where Heroku delineates why it is bad running long running jobs in Heroku environment: https://devcenter.heroku.com/articles/scaling#understanding-concurrency. Try to keep a job under 2 minutes.
We're running 3 Apache Passenger servers sharing the same file system, each running 11 Rails applications.
We've set
PassengerPoolIdleTime = 0 to ensure none of the applications ever die out completely, and
PassengerMaxPoolSize = 20 to ensure we have enough processes to run all the applications.
The problem is that when I run passenger-memory-stats on one of the servers I see 210 VM's!
And when I run passenger-status I see 20 application instances (as expected)!
Anyone know what's going on? How do I figure out which of those 210 instances are still being used, and how do I kill those on a regular basis? Would PassengerMaxInstancesPerApp do anything to reduce those seemingly orphaned instances?
Turns out we actually have that many Apache worker processes running, and only 24 of them are Passenger processes (asked someone a little more experienced than myself). We are actually hosting many more websites and shared hosting accounts than I thought.
Thanks for all the replies though!
You can get a definitive answer as to how many Rails processes you have by running this command:
ps auxw | grep Rails | wc -l
I doubt you really do have over 100 processes running though, as at about 50 mb each they'd collectively be consuming over 5 gb of RAM and/or swap and your system would likely have slowed to a crawl.
Not so much an answer, but some useful diagnostic advice.
Try adding the Rack::ProcTitle middleware to your stack. We have been using it in production for years. Running ps aux should give info on what the worker is up to (idle, handing a specific request etc…).
I would not assume that these processes are being spawned directly by passenger. It could be something forking deeper into your app.
Maybe the processes are stuck during shutdown. Try obtaining their backtraces to see what they're doing.