I am working on uber like/cab booking app. I am using Action Cable for this purpose. After creation of new order server get list of 10 nearest drivers and send each in turn order details (with a pause of 40 seconds).
Thread.new do
nearest_drivers.each do |id|
order_data_for_driver = { ... }
ActionCable.server.broadcast("driver_#{id}", order_data_for_driver)
sleep 40
Thread.exit if order.reload.canceled_by_user || order.trip
end
cancel_data = {canceled_by_timeout: true }
ActionCable.server.broadcast("order_#{order.id}", cancel_data )
end
Is there a limit to the number of threads that rails in production mode can run at the same time? For example, if 100 users will create new orders. What more elegant solution can be used?
Usually this kind of tasks is referred as having back pressure. Maximum amount of threads on UNIX systems may vary from circa 10K to max allowed 500K.
The most common way to handle back pressure is to plug in a fast queue in between (like RabbitMQ or something) and increase the amount of queue consumers as the load of requests to proceed increases.
100/s is nothing, but if you plan to handle thousands of concurrent connections, I strongly encourage to rethink the language of choice twice. Rails is not a software created for this kind of task. Neither is Ruby.
Related
I have model called AisSignal with about 3000 records and I am running each one against another model called Footprint with about 10 records, so we have a loop 3000 x 10.
I tried:
Parallel.each(AisSignal.all, in_processes: 8) do |signal|
Footprint.all.each do |footprint|
if footprint.cover([signal.lon, signal.lat])
signal.update(imo: 'in')
break
end
end
end
but it runs in 10 seconds just like normal block.
I tried to change from processes to threads like below but this causes application freezing.
Parallel.each(AisSignal.all, in_threads: 8) do |signal|
Footprint.all.each do |footprint|
if footprint.cover([signal.lon, signal.lat])
signal.update(imo: 'in')
break
end
end
end
I have 50 pool size in database.yml
Any idea or approach to have multiple threads that run in parallel to update records.
I will need to update more records actually which can take about minutes.
Threads and forks often don't play well with database connections. If not handled correctly the threads/processes can wind up trying to use the same connection at the same time.
Parallel mentions this in their documentation. You need to make use of connection pooling.
A connection pool synchronizes thread access to a limited number of database connections. The basic idea is that each thread checks out a database connection from the pool, uses that connection, and checks the connection back in. ConnectionPool is completely thread-safe, and will ensure that a connection cannot be used by two threads at the same time, as long as ConnectionPool's contract is correctly followed. It will also handle cases in which there are more threads than connections: if all connections have been checked out, and a thread tries to checkout a connection anyway, then ConnectionPool will wait until some other thread has checked in a connection.
Parallel.each(AisSignal.all, in_threads: 8) do |signal|
ActiveRecord::Base.connection_pool.with_connection do
Footprint.all.each do |footprint|
if footprint.cover([signal.lon, signal.lat])
signal.update(imo: 'in')
break
end
end
end
end
Note that this code is very inefficient.
It loads the entire AisSignal table.
For each signal it loads and scans the entire Footprint table.
It will use a lot of memory, and it will run in s*f time where s is the number of signals and f is the number of footprints.
You can reduce the memory footprint by replacing Footprint.all.each with Footprint.find_each. This will load rows in batches.
Threading is not how you make database queries faster. The fundamental problem is you're scanning Footprint multiple times in Ruby rather than letting the database do it. if footprint.cover([signal.lon, signal.lat]) should instead be a where clause.
AisSignal.find_each do |signal|
# With ... being the equivalent of `cover([signal.lon, signal.lat])`
# as a where clause.
signal.update!(imo: 'in') if Footprint.exists?(...)
end
This could be done even faster as a join.
# ... is the equivalent of `cover([signal.lon, signal.lat])`
AisSignal.joins("inner join footprints on ...").update_all(imo: 'in')
I have a Rails app, that runs 50 concurrent Sidekiq workers, that receive data from external APIs, manipulate it and store it in a Database.
In order to prevent to store the same data multiple times, I create unique strings, that I store into Redis.
r = Redis.current
r.pipelined do
options.each do |o|
r.setex(o.generate_index, 1.hour, o.value)
end
end
Before saving a new object to the Database I check if it's unique string exists in Redis:
r = Redis.current
option_ids = r.pipelined do
new_options.each do |o|
r.get(o.generate_index) # check if unique string already exists
end
end
I also tested without pielining with the same result:
Redis GETs/SETs are extremely slow.
Some GET-Request take more that one second!
weird thing: While running 50 Sidekiq workers, that had a bad Redis performance, I benchmarked Redis from the shell, with very good results:
r = Redis.current
10000000.times { puts Benchmark.realtime{ 1000.times{|i| r.get(i)} }}
with output like:
0.197379041
0.192059888
0.196165358
0.207617425
0.198095963
0.195844917
0.211404108
0.203188759
0.208119298
0.184018683
No Benchmark took longer than 0.5 second.
I also tried Redis.new instead of Redis.current with the same result.
I can't figure out what's the problem
EDIT:
After some more testing and benchmarking I noticed that the benchmarks in my code seemed to block other workers and slow them down much more than expected. Without benchmarking the workers are up to 10 times faster.
Furthermore it seems that my vserver just cant handle that many workers. My development laptop can take 100 workers and still performs very fast.
Moreover when the load is heavy, database queries seem to be very slow on the ruby side.
I benchmarked simple Mongoid lookups like
time = Benchmark.realtime do
existing_option = Option.find_by(index: o.generate_index)
end
These benchmarks sometimes took longer than one second, but the query did not show up in the mongodb slow log (which means the actual query took less than 100ms on the database).
However pure rubycode (without database hits) seems to perform ok.
I wonder if it performs better when I implement the HTTP-requests with Typhoeus Hydra instead of sidekiq.
One thread contacting Redis is very different from 50 threads.
Use Sidekiq's redis connection pool within your workers. It will scale much better.
Sidekiq.redis do |conn|
conn.pipelined do
...
end
end
My Heroku Rails app maintains a large frequently-changing list of keywords.
I want to spawn up N amount of workers that will equally divide up this list of keywords and work on them until they are restarted (I restart them every time the list of keywords changes). Once restarted, they divide up the keywords again and churn away.
For example: Let's say I have 1,000 keywords.
If I spawn up 1 worker, that worker will take 1,000 keywords.
If I spawn up 10 workers, each worker will take 100 keywords.
If I spawn up 1,000 workers, each worker will take 1 keyword.
Workers basically just open a connection with Twitter for their set of keywords and process incoming tweets that match those keywords.
Any ideas on how to set up the Procfile and delegate X keywords between N workers?
Here's a naive/pseudo manual approach just for demonstration. However, I want to be able to spawn up an arbitrary amount of workers that will automatically split the keywords amongst themselves.
Procfile:
keywordstreamer0: bundle exec ruby keyword_streamer.rb 0
keywordstreamer1: bundle exec ruby keyword_streamer.rb 1
keyword_streamer.rb
streamer_id = ARGV.shift # 0 or 1
# Split all keywords into array of two groups and take the group
# that matches this worker id (so the two workers take different groups)
keywords = Keyword.all.split_into_groups_of(2)[streamer_id]
# Example work loop
TwitterStream.track(keywords).each do |incoming_tweet|
process(incoming_tweet)
end
Then, in my app, when I need to restart my keyword workers:
["keywordstreamer0", "keywordstreamer1"].each do |streamer|
restart(streamer)
end
I'd like to instead be able to spawn N amount of these workers but I'm am having trouble parceling out a solution. I'd appreciate any high-level overview suggestions!
If you're just processing one keyword at a time, in no particular order or grouping, you could just use a queue.
Each worker simply fetches the next keyword off the queue (or perhaps the next batch of keywords, for performance), does the work, and then saves the results somewhere. You don't need to worry about partitioning the workload, since the workers will simply ask for more work when they're ready, allowing you to scale to N workers without needing each worker to know about the total size of the available workload.
There are many possible ways you can implement queues for your data. A couple of more specialized ones that I've used before are AMQP and Redis, but that's hardly an exhaustive list.
I'm going to take a guess and say that since you've got Keyword.all in your example code, and you're on Heroku, that you're using postgres. You can also emulate a queue in postgres without too much difficulty, although it obviously won't perform as well as a purpose-built queue.
Here's one way of doing it:
Add a status column to your keywords. It will have 3 values: ready, in-progress, and complete. The default value for the status column is ready.
The pseudocode for your worker would look like this:
loop do
keyword = Keyword.where(:status => "ready").limit(1).first
keyword.update_attributes!(:status => "in-progress")
result = process(keyword)
save_result_somewhere(result)
keyword.update_attributes!(:status => "complete")
end
I've left out a bunch of implementation details like gracefully handling the queue being empty, initial setup of the queue, batching, and so on. But that's the gist of it. This should perform adequately for modest sizes of N, probably at least 10 or more workers. Beyond that you may want to consider a purpose-built queuing technology.
Once your queue is set up, every single worker is identical and autonomous. Just heroku ps:scale worker=N and you're done!
First, let me explain the situation, I've got following:
A "Node" Class with following attributes:
node_id (unique)
node_name (unique)
And a "NodeConnection" Class with following attributes:
node_from
node_to
We'll have around 1 to 3 million nodes and something around 3 to 10 million NodeConnections.
After the nodes and connections are imported once, they won't change.
On each request to the Rails-Application, we'll have to look up around 10 to 100 node_ids by possible node_names. And we have to lookup a few hundred to a few thousands node_connections.
We currently prototyped this without any caching (so, a LOT of database-queries) and response times were horrible (like 2 Minutes).
So we switched over to cache the nodes and connections via memcached.
Got a performance boost, but still lacking of performance. (Because we're calling Cache.read for every NodeConnection, that's a few thousand calls per request)
Now, we tried caching via Classvariable and got a huge performance boost. (Response times within a few hundred ms)
# Pseudocode below
class Node
def nodes
##nodes ||= get_nodes
end
def node_connections
##node_connections ||= get_node_connections
end
end
So, I'd like to ask about Pros and Cons of this solution.
Cons I've got yet:
Every Rails instance has to build up its own cache (it's own ClassVariables) -> higher total memory usage
Initializing the cache is time consuming (1-3 minutes), so we can't do this within a request
Any other solutions out there to cache large (>100MB) and static (data won't change during applications lifetime) data efficiently, so all rails instances within the same machine can access this cache very fast (!)?
It sounds like a very specific situation, but in order to avoid the need for a per-process in-memory cache (i.e. your class variables) to naturally warm up, I'd be investigating the feasibility of scripting the warm-up process and running it from inside an initializer... your app may take longer to start up, but your users would not have to wait.
EDIT | Note that if you were using something like Unicorn, which supports pre-loading application code before forking worker processes, you could minimize the impact of such initialization.
I am planning on using delayed job to run some background analytics. In my initial test I saw tremendous amount of memory usage, so I basically created a very simple task that runs every 2 minutes just to observe how much memory is is being used.
The task is very simple and the analytics_eligbile? method always return false, given where the data is now, so basically none of the heavy hitting code is being called. I have around 200 Posts in my sample data in development. Post has_one analytics_facet.
Regardless of the internal logic/business here, the only thing this task is doing is calling the analytics_eligible? method 200 times every 2 minutes. In a matter of 4 hours my physical memory usage is at 110MB and Virtual memory at 200MB. Just for doing something this simple! I can't even begin to imagine how much memory this will eat if its doing real analytics on 10,000 Posts with real production data!! Granted it may not run evevery 2 minutes, more like every 30, still I don't think it will fly.
This is running ruby 1.9.7, rails 2.3.5 on Ubuntu 10.x 64 bit. My laptop has 4GB memory, dual core CPU.
Is rails really this bad or am I doing something wrong?
Delayed::Worker.logger.info('RAM USAGE Job Start: ' + `pmap #{Process.pid} | tail -1`[10,40].strip)
Post.not_expired.each do |p|
if p.analytics_eligible?
#this method is never called
Post.find_for_analytics_update(p.id).update_analytics
end
end
Delayed::Worker.logger.info('RAM USAGE Job End: ' + `pmap #{Process.pid} | tail -1`[10,40].strip)
Delayed::Job.enqueue PeriodicAnalyticsJob.new(), 0, 2.minutes.from_now
Post Model
def analytics_eligible?
vf = self.analytics_facet
if self.total_ratings > 0 && vf.nil?
return true
elsif !vf.nil? && vf.last_update_tv > 0
ratio = self.total_ratings / vf.last_update_tv
if (ratio - 1) >= Constants::FACET_UPDATE_ELIGIBILITY_DELTA
return true
end
end
return false
end
ActiveRecord is fairly memory-hungry - be very careful when doing selects, and be mindful that Ruby automatically returns the last statement in a block as the return value, potentially meaning that you're passing back an array of records that get saved as a result somewhere and thus aren't eligible for GC.
Additionally, when you call "Post.not_expired.each", you're loading all your not_expired posts into RAM. A better solution is find_in_batches, which specifically only loads X records into RAM at a time.
Fixing it could be something as simple as:
def do_analytics
Post.not_expired.find_in_batches(:batch_size => 100) do |batch|
batch.each do |post|
if post.analytics_eligible?
#this method is never called
Post.find_for_analytics_update(post.id).update_analytics
end
end
end
GC.start
end
do_analytics
A few things are happening here. First, the whole thing is scoped in a function to prevent variable collisions from holding onto references from the block iterators. Next, find_in_batches retrieves batch_size objects from the DB at a time, and as long as you aren't building references to them, become eligible for garbage collection after each iteration runs, which will keep total memory usage down. Finally, we call GC.start at the end of the method; this forces the GC to start a sweep (which you wouldn't want to do in a realtime app, but since this is a background job, it's okay if it takes an extra 300ms to run). It also has the very distinct benefit if returning nil, which means that the result of the method is nil, which means we can't accidentally hang on to AR instances returned from the finder.
Using something like this should ensure that you don't end up with leaked AR objects, and should vastly improve both performance and memory usage. You'll want to make sure you aren't leaking elsewhere in your app (class variables, globals, and class references are the worst offenders), but I suspect that this'll solve your problem.
All that said, this is a cron problem (periodic recurring work), rather than a DJ problem, in my opinion. You can have a one-shot analytics parser that runs your analytics every X minutes with script/runner, invoked by cron, which very neatly cleans up any potential memory leaks or misuses per-run (since the whole process terminates at the end)
Loading data in batches and using the garbage collector aggressively as Chris Heald has suggested is going to give you some really big gains, but another area people often overlook is what frameworks they're loading in.
Loading a default Rails stack will give you ActionController, ActionMailer, ActiveRecord and ActiveResource all together. If you're building a web application you may not be using all of these, but you're probably using most.
When you're building a background job, you can avoid loading things you don't need by creating a custom environment for that:
# config/environments/production_bg.rb
config.frameworks -= [ :action_controller, :active_resource, :action_mailer ]
# (Also include config directives from production.rb that apply)
Each of these frameworks will just be sitting around waiting for an email that will never be sent, or a controller that will never be called. There's simply no point in loading them. Adjust your database.yml file, set your background job to run in the production_bg environment, and you'll have a much cleaner slate to start with.
Another thing you can do is use ActiveRecord directly without loading Rails at all. This might be all that you need for this particular operation. I've also found using a light-weight ORM like Sequel makes your background job very light-weight if you're doing mostly SQL calls to reorganize records or delete old data. If you need access to your models and their methods you will need to use ActiveRecord, though. Sometimes it's worth re-implementing simple logic in pure SQL for reasons of performance and efficiency, though.
When measuring memory usage, the only number to be concerned with is "real" memory. The virtual amount contains shared libraries and the cost of these is spread amongst every process using them even though it is counted in full for each one.
In the end, if running something important takes 100MB of memory but you can get it down to 10MB with three weeks of work, I don't see why you'd bother. 90MB of memory costs at most about $60/year on a managed provider which is usually far less expensive than your time.
Ruby on Rails embraces the philosophy of being more concerned with your productivity and your time than about memory usage. If you want to trim it back, put it on a diet, you can do it but it will take a bit of effort.
If you are experiencing memory issues, one solution is to use another background processing tech, like resque. It is the BG processing used by github.
Thanks to Resque's parent / child
architecture, jobs that use too much
memory release that memory upon
completion. No unwanted growth
How?
On certain platforms, when a Resque
worker reserves a job it immediately
forks a child process. The child
processes the job then exits. When the
child has exited successfully, the
worker reserves another job and
repeats the process.
You can find more technical details in README.
It is a fact that Ruby consumes (and leaks) memory. I don't know if you can do much about it, but at least I recommend that you take a look on Ruby Enterprise Edition.
REE is an open source port which promises "33% less memory" among all the other good things. I have used REE with Passenger in production for almost two years now and I'm very pleased.