When I run a Model.all query (e.g. Product.all) for a table with a few thousand records, the Rails console is killed after 15 seconds. Can I extend this threshold (I assume) or in another way run my query without exiting IRB?
I only have this problem in production. In my development environment (sqlite database) with as many records, it works just fine.
I'm running Ruby 1.9.3, Ruby on Rails 3 (with Unicorn) and use a postgres database.
When you load Product.all, it tries to load all products at once so the process fails to allocate sufficient memory. It is a bad practice to load all product at once. Try to load products in batches. You can do something like
Product.find_each do { |p| p.my_awesome_work }
instead of using Product.all.each {....}
It isn't happening locally because you might have powerful computer,the computer is capable to assign that much of memory to some process which is not the case in heroku.
You're probably overloading the database mem usage. Do you have monit running there? Maybe monit is restarting the process based on usage.
you can issue top on another terminal screen to see the usage, also tail the rails log.
See if it gives you something like "lost connection to database"
As an aside, make sure to run the console like this in prod:
bundle exec rails c production -s
-s is sandbox
Related
I am trying to run a fairly complex query on our production database which is hosted on heroku. These are generally one-off fact finding queries but I am being kicked out each time I try to run it. Locally, the query runs fine and is fairly quick. It's also worse if I assign the result to a variable.
Any help regarding extending the time before heroku kicks me out or other ways to query the database would be greatly appreciated.
FYI - query I was running
authors = Author.includes(:books).where(books: {book_release_date: ('01/01/2020'.to_date.beginning_of_day..'30/12/2022'.to_date.end_of_day)})
The console closes without error which is deeply unhelpful. I am running this from the Heroku CLI i.e. heroku run rails console.
Solved this now.
This may not be applicable to everyone as you will need to be using dynos other than the basic and hobby types but you can run a one-off dyno using a different size. As the query I was running was potentially using too much memory (although this seemed unlikely) I needed to use a dyno with more ram. As this is a production build and we are using Standard-2x dynos, I could request this through the CLI with the following command
heroku run --size=standard-2x rails console
More information can be found here
I think you should extract the year in sql for your case. Something like that should be much more efficient:
Author.includes(:books).where("extract(year from books.book_release_date) = ?", 2020)
I have a problem on my Rails application.
I am in version 3.2.22 of rails and 2.2.5 of ruby connect to a mongodb 2.6.
The problem is that I have huge difference in performance on simple or even more complex queries.
For example :
I run rails c development and then I execute my function (quite complex) it responds after 30 seconds
I run rails c production, I perform the same function as the previous one, it responds after 6 minutes 30 seconds, 7 times slower.
So I try to copy pasted the configuration 'development' in 'production', but the result remains the same, same for the Gemfile.
I look in all the code of the project no difference between the environment production and development.
Do you know the differences in the heart of rails between these two environments? did anyone ever encounter the problem?
Importantly, I am of course connecting to the same database.
Thanks in advance.
You have not specified your mongo (Ruby driver) and mongoid versions, if they are old you may need to upgrade and/or adjust the code to your environment.
To determine whether the slowdown happens in the database or in your application, use command monitoring as described here: https://docs.mongodb.com/ruby-driver/current/tutorials/ruby-driver-monitoring/#command-monitoring
Look at the log entries corresponding to your queries and make note of how log they take in each environment. By implementing a custom event subscriber you can also save the commands being sent and verify that they are identical between the two environments.
I got this!
When I saw the number of requests in production, I immediately thought of the query cache.
I found the 'identity_map_enabled' parameter for mongo, so I changed it to true, and hop magic!
I created a simple stock screener (filters out stocks given certain criteria) in Rails. On my localhost the stocks update instantly, but on Heroku it can take anywhere from 10-15 seconds before the stock list is updated.
My Heroku app is here: http://fruthscreener.herokuapp.com/
Github is here: github dot com/anfruth/Fruth_Screener_Rails
The code involved in updating the queries can be found in the user_stocks model and in the stocks controller under def create.
Any ideas why this is happening and suggestions as to how to fix it?
Thank you.
Not slow for me
--
Heroku
The only thing which will slow Heroku down is if your db connection is "off-site"
We've had apps before which ran super slowly due to the case that the database provider was a different host, in a different country.
Heroku runs on AWS, meaning it will run super fast if you have all the dependencies in the same data-center. One of the drawbacks of using one of these powerful "cloud" hosting providers is they need to keep all requests local to help their system run quickly; hence if your DB is "off-site", it will slow it down profusely.
You must remember that Rails apps can't run unless they have a db connection; so if your connectivity is slow, your app's performance is going to be hit hard
-
Postgres
If your app is running slow on Heroku, the best thing to do is to make sure you're using Heroku's postgres database. This is deployed on Heroku's AWS cloud, meaning it's on the same network as your app, hence allowing it to run as quickly as possible
You'll need to change your app's database connection to the new production server like this:
#config/database.yml
production:
.... #-> your Heroku db details here
This will allow you to run heroku run rake db:migrate after you push this new code to Heroku - which should define the db structure for you, allowing you to populate it as you wish
It sounds like you would benefit from using New Relic or another performance management package for Heroku in order to find out what is causing you trouble exactly. The free tier of New Relic should be enough to get you started.
By the way, if your app is a Heroku free tier app (one single web dyno), then your dyno will go to sleep when not in use, and you may be encountering dyno spin-up costs, which are frequently about 5-15 seconds. Repeat the same query several times in several minutes and see if the slowness persists for every request, or only the first one.
I have a task that creates new activerecord records which I have recently moved to a background task using delayed_job and foreman, as recommended by Heroku
Sometimes this works fine, but sometimes it causes the Rails app in the browser to stop responding.
At this point I can see from the database that all the delayed jobs have completed, and that all the new records have been created.
However when I kill the processes I get a futher 11,200 lines of terminal output. This mostly consists of the execution by the web process of two methods on the model, both of which involve calls to the database:
validate :hit_database_to_see_if_model_exists?
before_save :get_rows_from_database_and_perform_calculation
There are also a number of INSERT statements which I'm sure have already hit the database because the number of records does not change before/after killing the processes
Here is my Procfile:
web: bundle exec rails server thin -p $PORT -e $RACK_ENV
worker: bundle exec rake jobs:work
So it feels like I am getting a 'stack overflow' (woop). Can you shed any light on:
What generally is going on?
Where in Rails this 'stack overflow' is taking place
Whether these things are actually happening after I hit 'Ctrl + C' or just being printed to the terminal at that point?
What might be causing this?
How I could debug / fix it?
UPDATE
It looks like there are certain tasks that are being assigned to the web process by the background task, but not being executed until the browser is 'prodded.' In some circumstances they all execute, but if there are too many the app falls over. Any idea on what might be causing this?
UPDATE
I tried running the web and worker processes in two separate windows.
In this scenario I have been unable to replicate the problem of the browser hanging, and in each case the worker process completed properly.
However I did make the interesting observation that if I don't touch the browser then no output appears in the web window. However if I do touch the browser then thousands of lines of what the worker process is doing at that moment appear in the web window.
Is this normal? Does this shed any light on what the problem might be?
Update
At the bottom of the terminal output after I kill the processes it says "Killed: 9"
07:45:21 system | sending SIGKILL to all processes
Killed: 9
What exactly does this 9 refer to? Is this unusual?
Update
I am using:
delayed_job 3.0.4
delayed_job_active_record 0.3.3
delayed_job_web 1.1.2
foreman 0.60.2
RESOLUTION
Thanks to #Justin's answer below (and this related question). It seems that Ruby buffers stdout by default, and that this buffer was overflowing, causing the app to stop responding. I added $stdout.sync = true at the top of config/environments/development.rb, and the problem appears to have gone away.
This is only a partial answer, but it might help you with debugging.
Rails buffers logging by default, and flushes it after every web request. One option is to simply replace the logger with a simpler logger
Rails.logger = Logger.new(STDOUT)
You can also configure the buffered logger to flush more often
Rails.logger.auto_flushing = (Rails.env.development? || Rails.env.test?)
You also have to be careful about STDOUT. In my current project I have stdout flushing enabled in both config.ru and my background bootup (I'm using sidekiq, so the boot process may be a little different)
STDOUT.sync = true
As to your larger problem,
I'm surprised that the rails process is running background tasks. Is there an option to disable that, at least for experimentation?
Then there are the standard debugging tools - all the database calls on save worry me, so I'd try various combinations of disabling them to see if anything improves. Especially with two of them - for instance, if your before_save hook changes a value in the model, it might be trigger a validation; if that resets the before_save hooks, you'd have a loop.
I have one rubyonrails app that turned really slow in development mode. Everything is fine in production, but even a simple "hello world" takes seconds in dev. I checked the session store and every possible reason i found on the net, but I didn't find the problem. Am I missing something that is common knowledge? "Completed in 1657ms (View: 226, DB: 39)"
Development is definitely slower, because it reloads all components. Production mode only loads the components when the server is started.
If you find your app is still too slow in Production mode then you can start hunting down bottlenecks. You can start by optimizing DB queries, with :include and indicies. You can also try removing your gems and plugins systematically to find the parts that are slowing down your code.
This is usually the case if you are running webrick, its so slow it makes eyes bleed.
Try installing mongrel in dev
gem install mongrel
Create a new app, see if that's slow too - that'll point to your server stack (apache, mongrel, passenger, etc.) rather than your application. If it's just your application then google rails profiling - and pick one of the many options for profiling an application.
WEBrick is doing a reverse DNS lookup on connecting IPs by default. In other words, it's trying to see if your IP address is associated with a domain name. This is unnecessary and takes too long, so you can disable it.
Open the file "l/ruby/lib/ruby/1.9.1/webrick/config.rb" and locate the line with ":DoNotReverseLookup => nil".
Change nil to true.
Enjoy!