I am trying to run a fairly complex query on our production database which is hosted on heroku. These are generally one-off fact finding queries but I am being kicked out each time I try to run it. Locally, the query runs fine and is fairly quick. It's also worse if I assign the result to a variable.
Any help regarding extending the time before heroku kicks me out or other ways to query the database would be greatly appreciated.
FYI - query I was running
authors = Author.includes(:books).where(books: {book_release_date: ('01/01/2020'.to_date.beginning_of_day..'30/12/2022'.to_date.end_of_day)})
The console closes without error which is deeply unhelpful. I am running this from the Heroku CLI i.e. heroku run rails console.
Solved this now.
This may not be applicable to everyone as you will need to be using dynos other than the basic and hobby types but you can run a one-off dyno using a different size. As the query I was running was potentially using too much memory (although this seemed unlikely) I needed to use a dyno with more ram. As this is a production build and we are using Standard-2x dynos, I could request this through the CLI with the following command
heroku run --size=standard-2x rails console
More information can be found here
I think you should extract the year in sql for your case. Something like that should be much more efficient:
Author.includes(:books).where("extract(year from books.book_release_date) = ?", 2020)
Related
I have a problem on my Rails application.
I am in version 3.2.22 of rails and 2.2.5 of ruby connect to a mongodb 2.6.
The problem is that I have huge difference in performance on simple or even more complex queries.
For example :
I run rails c development and then I execute my function (quite complex) it responds after 30 seconds
I run rails c production, I perform the same function as the previous one, it responds after 6 minutes 30 seconds, 7 times slower.
So I try to copy pasted the configuration 'development' in 'production', but the result remains the same, same for the Gemfile.
I look in all the code of the project no difference between the environment production and development.
Do you know the differences in the heart of rails between these two environments? did anyone ever encounter the problem?
Importantly, I am of course connecting to the same database.
Thanks in advance.
You have not specified your mongo (Ruby driver) and mongoid versions, if they are old you may need to upgrade and/or adjust the code to your environment.
To determine whether the slowdown happens in the database or in your application, use command monitoring as described here: https://docs.mongodb.com/ruby-driver/current/tutorials/ruby-driver-monitoring/#command-monitoring
Look at the log entries corresponding to your queries and make note of how log they take in each environment. By implementing a custom event subscriber you can also save the commands being sent and verify that they are identical between the two environments.
I got this!
When I saw the number of requests in production, I immediately thought of the query cache.
I found the 'identity_map_enabled' parameter for mongo, so I changed it to true, and hop magic!
I am developing a webapp (deployed in Heroku) that collect live football (soccer) data and then transforms it in stats.
Currently, as I am from Uruguay, the app only collects data from the Uruguayan, Libertadores and Sudamericana competitions, that has at most, 5 matches at the same time.
The problem I am having is that I need to start a cron when each of the matches starts and end it when they end, so I can manage with more precision the current minutes of the matches.
I have been reading some posts here, but I did not found in any of them that explained how to do what I need, neither how to use it on Heroku.
If it helps, I will provide the webapp infrastrucure information:
Ruby version: 2.4.1
Rails version: 5.0.1
Heroku dyno plan: Standard-2X with 1 dyno
Heorku Postrgres plan: Hobby Basic
Heroku stack: heroku-16
Thank you very much! And sorry for my english, clearly is not my first language.
Personaly I like to do it like this:
create a cron job that is executed really frequnetly, let say once per minute
I use ruby library for scheduling that lets me set executions as I want, and they are written nicely in code, which at least in my opinion makes them more maintanable then changing cron jobs on the server. Personally I use this library (there are many to pick from).
If you are on the free tier Heroku, I'm not sure it's going to let you execute cron really frequent, but still, you can use other services that can execute code whenever you want.
For example if you are hosting your code on GitLab, it already has scheduler built in as you can use it.
I created a simple stock screener (filters out stocks given certain criteria) in Rails. On my localhost the stocks update instantly, but on Heroku it can take anywhere from 10-15 seconds before the stock list is updated.
My Heroku app is here: http://fruthscreener.herokuapp.com/
Github is here: github dot com/anfruth/Fruth_Screener_Rails
The code involved in updating the queries can be found in the user_stocks model and in the stocks controller under def create.
Any ideas why this is happening and suggestions as to how to fix it?
Thank you.
Not slow for me
--
Heroku
The only thing which will slow Heroku down is if your db connection is "off-site"
We've had apps before which ran super slowly due to the case that the database provider was a different host, in a different country.
Heroku runs on AWS, meaning it will run super fast if you have all the dependencies in the same data-center. One of the drawbacks of using one of these powerful "cloud" hosting providers is they need to keep all requests local to help their system run quickly; hence if your DB is "off-site", it will slow it down profusely.
You must remember that Rails apps can't run unless they have a db connection; so if your connectivity is slow, your app's performance is going to be hit hard
-
Postgres
If your app is running slow on Heroku, the best thing to do is to make sure you're using Heroku's postgres database. This is deployed on Heroku's AWS cloud, meaning it's on the same network as your app, hence allowing it to run as quickly as possible
You'll need to change your app's database connection to the new production server like this:
#config/database.yml
production:
.... #-> your Heroku db details here
This will allow you to run heroku run rake db:migrate after you push this new code to Heroku - which should define the db structure for you, allowing you to populate it as you wish
It sounds like you would benefit from using New Relic or another performance management package for Heroku in order to find out what is causing you trouble exactly. The free tier of New Relic should be enough to get you started.
By the way, if your app is a Heroku free tier app (one single web dyno), then your dyno will go to sleep when not in use, and you may be encountering dyno spin-up costs, which are frequently about 5-15 seconds. Repeat the same query several times in several minutes and see if the slowness persists for every request, or only the first one.
When I run a Model.all query (e.g. Product.all) for a table with a few thousand records, the Rails console is killed after 15 seconds. Can I extend this threshold (I assume) or in another way run my query without exiting IRB?
I only have this problem in production. In my development environment (sqlite database) with as many records, it works just fine.
I'm running Ruby 1.9.3, Ruby on Rails 3 (with Unicorn) and use a postgres database.
When you load Product.all, it tries to load all products at once so the process fails to allocate sufficient memory. It is a bad practice to load all product at once. Try to load products in batches. You can do something like
Product.find_each do { |p| p.my_awesome_work }
instead of using Product.all.each {....}
It isn't happening locally because you might have powerful computer,the computer is capable to assign that much of memory to some process which is not the case in heroku.
You're probably overloading the database mem usage. Do you have monit running there? Maybe monit is restarting the process based on usage.
you can issue top on another terminal screen to see the usage, also tail the rails log.
See if it gives you something like "lost connection to database"
As an aside, make sure to run the console like this in prod:
bundle exec rails c production -s
-s is sandbox
I have one rubyonrails app that turned really slow in development mode. Everything is fine in production, but even a simple "hello world" takes seconds in dev. I checked the session store and every possible reason i found on the net, but I didn't find the problem. Am I missing something that is common knowledge? "Completed in 1657ms (View: 226, DB: 39)"
Development is definitely slower, because it reloads all components. Production mode only loads the components when the server is started.
If you find your app is still too slow in Production mode then you can start hunting down bottlenecks. You can start by optimizing DB queries, with :include and indicies. You can also try removing your gems and plugins systematically to find the parts that are slowing down your code.
This is usually the case if you are running webrick, its so slow it makes eyes bleed.
Try installing mongrel in dev
gem install mongrel
Create a new app, see if that's slow too - that'll point to your server stack (apache, mongrel, passenger, etc.) rather than your application. If it's just your application then google rails profiling - and pick one of the many options for profiling an application.
WEBrick is doing a reverse DNS lookup on connecting IPs by default. In other words, it's trying to see if your IP address is associated with a domain name. This is unnecessary and takes too long, so you can disable it.
Open the file "l/ruby/lib/ruby/1.9.1/webrick/config.rb" and locate the line with ":DoNotReverseLookup => nil".
Change nil to true.
Enjoy!