I'm trying to optimize my site, in order to do that I want to know which action is making more queries than others. Is there anyway to know the amount of DB hits made by one action?
i found these gem very helpful for inspecting issues and optimizing queries
https://github.com/noahd1/oink
https://github.com/flyerhzm/bullet
This gem will do exactly what you need: show the number of db queries issued per action:
https://github.com/makandra/query_diet
You could look at the rails log to see what & how many queries are being fired for each request. But its usually pain go though the log each time to see which request are taking time.
Usually I use newrelic gem in development to see which actions are taking more time and will try to optimize the queries. Refer to http://newrelic.com/docs/ruby/developer-mode for more info on newrelic in development model.
Also based on the database and rails version there are other gems ( https://github.com/flyerhzm/bullet ) which tells you how queries are performed in a particular request.
Related
I have built a rails app for storing events. I want to generate a report to find the number of events happened in a day. I want to do it asynchronously. I am new to sidekiq and Redis. Can anyone suggest a good resource to study?
My suggestion for this would be to do this in a rake task that would be run on the server once a day.
You can find good resources on how to create rake tasks online and then use this simple gem to make sure the rake task runs once a day on the server.
https://github.com/javan/whenever
I am assuming you have a Profile model. You could use the timestamps in this model created_at to get all the profiles created on a given day. You could then create a CSV or whatever you like with that data and email it to whoever needs the report (how you handle the data is up to you)
You can do all the above in Sidekiq if you wish, I would recommend reading through the gem docs and this getting started guide from the official wiki https://github.com/mperham/sidekiq/wiki/Getting-Started
It's fairly straightforward and once you get your first process working it will start to make more sense.
I would also highly reccomend this video before you start working with sidekiq and redis, to give you an overall background of how sidekiq works and in what use cases it may be helpful to you.
https://www.youtube.com/watch?v=GBEDvF1_8B8
In the app I'm working on we compare against Time.now in some places, and Database.time in others. I would like to use only one of these throughout the whole application. Unfortunately I haven't been able to find much information on them through google searches. Which is preferred? How are they different from each other? Thanks!
There are a couple of articles on thoughtbot's blog that cover this:
https://robots.thoughtbot.com/its-about-time-zones
https://robots.thoughtbot.com/a-case-study-in-multiple-time-zones
Hopefully those help.
The function Time.now gives you the time in the application server running your Rails app request. This is reliable in most cases.
Rails uses the app server time when filling the updated_at and created_at columns of its models.
I'm not sure about the Database.time function you are referring to... This is probably a class defined in your code or inherited from gem in your app. Because of that I would suggest moving to the Ruby core time functions. In your case: Time.now
I've added sunspot gem in my application and tried to send it to production in heroku, but I'm trying to reindex my database, however, I'm getting an error. I did some more digging and I think I have to add websolr as an add-on? This costs $20/month. Is this the only option?
THanks
Founder of Websolr + Bonsai here (Heroku addons for Solr and Elasticsearch).
Rich's answer is pretty solid, with the exception of the SQL LIKE operator, which I do not recommend. The performance does not scale, and you're either going to sink in a lot more time than you might expect in order to eke out baseline search functionality. End result: a lot of time spent, and unhappy users.
Postgres full text search is a reasonable alternative, though the term analysis and result ranking will be lacking compared to Solr/Elasticsearch as your search traffic starts to grow in production.
You might also consider our sister service, Bonsai, which does offer a free Starter plan. It uses Elasticsearch, which means you'd want to use the official Ruby bindings for Elasticsearch rather than Sunspot.
Lastly, if you already have a production app on Heroku, you are welcome to create more than one index in your account, and share those indexes with your staging/qa and other apps.
I've done some more research and found out that there are other options if you don't want to take the websolr path. These other answers are good for some insights, but doesn't give an alternative to what can be used.
For some that's still looking, I suggest taking a look at Elastic Search
Rails Cast has a good tutorial on this as well.
And to use it with heroku, look into Bonsai which gives users a free option.
Hopefully this answer will help those that are also seeking other options than using sunspot gem with solr
Solr on Heroku uses their own add-on, which starts at $20pm:
Although I don't know why it costs up front, and doesn't have a "trial" option like many of the other Heroku Add-ons, there are certain ways around it
Full Text Search
Full text search is what you're performing, and Solr is a tool to make the process much more efficient. Despite being quite DB-expensive, you can use full text searching with Heroku, depending on your DB:
MYSQL
To perform full-text searching on MYSQL, you can simply use the "LIKE" operator with %variable% as your search phrase, like this:
SELECT * FROM `table` WHERE `name` LIKE `%benjamin%`
This basically finds all the records where the name column contains "benjamin" somewhere inside it. This is quite slow
POSTGRESQL
PostgreSQL offers more power in its full text searching, but is nonetheless still quite slow & expensive. You can read more about it here, but with rails, you can use a bunch of gems which do the task for you
We recently used a gem called textacular here: http://firststop.herokuapp.com
Here is the code we used for it:
#Search
def self.search(search)
basic_search(name: search, description: search)
end
Further Reading
You can see how full text searching works here: Any reason not use PostgreSQL's built-in full text search on Heroku?
I would recommend if you're just getting the foundations established for your app. Afterwards, you can upgrade to a more dedicated solution in the form of Solr et al
Here are
If you want to use the Heroku platform it starts for free, but you have to pay for almost every add-on, extra workers, extra storage, search engine, background tasks, you name it.
For $20/month you could also get a decent VPS, but you would have to install and manage that server by yourself.
As for sunspot/solr on Heroku, I don't think you can do that for free.
I'm imagining a rails gem that I can feed with a list of SQL queries such as count (*) from users where active = 1 and this gem would provide a rake task that I can run as a daily cron job and have a graphic interface that plots the results as a graph. For example, one graph for number of daily active users, another graph for number of new registrations etc. A very simple and basic BI for my rails database.
It might be simple enough to implement, still, I'd be happy to find a prepackaged gem.
Googling didn't find one, anyone knows one?
thanks!
There are several gems for reporting : https://www.ruby-toolbox.com/categories/reporting
Firstly let me mention that I'm new to web-frameworks.
I have to write my first web-app for a Uni project. I spent two weeks learning Grails and Django. Started working with Rails yesterday and loved it. So I've decided to go with it and discard my work in the other frameworks.
About the app
It's supposed to be a Twitter app that utilizes Twitter's Streaming API to record tweets which match a set of specified filters. (I'm going to use the Tweetstream gem which takes care of connecting to Twitter and capturing matching tweets).
The app's web interface should have the following functionality -
Creating new requests The user inputs a set of filter parameters (keywords to track) & URL/username/password of an existing PostgreSQL or MySQL database.
When a request is created, the web-app spawns a background ruby process. This process connects to Twitter via the Tweetstream gem. It also connects to the database specified by the user to stores received tweets.
View/terminate of existing requests
The user should be able to see a list of requests that are running as background processes by visiting a URL such as /listRequests.
See further details about a process/terminate the process
The user should be able to go to URL such as /requests/1/detail to view some details (e.g how long request has been running, number of tweets captured, etc). The user should also be able to terminate the process.
My inexperience is showing as I'm unable to comprehend -
what my models should be (maybe Request should be a model. Tweet doesn't need to be a model as it's not being stored locally)
how I'm going to connect to remote databases.
how I can create background processes (backgroundrb??) and associate them with request objects so that I can terminate then when the user asks.
At the end of the day, I've got to build this myself, so I'm not asking for you to design this for me. But some pointers in the right direction would be extremely helpful and appreciated!
Thanks!
Hmm.
Since the web app is just a thin wrapper around the heavy-lifting processes, it might be more appropriate to just use something like Sinatra here. Rails is a big framework that pulls in lots of stuff that you won't need for this project, even though it will work.
Does the "background process" requirement here strictly mean a separate process, or does it just mean concurrency? TweetStream uses the EventMachine gem to handle updates as they come, which uses a separate thread for each connection. It would be quite possible to spawn the TweetStream clients from a simple Sinatra web app, keep them in a big array, have them all run concurrently with no trouble, and simply run stop on a given client when you want it to stop. No need for a database or anything.
I'm not sure exactly what your prof is looking for you to do here, but MVC doesn't really fit. It's better to work with the requirements than to mush it into a design pattern that doesn't match it :/
Even so, I <3 Rails! Definitely get on that when you're working primarily with objects being represented in a database :)
Quite a project. Most of what will be challenging is not related to rails itself, but rather the integration with background processes. backgroundrb is a bit out of fashion. The last commit on the main github project is over a year ago, so it's likely not up to snuff for Rails 3. Search around and evaluate your options. Resque is popular, but I'm not sure if your real-time needs match with its queue-based structure.
As for your app, I see only a single model, but don't call it request. That's a reserved name in rails. Perhaps a Search model, or something along that line.
Connecting to different databases is straight forward but will require direct configuration of your ActiveRecord class during operation rather than using database.yml.