I'm imagining a rails gem that I can feed with a list of SQL queries such as count (*) from users where active = 1 and this gem would provide a rake task that I can run as a daily cron job and have a graphic interface that plots the results as a graph. For example, one graph for number of daily active users, another graph for number of new registrations etc. A very simple and basic BI for my rails database.
It might be simple enough to implement, still, I'd be happy to find a prepackaged gem.
Googling didn't find one, anyone knows one?
thanks!
There are several gems for reporting : https://www.ruby-toolbox.com/categories/reporting
Related
I have built a rails app for storing events. I want to generate a report to find the number of events happened in a day. I want to do it asynchronously. I am new to sidekiq and Redis. Can anyone suggest a good resource to study?
My suggestion for this would be to do this in a rake task that would be run on the server once a day.
You can find good resources on how to create rake tasks online and then use this simple gem to make sure the rake task runs once a day on the server.
https://github.com/javan/whenever
I am assuming you have a Profile model. You could use the timestamps in this model created_at to get all the profiles created on a given day. You could then create a CSV or whatever you like with that data and email it to whoever needs the report (how you handle the data is up to you)
You can do all the above in Sidekiq if you wish, I would recommend reading through the gem docs and this getting started guide from the official wiki https://github.com/mperham/sidekiq/wiki/Getting-Started
It's fairly straightforward and once you get your first process working it will start to make more sense.
I would also highly reccomend this video before you start working with sidekiq and redis, to give you an overall background of how sidekiq works and in what use cases it may be helpful to you.
https://www.youtube.com/watch?v=GBEDvF1_8B8
I've added sunspot gem in my application and tried to send it to production in heroku, but I'm trying to reindex my database, however, I'm getting an error. I did some more digging and I think I have to add websolr as an add-on? This costs $20/month. Is this the only option?
THanks
Founder of Websolr + Bonsai here (Heroku addons for Solr and Elasticsearch).
Rich's answer is pretty solid, with the exception of the SQL LIKE operator, which I do not recommend. The performance does not scale, and you're either going to sink in a lot more time than you might expect in order to eke out baseline search functionality. End result: a lot of time spent, and unhappy users.
Postgres full text search is a reasonable alternative, though the term analysis and result ranking will be lacking compared to Solr/Elasticsearch as your search traffic starts to grow in production.
You might also consider our sister service, Bonsai, which does offer a free Starter plan. It uses Elasticsearch, which means you'd want to use the official Ruby bindings for Elasticsearch rather than Sunspot.
Lastly, if you already have a production app on Heroku, you are welcome to create more than one index in your account, and share those indexes with your staging/qa and other apps.
I've done some more research and found out that there are other options if you don't want to take the websolr path. These other answers are good for some insights, but doesn't give an alternative to what can be used.
For some that's still looking, I suggest taking a look at Elastic Search
Rails Cast has a good tutorial on this as well.
And to use it with heroku, look into Bonsai which gives users a free option.
Hopefully this answer will help those that are also seeking other options than using sunspot gem with solr
Solr on Heroku uses their own add-on, which starts at $20pm:
Although I don't know why it costs up front, and doesn't have a "trial" option like many of the other Heroku Add-ons, there are certain ways around it
Full Text Search
Full text search is what you're performing, and Solr is a tool to make the process much more efficient. Despite being quite DB-expensive, you can use full text searching with Heroku, depending on your DB:
MYSQL
To perform full-text searching on MYSQL, you can simply use the "LIKE" operator with %variable% as your search phrase, like this:
SELECT * FROM `table` WHERE `name` LIKE `%benjamin%`
This basically finds all the records where the name column contains "benjamin" somewhere inside it. This is quite slow
POSTGRESQL
PostgreSQL offers more power in its full text searching, but is nonetheless still quite slow & expensive. You can read more about it here, but with rails, you can use a bunch of gems which do the task for you
We recently used a gem called textacular here: http://firststop.herokuapp.com
Here is the code we used for it:
#Search
def self.search(search)
basic_search(name: search, description: search)
end
Further Reading
You can see how full text searching works here: Any reason not use PostgreSQL's built-in full text search on Heroku?
I would recommend if you're just getting the foundations established for your app. Afterwards, you can upgrade to a more dedicated solution in the form of Solr et al
Here are
If you want to use the Heroku platform it starts for free, but you have to pay for almost every add-on, extra workers, extra storage, search engine, background tasks, you name it.
For $20/month you could also get a decent VPS, but you would have to install and manage that server by yourself.
As for sunspot/solr on Heroku, I don't think you can do that for free.
I have to call externals API to fill my database, hosted on Heroku each hours.
For this purpose, I've a ruby script that get all the data from externals API and output on the stdout. Now, I would like to store those results in my database, I have differents ways to do it (Please let a comment if you know a better way).
What I have (Constraints) :
Ruby on Rails Application running on Heroku
PG Database hosted on Heroku
"Cars" model, with "Title", "Description", "Price" attributes, and 1 other nested attribute from "Users" Model (So same schema in PG).
Ruby Script that query the differents externals API
Ruby Script have to be called each hours / 2 hours / days. The script is going to run for about 10 minutes -> 2 hours depending of the number of results
My 3 differents ways to do it :
Running the script on a EC2 Instance, and fill my database with external login directly to the database, not by the Ruby on Rails REST API.
The problem is that it never ask for the Ruby on Rails validators, so for example if my database changed, or if I have to validate some data, it won't.
Running the script on a EC2 Instance, and fill my database with cll to my RoR REST API, so filling the data with JSON / XML. The problem is that I think if I have > 1000 calls from the API, it can make my dynos suffer with high load.
Running my script on a specific dyno on Heroku (I need some informations, I can't find some informations on Heroku)
(Please let a comment if you know a better way)
What do you think ? I need something really evolutive, if tomorrow i change my "Cars" model, everything has to be easy to make the switch between old and new model.
Thank you.
I would think that the best approach would be to use a background process to perform the work. Gems like http://sidekiq.org/ and DelayedJob all have the ability to schedule jobs (which then reschedule themselves for 2 hours later in your case).
On Heroku, workers run seperate to your web dynos so won't interfere with the performance it also keeps things simple in that you don't need to expose an API since you'll have direct access to your models from the worker.
There are plenty of Heroku docs on this subject;
https://devcenter.heroku.com/articles/background-jobs-queueing
https://devcenter.heroku.com/articles/delayed-job
You can do this by writing your scripts as a Rake tasks and then use Heroku Scheduler to schedule your task(s) to run at specific intervals:
https://devcenter.heroku.com/articles/scheduler
You can separate your tasks by schedule if you have multiple, and then just add multiple schedulers. They run in one-off dynos (which you pay for at the normal rate), and since they're running from the same code base can leverage all your existing app code (models, libs, etc).
in my RoR app I need to develop a visual query generator as similar as possible to the one provided by MS ACCESS. (sample screenshot of how this looks in ms access)
The user would be able to choose db tables, from the tables choose fields and then add conditions to the fields.
Is there any gem / code that you are aware of, that would help me in this endavour?
This would need a lot of TLC to get to be like the MS Access Query, but you could use the Ransack gem to accomplish the queries and nested associations.
http://railscasts.com/episodes/370-ransack
I'm trying to optimize my site, in order to do that I want to know which action is making more queries than others. Is there anyway to know the amount of DB hits made by one action?
i found these gem very helpful for inspecting issues and optimizing queries
https://github.com/noahd1/oink
https://github.com/flyerhzm/bullet
This gem will do exactly what you need: show the number of db queries issued per action:
https://github.com/makandra/query_diet
You could look at the rails log to see what & how many queries are being fired for each request. But its usually pain go though the log each time to see which request are taking time.
Usually I use newrelic gem in development to see which actions are taking more time and will try to optimize the queries. Refer to http://newrelic.com/docs/ruby/developer-mode for more info on newrelic in development model.
Also based on the database and rails version there are other gems ( https://github.com/flyerhzm/bullet ) which tells you how queries are performed in a particular request.