I'm trying out InfluxDB to know if my usecase fits.
My app generates a bunch of events like product created, product deleted, product purchased, payment recieved, category created etc. Each event has some other properties such as who created the product or what the payment method was...
I want to know how many products were purchased or howmany payments were done using a specific payment method or howmany payments were done for a day or till now or with in a time specified. Same for all the events like payment, shipping etc. I am yet to understand the concept of TSDB. Every example I see has some value that is varying, ie; temperature 23,30,23,35,24,33 and so on. In my app each event has a value of 1, since each event contibute to one unit of that event.
Is InfluxDB a good choice for this usecase ? If yes, How would I model my data for use cases like this ?
You could use TSDB for your e-commerce analysis but it might be better to try data warehouse instead, especially when your data volume grows rapidly.
TSDB are best used with time series data plus time series analysis. For example, if you care when the shopping cart is filled and emptied but don't care that much about what is in the shopping cart.
Your use case is more like OLAP and you could check out ClickHouse.
Related
In my rails application, I want to keep several daily metrics in order to see how this data changes over times. In other words, if I want to see how many times a user logged in on a particular date (and therefore allowing me to accumulate this data over times).
Some of this data I can figure out through queries, such as the number of posts a user made on a particular day (because the post model includes a date). However, there are many different daily metrics I want to keep track of.
I thought of creating a DataPlayers model which has data for every player and every day creating a new instance of this, but I don't think that is the best approach.
Are there best practices for this type of data collection?
You could use a gem like SqlMetrics to track events as they happen.
It stores the events in your own database so its easy to query them via sql.
I have the following situation (in Rails 3): my table contains financial transactions for each user (users can buy and sell products). Since lots of such transactions occur I present statistics related to the current user on the website, e.g. current balance, overall profit, how many products sold/bought overall, averages, etc. (the same also on a per month/per year basis instead of overall). Parts of this information is displayed to the user on many forms/pages so that the user can always see his current account information (different bits of statistics is displayed on different pages though).
My question is: how can I optimize database performance (and is it worth it)? Surely, if the user is just browsing, there is no need to re-calculate all of the values every time a new page is loaded unless a change to the underlying database has been made?
My first solution would be to store these statistics in their own table and update them once a financial transaction has been added/edited (in Rails maybe using :after_update ?). Taking this further, if, for example, a new transaction has been made, then I can just modify the average instead of re-calculating the whole thing.
My second idea would be to use some kind of caching (if this is possible?), or to store these values in the session object.
Which one is the preferred/recommended way, or is all of this a waste of time as the current largest number of financial transactions is in the range of 7000-9000?
You probably want to investigate summary tables, also known as materialized views.
This link may be helpful:
http://wiki.postgresql.org/wiki/Materialized_Views
I'm not sure if its the best idea to create a background job that scans the database all the time for updated product prices, especially when in my application I expect to have 100,000s if not millions of products being observed for lower prices.
I have two models, Product and Price. a Product can have many different Prices. So the way I thought it should be done was:
Subscribe to a Product.
Create background job that scans database every hour for lower prices.
Notify users by email.
I'm not sure about #2 and if that's the best way for it to be done. What would you guys suggest?
If your Rails app is adding new Price objects to the Products in its database, why not have an ActiveRecord callback (after_create in this instance) every time a new Price object is added?
That ActiveRecord callback could start a delayed job to email the user about the new, lower price.
If it were me, I would use database triggers rather than a background job sweeper.
Here's an intro to triggers:
http://www.mysqltutorial.org/mysql-triggers.aspx
I would appreciate some advice on how to structure a database for the following scenario:
I'm using Ruby on Rails, and so I have the following tables:
Products
Salespeople
Stores
Products are manufactured in batches, so each product item has a Batch code, so I think I will also need a table of batches, which refers to a product type.
Batch
In the real world, Salespeople take Product items (from a specific Batch) and in due course issue it to a Store. Importantly, Batches are large, and may be spread across many Salespeople, and subsequently, Stores.
At some future date, I would like to run the following reports:
Show all Batches of a Product issued to a specific Store.
Show all Batches held by a Salesperson (i.e. not yet sold).
Now, I'm assuming I need to build a table of Transactions, something like,
Transaction
salesperson_id
batch_id (through which the product can be determined)
store_id
typeOfTransaction (whether the Salesperson has obtained some stock, or sold some stock)
quantity
By dynamically running through a table of Transaction records, I can could derive the above reports. However, this seems inefficient and, over time, increasingly slow.
My question is: what is the best way to keep track of transactions like this, preferably without requiring dynamic processing of all transactions to derive total items from a batch given to a given store.
I don't believe I can just keep a central record of stock as Product comes in Batches, and Batches are distributed by Salepeople across Stores.
Thank you.
My question is: what is the best way to keep track of transactions like this, preferably without requiring dynamic processing of all transactions to derive total items from a batch given to a given store.
I don't believe I can just keep a central record of stock as Product comes in Batches, and Batches are distributed by Salepeople across Stores.
Believe it. :-)
In my experience, the only correct way to store this kind of stuff, is to break it down to something akin to T-leger accounting, i.e. debit/credit with a chart of accounts. It requires dynamic processing to derive totals as you've found out, but anything short of that will lead to tricky queries when dealing with reports and audit trails.
You can speed things up significantly, by maintaining partial or complete aggregate balances using triggers (e.g. monthly stock movements per store). This will reduce the number of rows you need to sum when running larger queries. Which of these you'll want to maintain will depend on your app and your reporting requirements.
I'm currently developing an application where I need to implement some sort of way of handling credits which are purchased by users of my application. These credits will then be used to preform certain tasks inside my application such as purchasing hits in mechanical turk. The reason I need to do this is because in the case of mechanical turk there is a possibility that our orders won't be filled and instead of just keeping the extra money for hits they didn't get I want to credit them for future purchases.
The important parts I need help fleshing out is how do you accurately manage an ongoing total of credits. I can't obviously calculate it every time. Additionally I need to manage the adding and subtracting of the credits. Also I probably need to track the origin of these credits, ie money or free because it is possible we might give out free credits as a reward but we need to be careful how to handle turning free credits into cash because it opens an incentive for scammers to use the credits to purchase turk hits then do the turk hit themselves and keep the money.
I currently work on a system with something very similar.
Users have an Account which tracks all transactions - origin, type and amount. We have a finite number of transactions types (including obvious the ones like credit/debit). The Account table is therefore actually an archive of all of the transactions for a particular user account, rather than being a simple running total.
Account totals are therefore calculated every time we need them, although it would be pretty simple to add some aggregates to the database that increment/decrement a total value as transactions are processed. As with all performance-related issues - don't do it until you need to, summing values across some smartly indexed columns is fine for our current system.