How does one get records -- by value -- in a more efficient way?
Currently I’m doing this:
Coupon = [P || P <- kvs:all(company_coupon), P#company_coupon.company_id == C#company.id],
My question is geared at kvs:all(...). In databases it is usually pretty expensive to get all entries first and then match them.
Is there a better way?
PS: "lists:keyfind" also needs to be provided with ALL records first, to then run them through the loop.
How are you guys doing it?
Cheers!
One can use kvs:index(table, field, value) if one has set the field as a key before:
#table{name=user,fields=record_info(fields,user), keys = [field]}
When you are using a functional language like erlang or lisp, traversing data is unavoidable in most of cases while sql doesn't need it. So it is better to do it with sql if you are storing data in a database like postgres that supports the sql but if you don't need to save data, you are in the correct way.
Related
I've always worked with relational DBs and recently decided to migrate a performance-critial service from SQL Server to Tarantool with a hope to take advantage of the fast in-memory search and processing. I've got a couple of questions while planning for the migration.
I've got a table with about one million records containing pricing information which means I'm dealing mostly with numbers and uuids. First, I need to run a select containing multiple conditions to get a subset of the data, like
SELECT * FROM rates WHERE SupplierId = #SupplierId AND ProductId = #ProductId AND (LocalDistributionZoneId = #LocalDistributionZoneId OR LocalDistributionZoneId IS NULL)
Q1: What is the strategy of running such a query in Lua? Do I create an index for each field in the predicate or I can go along with one secondary composite index?
Q2: Will it be more covenient to run such a query in SQL (box.sql.execute) rather than in pure Lua? Will it be considerably slower than running the same query in pure Lua?
Q3: If I use SQL, is it possible to review the execusion plan to make sure that the query I run really uses the indexes I've defined in the space?
Ok, after I've get the results from the first query I need to analyse the data and then based on the results of analysis, run one more query on the dataset returned by the first query.
Q4: Can Tarantool help me in dealing with the intermediate dataset? More specifically, may I somehow run more queries against the intermediate subset of tuples leveraging the indexes created in the space? Or, I would need to implement alternative strategies like re-add the intrim results to a temporary space with pre-defined indexes and then do another select, or implement further search myself?
Thank you!
Don't. Use SQL, it's faster: it doesn't create garbage collected objects for intermediate execution results.
Yes, please use our SQL features for that.
Use EXPLAIN statement.
I don't know what you exactly mean by "help". You could try to whatever strategy works best: create a more complex query, save the original query in a view to use in the resulting query, create a temporary table and work with it. To give more details let's look if the execution plan Tarantool chooses is good enough or you have to manually optimize it.
For my program, I have multiple tables inside of each other for organization and readability. These tables can look something like this
local buttons = {
loadingScreen = {
bLoadingEnter = function()
end
}, ...
}
What I want to do is find the first element named bLoadingEnter in the table. I dont know that the element named bLoadingEnter will be in loadingScreen. I've thought of getting all the keys in the table then check them. I couldn't get that to work. Any help would be appreciated!
If execution speed is in any way relevant, the answer is:
you don't
Looking up a key in a nested table takes a lot longer than looking it up in just a normal table. The next problem is that of uniqueness. Two or more nested tables can have the same key with different values, which may lead to odd bugs down the road. You'd either have to check for that when inserting (making your code even slower) or just hope things miraculously go well and nothing explodes later on.
I'd say just use a flat table. If your keys are well named (like bLoadingEnter) you'll be able to infer meaning from the name either way, no nesting required.
That being said, nested tables can be a good option, if most of the time you know what path to take, or when you have some sort of ordered structure like a binary search tree. Or if speed really isn't an important factor to consider.
Okay try
for _, item in ipairs(buttons) do
if item.bLoadingEnter ~= nil then
return item.bLoadingEnter
end
end
I have been searching everywhere but I can't seem to find this problem anywhere. In Rails 5.0.0.beta3 I need to sort a my #record = user.records with an association and it's record.
The sort goes something like this.
#record = #record.sort_by { |rec|
If user.fav_record.find(rec.id)
User.fav_record(rec.id).created_at
Else
rec.created_at
End
This is just an example of what I do. But everything sorts fine.
The problem:
This returns an array and not an Active Record Class.
I've tried everything to get this to return an Active Record Class. I've pushed the sorted elements into an ID array and tried to extract it them in that order, I've tried mapping. Every result that I get turns my previous active record into an array or hash. Now I need it to go back into an active record. Does anyone know how to convert an array or hash of that active record back into an Active Record class?
There isn't a similarly easy way to convert ActiveRecord to array.
If you want to optimize the performance of your app, you should try to avoid converting arrays to ActiveRecord queries. Try and keep the object as a query as long as possible.
That being said, working with arrays is generally easier than queries, and it can feel like a hassle to convert a lot of array operations to ActiveRecord query (SQL) code.
It'd be better to write the sort code using ActiveRecord::Query methods or even writing it in plain SQL using find_by_sql.
I don't know what code you should specifically use here, but I do see that your code could be refactored to be clearer. First of all, If and Else should not be capitalized, but I'm assuming that this is just pseudocode and you already realize this. Second, your variable names should be pluralized if they are queries or arrays (i.e. #record.sort_by should be #records.sort_by instead).
It's worth mentioning that ActiveRecord queries are difficult to master and a lot of people just use array operations instead since they're easier to write. If "premature optimization is the root of all evil", it's really not the end of the world if you sacrifice a bit of performance and just keep your array implementation if you're just trying to make an initial prototype. Just make sure that you're not making "n+1" SQL calls, i.e. do not make a database call every iteration of your loop.
Here's an example of an array implementation which avoids the N+1 SQL issue:
# first load all the user's favorites into memory
user_fav_records = user.fav_records.select(:id, :created_at)
#records = #records.sort_by do |record|
matching_rec = user.fav_records.find { |x| x.id.eql?(rec.id) }
# This is using Array#find, not the ActiveRecord method
if matching_rec
matching_rec.created_at
else
rec.created_at
end
end
The main difference between this implementation and the code in your question is that I'm avoiding calling ActiveRecord's find each iteration of the loop. SQL read/writes are computationally expensive, and you want your code to make as little of them as possible.
I have a 3rd or 4th normal form mnesia database and the table in question should be a key/value hash, however, the architect put the keys and values in the key portion of the record.
It looks something like:
-record(idx,{key,value}).
...
Invoice = 1,
Date = now(),
K1 = {?NORMAL_TYPE1,Invoice,Date},
mnesia:write(#idx{key=K1}).
...
So the question is... if I only know the Invoice number can I still get the data from the DB in the same time as if the Invoice was the only key instead of the tuple?
K2 = {?NORMAL_TYPE1,Invoice,'_'},
Rec = mnesia:read(#idx{key=K2}).
The short answer: No.
The longer answer: You may have a chance if the table is an orderet_set or something such, but I would not count on it.
The Aside: mnesia is usually not too effective w.r.t. normalized data. It is usually better to use standard RDBMS systems for that.
You can use mnesia:match_object/1/3 and mnesia:select/2/3/4/1 where you provide a pattern which can contain '_' as a don't care variable. I assume that is what you meant.
I'm using Rails3.rc and Active Record 3 (with meta_where) and just started to switch to Sequel, because it seems to be much faster and offers some really great features.
I'm already using the active_model plugin (and some others).
As far as I know, I should use User[params[:id]] instead of User.find(params[:id]). But this doesn't raise if no record exists and doesn't convert the value to an integer (type of PK), so it's as a string in the where clause. I'm not sure if this is causing any performance issues. Does this harm identity_map? What's the best way to solve both these issues?
Is there an easy way to flip the usage of associations like User.messages_dataset and User.messages so that User.messages behaves like in Active Record (User.messages_data_set). I guess I'd use the #..._dataset a lot but never need the array method, because I could just add .all?
I noticed some same (complex) queries are executed several times within one action sometimes. Is there something like the Active Record query cache? (identity_map doesn't seem to work for these cases).
Is there a to_sql I can call to get the raw SQL a dataset would produce?
You can either use:
User[params[:id].to_i] || raise Sequel::Error
Or write your own method that does something like that. Sequel supports non-integer primary keys, so it wouldn't do the conversion automatically. It shouldn't have any affect on an identity map. Note that Sequel doesn't use an identity map unless you are using the identity_map plugin. I guess the best way is to write your own helper method.
Not really. You can use the association_proxies plugin so that non-array methods are sent to the dataset instead of the array of objects. In general, you shouldn't be using the association dataset method much. If you are using it a lot, it's a sign that you should have an association for that specific usage.
There is and will never be a query cache. You should write your actions so that the results of the first query are cached and reused.
Dataset#sql gives you the SELECT SQL for the dataset.