Lets say I have a collection with a views_count, males_views_count and females_views_count fields. I would like to be able to $inc (increment) the count fields respectively when a page is viewed.
The issue with this is I receive several concurrent connections, and a race condition may occur.
I've been reading about atomic operations in Mongodb. Where they typically have a succeed or fail approach. Either the record is written or it isn't. Does that mean I need to create the logic to determine whether the operation failed and if so, retry it?
Going back to my scenario. What if I wanted to make every view count (even when a race conditions occur). I know Mongodb locks are slightly different than traditional RDBMS'. Typically, I would implement an optimistic locking technique. Similar to:
begin
# .. do work .. determine if user is a male or a female
stat.save
rescue StaleDocumentError
stat.reload
retry
end
Or are atomic operations meant to prevent race conditions as it is the server who does the update and it is authoritative about what the truth is? If so, do we still need to implement optimistic/pessimistic locking techniques, or will Mongodb take care of that for us if we use atomic operations?
If you are using atomic operations such as $inc there is no race condition. If you have two increments from different threads they will be applied in the order received by the server and the end outcome on the document will be the same after both operations are applied.
If you are updating fields to specific values using $set, the "winning" value will be the last $set applied. If your application use case may lead to conflicting updates of the same field, then optimistic locking/versioning would be useful.
As far as exception handling goes, you should assume that any database operation could potentially result in a server exception (network error, duplicate key, ...) and retry as appropriate. There isn't any special logic required for atomic updates versus non-atomic updates, unless you choose to implement your own optimistic locking (in which case you would probably refetch the current version of the document and retry the operation).
The MongoDB manual covers a few different patterns in Isolate Sequence of Operations.
Related
I have X amount of cores doing unique work in parallel, however, their output needs to be printed in order.
Object {
Data data
int order
}
I've tried putting the objects in a min heap after they're done with their parallel work, however, even that is too much of a bottleneck.
Is there any way I could have work done in parallel and guarantee the print order? Is there a known term for my problem? Have others encountered it before?
Is there any way I could have work done in parallel and guarantee the print order?
Needless to say, we design parallelized routines with focus on an efficiency, but not constraining the order of the calculations. The printing of the results at the end, when everything is done, should dictate the ordering. In fact, parallel routines often do calculations in such a way that they’re conspicuously not in order (e.g., striding on each thread) to minimize thread and synchronization overhead.
The only question is how you structure the results to allow efficient storage and efficient, ordered retrieval. I often just use a mutable buffer or a pre-populated array. It’s very efficient in terms of both storage and retrieval. Or you can use a dictionary, too. It depends upon the nature of your Data. But I’d avoid the order property pattern in your result Object.
Just make sure you’re using optimized build if using standard Swift collections, as this can have a material impact on performance.
Q : Is there a known term for my problem?
Yes, there is. A con·tra·dic·tion:
Definition of contradiction…2a : a proposition, statement, or phrase that asserts or implies both the truth and falsity of something// … both parts of a contradiction cannot possibly be true …— Thomas Hobbes
2b : a statement or phrase whose parts contradict each other// a round square is a contradiction in terms
3a : logical incongruity
3b : a situation in which inherent factors, actions, or propositions are inconsistent or contrary to one anothersource: Merriam-Webster
Computer science, having borrowed the terms { PARALLEL | SERIAL | CONCURRENT } from the theory of systems, respects the distinctive ( and never overlapping ) properties of each such class of operations, where:
[PARALLEL] orchestration of units-of-work implies, that any and every work-unit: a) starts and b) gets executed and c) gets finished at the same time, i.e. all get into/out-of [PARALLEL]-section at once and being elaborated at the very same time, not otherwise.
[SERIAL] orchestration of units-of-work implies, that all work-units be processed in a one, static, known, particular order, starting work-unit(s) in such an order, just a (known)-next one after previous one has finished its work - i.e. one-after-another, not otherwise.
[CONCURRENT] orchestration of units-of-work permits to start more than one unit-of-work, if resources and system conditions permit (scheduler priorities obeyed), resulting in unknown order of execution and unknown time of completion, as both the former and the latter depend on unknown externalities (system conditions and (non)-availability of resources, that are/will be needed for a particular work-unit elaboration)
Whereas there is an a-priori known, inherently embedded sense of an ORDER in [SERIAL]-type of processing ( as it was already pre-wired into the units-of-work processing-orchestration-code ), it has no such meaning in either [CONCURRENT], where opportunistic scheduling makes a wished-to-have order an undeterministically random result from the system states, skewed by the coincidence of all other externalities, and the same wished-to-have order is principally singular value in true [PARALLEL] by definition, as all start/execute/finish at-the-same-time - so all units-of-work being executed in [PARALLEL] fashion have no other chance, but be both 1st and last at the same time.
Q : Is there any way I could have work done in parallel and guarantee the print order?
No, unless you intentionally or unknowingly violate the [PARALLEL] orchestration rules and re-enter a re-[SERIAL]-iser logic into the work-units, so as to imperatively enforce any such wished-to-have ordering, that is not known, the less natural for the originally [PARALLEL] work-units' orchestration ( as is a common practice in python - using a GIL-monopolist indoctrinated stepping - as an example of such step )
Q : Have others encountered it before?
Yes. Since 2011, each and every semester this or similar questions reappear here, on Stack Overflow at growing amounts every year.
I'm running into issues with deadlocks during concurrent merge operations (REST API). I have a function that processes text with some metadata, and for each item in the metadata dictionary, I'm performing a merge to add either a node or connect the text node with the metadata[n] node. Issues come up when the message rate is around 500-1000 per second.
In this particular function, there are 11 merges between 6 queries, which go something like this:
q1 = "MERGE (n:N { id: {id} }) ON CREATE SET ... ON MATCH SET "
"WITH . MERGE (...) "
"WITH ., . MERGE (.)-[:some_rel]->(.)"
params = {'the': 'params'}
cypher.execute(q1, params)
if some_condition:
q2 = "MATCH (n:N { id: {id} }) ... "
"WITH n, . MERGE (n)-[:some_rel]->(.)"
params = {'more': 'params'}
cypher.execute(q2, params)
if some_condition2:
q3
...
if some_condition_n:
qn
I'm running the above with Python, via Celery (for those not familiar with Celery, it's a distributed task queue). When the issue first came up, I was executing the above in a single transaction, and had a ton of failures due to deadlock exceptions. My initial thought was simply to implement a distributed blocking lock at the function level with Redis. This, however, causes a bottleneck in my app.
Next, I switched from a single Cypher transaction a few atomic transactions as in the above and removed the lock. This takes care of the bottleneck, as well as greatly reducing the number of deadlock exceptions, but they're still occurring, albeit at the reduced level.
Graph databases aren't really my thing, so I don't have a ton of experience with the in's and out's of Neo4j and Cypher. I have a secondary-index in Redis of the uuid's of existing nodes, so there is a pre-processing step prior to the merge's to try and keep the graph access down. Are there any obvious solutions that I should try? Maybe there's some way to queue the operations on the graph-side, or maybe I'm overlooking some server optimizations? Any advice on where to look would be appreciated. Thanks!
Okay, after thinking about this some more, I realized that the way my queries were executed was inefficient and could do with some refactoring. Since all the queries are within the same general context, there is no reason to execute them all individually, or even no reason to open a transaction and have them executed that way.
Instead, I changed function to go through the conditionals, and concatenate the query strings into one long string, and add the params that I need to the param dictionary. So, now, there's only one execution at the end, with one statement. This also takes out some of the 'MATCH' statements.
This fix doesn't wholly fix the issue, though, as there are still some deadlock exceptions being thrown.
I think I found the issue, mainly that there wasn't an issue to begin with. That is:
The Enterprise version of Neo4j has an alternative lock manager than
the Community version, meant to provide scalable locking on
high-CPU-count machines.
The Enterprise Lock Manager uses a deadlock detection algorithm that
does not require (much) synchronization, which gives it some very
desirable scalability attributes. The drawback is that it may
sometimes detect false-positives. This normally does not happen in
production usage, but becomes evident in stress testing individual
operations. These scenarios see much lower churn in CPU cache
invalidation, which the enterprise lock manager needs to communicate
across cores.
As a deadlock detection error is a safe-to-retry error and the user is
expected to handle these in all application code, since there may be
legitimate deadlocks at any time, this behavior is actually by design
to gain scalability.
I simply caught the exception and then retry after a few seconds and now:
I have a ruby on rails app that takes around 10000 requests per minute.
And some of these requests perform a write to a database table. The maximum amount of connections to the database is 200.
And I would like to know what is more efficient. Writing to an array in cache and saving the data in the background in one operation, or saving each request directly to the database?
Are there any race conditions or performance issues if I write the data to an array in cache?
Are there any better approaches to optimize performance and avoid a database bottleneck?
Sample Code
#...
def self.add_data_message_to_queue(event_id, chat_item)
bucket_name = 'BUCKET_GET_CHAT_' + event_id.to_s
chat_queue = Rails.cache.fetch(bucket_name)
if chat_queue.blank?
chat_queue = []
end
chat_queue.push(chat_item)
Rails.cache.write(bucket_name, chat_queue, expires_in:Integer(30).days)
end
Server: Unicorn (High Concurrency)
Thanks in advance
SOLUTION
According to benchmarks writing to memcache is way more efficient.
Although it is necessary to handle race conditions. According to feedback from the memcachier team.
Test Saving Chat Messages to DB
Same Test - Not Saving Chat Messages to current DB
Response time is way better. The app can serve more requests per minute as well.
Handling Race Conditions
*( Feedback from memcachier team )
There are, in general, two ways to address this in memcache:
Since you're appending to an array, you could instead use memcache's APPEND and PREPEND operations. They are not supported in Rails.cache, but the underlying library, Dalli, supports these commands. Their semantic is that they will append/prepend a string to the existing value. When you fetch the value, you'll get all the strings you "appended" concatenated together, so you'd have to, e.g., separate each element by a semi-colon or something like that to break it into an array.
A more general solution (which works for any data-race conditions) is to use the versioning support in memcache. Specifically, each value in memcache is assigned a version, which is returned on any get requests. Set operations in memcache can take an optional CAS (for compare-and-swap) field, such that the operation will succeed only if the version matches the current value stored. Again, I believe Rails.cache doesn't support this, but Dalli does, through the cas method:
cache.add("bucket_get_chat", [])
while(!cache.cas("bucket_get_chat") {|val| val.push(chat_item)}); end
I use Delphi XE2 along with DISQLite v3 (which is basically a port of SQLite3). I love everything about SQLite3, except the lack of concurrent writing, especially that I extensively rely on multi-threading in this project :(
My profiler made it clear I needed to do something about it, so I decided to use this approach:
Whenever I need to insert a record in DB, Instead of doing an INSERT, I write the SQL query in a special foler, ie.
WriteToFile_Inline(SPECIAL_FOLDER_PATH + '\' + GUID, FileName + '|' + IntToStr(ID) + '|' + Hash + '|' + FloatToStr(ModifDate) + '|' + ...);
I added a timer (in the main app thread) that fires every minute, parse these files and then INSERT the queries using a transaction.
Delete those temporary files at the end.
The result is I have like 500% performance gain. Plus this technique is ACID, as I can always scan the SPECIAL_FOLDER_PATH after a power failure and execute the INSERTs I find.
Despite the good results, I'm not very happy with the method used (hackish to say the least), I keep thinking that if I could have a generics-like with fast lookup access, thread-safe, ACID list, this would be much cleaner (and possibly faster?)
So my question is: do you know anything like that for Delphi XE2?
PS. I trust many of you reading the code above be in shock and will start insulting me at this point! Please be my guest, but if you know a better (ie. faster) ACID approach, please share your thoughts!
Your idea of sending the inserts to a queue, which will rearrange the inserts, and join them via prepared statements is very good. Using a timer in the main thread or a separated thread is up to you. It will avoid any locking.
Do not forget to use a transaction, then commit it every 100/1000 inserts for instance.
About high performance using SQLite3, see e.g. this blog article (and graphic below):
In this graphic, best performance (file off) comes from:
PRAGMA synchronous = OFF
Using prepared statements
Inside a transaction
In WAL mode (especially in concurrency mode)
You may also change the page size, or the journal size, but settings above are the best. See https://stackoverflow.com/search?q=sqlite3+performance
If you do not want to use a background thread, ensure WAL is ON, prepare your statements, use batchs, and regroup your process to release the SQLite3 lock as soon as possible.
The best performance will be achieved by adding a Client-Server layer, just as we did for mORMot.
With files you organized an asynchronous job queue with persistance. It allows you to avoid one-by-one and use batch (records group) approach to insert the records. Comparing one-by-one and batch:
first works in auto-commit mode (probably) for each record, second wraps a batch into a single transaction and gives greatest performance gain.
first prepares an INSERT command each time when you need to insert a record (probably), second once per batch and gives second by value gain.
I dont think, that SQLite concurrency is a problem in your case (at least not the main issue). Because in SQLite a single insert is comparably fast and concurrency performance issues you will get with high workload. Probably similar results you will get with other DBMS, like Oracle.
To improve your batch approach, consider the following:
consider to set journal_mode to WAL and disable shared cache mode.
use a background thread to process your queue. Instead of a fixed time interval (1 min), check SPECIAL_FOLDER_PATH more often. And if the queue has more than X Kb of data, then start processing. Or use a count of queued records and event to notify the thread, that the queue should start processing.
use multy-record prepared INSERT instead of single-record INSERT. You can build an INSERT for 100 records and process your queue data in a single batch, but by 100 record chanks.
consider to write / read a binary field values instead of a text values.
consider to use a set of files with preallocated size.
etc
sqlite3_busy_timeout is pretty inefficient because it doesn't return immediately when the table it's waiting on is unlocked.
I would try creating a critical section (TCriticalSection?) to protect each table. If you enter the critical section before inserting a row and exit it immediately thereafter, you will create better table locks than SQLite provides.
Without knowing your access patterns, though, it's hard to say if this will be faster than batching up a minute's worth of inserts into single transactions.
I have a MVC application which returns 2 types of Json responses from 2 controller methods; AnyRemindersExist() and GetAllUserReminders(). The first returns a boolean, 2nd returns an array, both wrapped as Json.
I have a JavaScript timer checking for calendar reminders against a user. It makes the first call (AnyRemindersExist) to check whether reminders exist and whether the client should then make the 2nd call.
For example, if the result of the Json response is false from the Any() query, it doesn't then make the 2nd controller action which makes a LINQ select call. If there are reminders that exist, it then goes further and then requests them (making use of the LINQ SELECT).
Imagine a system ramped up where 100-1000s users use the system and on the client, every 30-60 seconds a request comes in to load in the reminders. Does this Any() call help in anyway in reducing load on the server?
If you're always going to get the actual values afterwards, then no - it would make more sense to have fewer requests, and just always give the full results. I very much doubt that returning no results is slower than returning an indication that there are no results.
EDIT: tvanfosson's latest comment (at the time of this writing) is worth promoting:
You can really only tell by measuring and I'd only resort to it IFF the performance of the select only option didn't meet the requirements.
That's the most important thing about performance: the value of a guess is much less than the value of test data.
I would say that it depends on how the underlying queries are translated. If the any call is translated into an indexed lookup when the select (perhaps due to a join to get related data) must do some sort of table scan, then it will save some work in the case when there are no reminders to be found. It will cause a little extra work when there are reminders. It might be useful if the majority of the calls don't result in any results.
In the general case, though, I would just select the data and only try to optimize IF that turns out to not be fast enough. The conditions under which it will actually save effort on the server are pretty narrow and might only apply if you hand-craft the SQL rather than depend on your ORM.
Any only checks to see if there is at least one item in the Collection that is being returned. Versus using something like Count > 0 which counts the total amount of items in the collection then yes this is more optimal.
If your AnyRemindersExist method is operating on a similar principle then not calling a second call to the server would reduce your load.
So you are asking if not doing work the application doesn't need to do would reduce the workload on the server?
Of course. How would this answer every be "yes, doing extra work for no reason won't effect the server load".
It ultimately depends on how much faster the Any check is compared to getting the results and how often it will be false.
If the Any call takes near as long as the select then it pretty
much never makes sense.
If the Any call is much faster than the select but 90% of the
time it's true, then it probably isn't worth it (best case you
get 10% improvement, worst case it's actually more work).
If the Any call is much faster than the select and 90% of the
time it's false, then it probably makes sense to check if there
are any before actually getting results.
So the answer is it depends on your specific scenario. Ultimately you're going to need to measure both the relative performance (on different typical loads, maybe some queries are more intensive than others) as well as the frequency that there are no results to return.
Actually it should almost never make sense to check Any in this case.
If Any returns false then you don't need to grab the results.
However this means it would have returned no results anyway, so
unless your Any check is significantly faster than a select
returning 0 results, there's no added benefit here.
On the other hand, if Any returns true, then you'll need to get the
results anyway, so in this case Any is purely additional work done.