I'm curious about yielding during a transaction. Will it be interrupted immediately? Or will other fibers be able to read changes that have not yet been committed?
I've overlooked the documentation but didn't see it.
Currently, there is a way to yield in a transaction. This is because there is now a fully functional transaction manager for both memtx and vinyl engines.
Here's a working example:
local fiber = require('fiber')
-- If you just use box.cfg{}, this app will fail
box.cfg{memtx_use_mvcc_engine=true}
--box.cfg{}
box.schema.space.create('account', {if_not_exists=true})
box.space.account:format({ {name='id',type='unsigned'},
{name='first_name',type='string'},
{name='last_name',type='string'},
})
box.begin()
box.space.account:put({2, "John", "Doe"})
print("sleeping")
fiber.sleep(1)
print("woke up")
box.space.account:put({3, "Ivan", "Ivanov"})
box.commit()
os.exit(0)
Note the memtx_use_mvcc_engine option. It enables the transaction engine. Without this option, a yield inside the transaction will result in immediate rollback.
The new transaction manager will also make changes inside the transaction to be invisible to other fibers and transactions before it is committed. And in case of conflict (parallel changes to the same data), the transaction will be rolled back.
Related
I am trying to synchronize a resource with spring webClient:
this.semaphore.acquire()
webClient
.post()
.uri("/a")
.bodyValue(payload)
.retrieve()
.bodyToMono(String.class)
// release
.doFinally(st -> this.semaphore.release())
.switchIfEmpty(Mono.just("a"))
.onErrorResume(Exception.class, e -> Mono.empty())
.doOnNext()
.subscribe();
Is doFinally sufficient to handle the release?
If not, what are the "escape" points?
This will clean up your resources if your mono is cancelled, completes, or errors out, which are all the ways in which a mono can end.
However, a Mono does not necessarily have to end and the doFinally hook will not be executed.
So it depends on how your webClient is configured in cases where the external api fails to respond: Normally, there should be a timeout and a maximum number of retries. In that case, your code should be correct.
NOTE: the release may not happen on the same thread as the acquire. Depending on the resource, this might actually be a problem. For example, a ReentrantReadWriteLock has semantics that it is owned by the thread that created it. I do not know if this problem exists with your semaphore.
I'm using Delphi 10.4 with a Firebird-Database (3) and FireDac. I have a general question about transaction handling.
Please have a look at the following pseudo code:
write_transaction := TFDTransaction.Create;
write_connection := TFDConnection.Create;
write_transaction.Connection := write_connection;
write_connection.StartTransaction;
There is a connection assigned to a transaction. Then I start working with:
write_transaction.StartTransaction;
...
write_transaction.Commit;
But when I call write_connection.StartTransaction what will happen? So my questions...
Will there be started a transaction for the write_transaction, as if write_transaction.StartTransaction is called? Or will there be started a completely different transaction?
Will there be started a transaction for the write_transaction, as if
write_transaction.StartTransaction is called?
Yes, this is the same. The TFDTransaction component wraps the transaction control functionality into a component. Practically, it offers the same transaction functionality as TFDConnection, but allows you to group commands and datasets by linking them to a specific transaction object.
I have an XQuery query intended to wipe test documents from the database before each test that is run. Essentially it looks for a certain element to be present as a top level element in a document (called 'forTestOnly') and if it finds it it deletes the document. This query is run before each test in order to ensure the tests don't interfere with one another (we have about 200 tests using this). The exact XQuery is as such:
xquery version "1.0-ml";
import module namespace dls = "http://marklogic.com/xdmp/dls" at "/MarkLogic/dls.xqy";
let $deleteNonManagedDocs := for $testDoc in /*[forTestOnly]
let $testDocUri := fn:base-uri($testDoc)
where fn:not(dls:document-is-managed($testDocUri))
return xdmp:document-delete($testDocUri)
let $deleteManagedDocs := for $testDoc in cts:search(/*[forTestOnly], dls:documents-query())
let $testDocUri := fn:base-uri($testDoc)
return dls:document-delete($testDocUri, fn:false(), fn:false())
return ($deleteManagedDocs, $deleteNonManagedDocs)
While it seems to work fine most of the time, it recently has begun to sporadically spiral out of control. At some point during the test execution it begins to run for a near indefinite amount of time (I usually stop it after 600-700 seconds), most of the time it takes less than a second though. The database used for testing is not large (it has a few basic seed documents but nothing compared to a production database), and typically each test only creates a handful of documents with the 'forTestOnly' (if not less).
The query seems simple enough, and although running it 200 times in relatively quick succession would understandably put a strain on the database I can't imagine it would cause this kind of lagging (the tests are Grails integration tests and the entire execution takes a little over two minutes). Any ideas why the long run time?
As a side note I have verified that when the tests stall it is indeed after the XQuery has begun to run and not before in some sort of test wiring/execution.
Any help is greatly appreciated.
The query might look simple, but it isn't necessarily simple to evaluate. Those dls function calls could be doing anything, so it's tricky to estimate the complexity. The use of DLS also means that we don't know how much version history has to be deleted to delete each document.
One possibility is that you've discovered a bug. It might already be fixed, which is a good reason why you should always report the full version of the software you're using. The answer may be as simple as upgrading to pick up the fix.
Another possibility is that your test suite ends up running all of this work in a single high-level evaluation, so everything's in memory until the end. That could use enough memory to drive the server into swap. That would explain the recent "spiral out of control" behavior. Check the OS and see what it says.
Next, set the group file-log-level=Debug and check ErrorLog.txt while one of these slow events is happening. If you see XDMP-DEADLOCK messages, you may have a problem where two or more copies of this delete query are running at the same time. MarkLogic has automatic deadlock detection and resolution, but it's faster to avoid the deadlock in the first place.
Some logging might also help determine where the time is spent. Something like:
let $deleteNonManagedDocs := for $testDoc in /*[forTestOnly]
let $testDocUri := fn:base-uri($testDoc)
where fn:not(dls:document-is-managed($testDocUri))
return (
xdmp:log(text { 'unmanaged', $testDocUri }),
xdmp:document-delete($testDocUri))
let $deleteManagedDocs := for $testDoc in cts:search(/*[forTestOnly], dls:documents-query())
let $testDocUri := fn:base-uri($testDoc)
let $_ := xdmp:log(text { 'managed', $testDocUri })
return dls:document-delete($testDocUri, fn:false(), fn:false())
return ()
Finally you should also be able to simplify the query a bit. Since you're deleting everything, you can just ignore DLS.
xdmp:document-delete(
cts:uris(
(), (),
cts:element-query(xs:QName('forTestOnly'), cts:and-query(())))
This would be even simpler and more efficient if you set a collection on every test document: xdmp:collection-delete('test-docs').
I've a rails app that I'm crashing on purpose.. it's local and I'm just hitting ctrl + c and killing it mid way through processing records..
To my mind the records in the block shouldn't have been committed.. Is this a postgres "error" or a rails "error", or a dave ERROR?
ActiveRecord::Base.transaction do
UploadStage.where("id in (#{ids.join(',')})").update_all(:status => 2);
records.each do |record|
record.success = process_line(record.id, klas, record.hash_value).to_s[0..250]
record.status = 1000
record.save();
end
end
I generate my ids by reading out all the records where the status is 1.
Nothing but this function sets the status to 1000..
If the action crashes for what ever reason, I'd expect there to be no records in the database with status = 2...
This is not what I'm seeing though. Half the records have status 1000, the other half have status 2.. .
Am I missing something?
How can I make sure there are no 2's if the app crashes?
EDIT:
I found this link http://coderrr.wordpress.com/2011/05/03/beware-of-threadkill-or-your-activerecord-transactions-are-in-danger-of-being-partially-committed/
As I suspected and as confirmed by dave's update, it looks like ActiveRecord will commit a half-finished transaction under some circumstances when you kill a thread. Woo, safe! See dave's link for detailed explanation and mitigation options.
If you're simulating hard crash (host OS crash or plug-pull), control-C is absolutely not the right approach. Use Control-\ to send a SIGQUIT, which is generally not handled, or use kill -KILL to hard-kill the process with no opportunity to do cleanup. Control-C sends SIGINT which is a gentle signal that's usually attached to a clean shutdown handler.
In general, if you're debugging issues like this, you should enable detailed query logging and see what Rails is doing. Use log_statement = 'all' in postgresql.conf then examine the PostgreSQL logs.
I wrote a admin script that tails a heroku log and every n seconds, it summarizes averages and notifies me if i cross a certain threshold (yes I know and love new relic -- but I want to do custom stuff).
Here is the entire script.
I have never been a master of IO and threads, I wonder if I am making a silly mistake. I have a couple of daemon threads that have while(true){} which could be the culprit. For example:
# read new lines
f = File.open(file, "r")
f.seek(0, IO::SEEK_END)
while true do
select([f])
line = f.gets
parse_heroku_line(line)
end
I use one daemon to watch for new lines of a log, and the other to periodically summarize.
Does someone see a way to make it less processor-intensive?
This probably runs hot because you never really block while reading from the temporary file. IO::select is a thin layer over POSIX select(2). It looks like you're trying to block until the file is ready for reading, but select(2) considers EOF to be ready ("a file descriptor is also ready on end-of-file"), so you always return right away from select then call gets which returns nil at EOF.
You can get a truer EOF reading and nice blocking behavior by avoiding the thread which writes to the temp file and instead using IO::popen to fork the %x[heroku logs --ps router --tail --app pipewave-cedar] log tailer, connected to a ruby IO object on which you can loop over gets, exiting when gets returns nil (indicating the log tailer finished). gets on the pipe from the tailer will block when there's nothing to read and your script will only run as hot as it takes to do your line parsing and reporting.
EDIT: I'm not set up to actually try your code, but you should be able to replace the log tailer thread and your temp file read loop with this code to get the behavior described above:
IO.popen( %w{ heroku logs --ps router --tail --app my-heroku-app } ) do |logf|
while line = logf.gets
parse_heroku_line(line) if line =~ /^/
end
end
I also notice your reporting thread does not do anything to synchronize access to #total_lines, #total_errors, etc. So, you have some minor race conditions where you can get inconsistent values from the instance vars that parse_heroku_line method updates.
select is about whether a read would block. f is just a plain old file, so you when get to the end reads don't block, they just return nil instantly. As a result select returns instantly rather than waiting for something to be appending to the file as I assume you're expecting. Because of this you're sitting in a tight busy loop, so high cpu is to be expected.
If you are at eof (you could either check f.eof? or whether gets returns nil), then you could either start sleeping (perhaps with some sort of back off) or use something like listen to be notified of filesystem changes