XQuery query to delete marked documents sometimes runs indefinitely - grails

Question

I have an XQuery query intended to wipe test documents from the database before each test that is run. Essentially it looks for a certain element to be present as a top level element in a document (called 'forTestOnly') and if it finds it it deletes the document. This query is run before each test in order to ensure the tests don't interfere with one another (we have about 200 tests using this). The exact XQuery is as such:
xquery version "1.0-ml";
import module namespace dls = "http://marklogic.com/xdmp/dls" at "/MarkLogic/dls.xqy";
let $deleteNonManagedDocs := for $testDoc in /*[forTestOnly]
let $testDocUri := fn:base-uri($testDoc)
where fn:not(dls:document-is-managed($testDocUri))
return xdmp:document-delete($testDocUri)
let $deleteManagedDocs := for $testDoc in cts:search(/*[forTestOnly], dls:documents-query())
let $testDocUri := fn:base-uri($testDoc)
return dls:document-delete($testDocUri, fn:false(), fn:false())
return ($deleteManagedDocs, $deleteNonManagedDocs)
While it seems to work fine most of the time, it recently has begun to sporadically spiral out of control. At some point during the test execution it begins to run for a near indefinite amount of time (I usually stop it after 600-700 seconds), most of the time it takes less than a second though. The database used for testing is not large (it has a few basic seed documents but nothing compared to a production database), and typically each test only creates a handful of documents with the 'forTestOnly' (if not less).
The query seems simple enough, and although running it 200 times in relatively quick succession would understandably put a strain on the database I can't imagine it would cause this kind of lagging (the tests are Grails integration tests and the entire execution takes a little over two minutes). Any ideas why the long run time?
As a side note I have verified that when the tests stall it is indeed after the XQuery has begun to run and not before in some sort of test wiring/execution.
Any help is greatly appreciated.

Answer 1

The query might look simple, but it isn't necessarily simple to evaluate. Those dls function calls could be doing anything, so it's tricky to estimate the complexity. The use of DLS also means that we don't know how much version history has to be deleted to delete each document.
One possibility is that you've discovered a bug. It might already be fixed, which is a good reason why you should always report the full version of the software you're using. The answer may be as simple as upgrading to pick up the fix.
Another possibility is that your test suite ends up running all of this work in a single high-level evaluation, so everything's in memory until the end. That could use enough memory to drive the server into swap. That would explain the recent "spiral out of control" behavior. Check the OS and see what it says.
Next, set the group file-log-level=Debug and check ErrorLog.txt while one of these slow events is happening. If you see XDMP-DEADLOCK messages, you may have a problem where two or more copies of this delete query are running at the same time. MarkLogic has automatic deadlock detection and resolution, but it's faster to avoid the deadlock in the first place.
Some logging might also help determine where the time is spent. Something like:
let $deleteNonManagedDocs := for $testDoc in /*[forTestOnly]
let $testDocUri := fn:base-uri($testDoc)
where fn:not(dls:document-is-managed($testDocUri))
return (
xdmp:log(text { 'unmanaged', $testDocUri }),
xdmp:document-delete($testDocUri))
let $deleteManagedDocs := for $testDoc in cts:search(/*[forTestOnly], dls:documents-query())
let $testDocUri := fn:base-uri($testDoc)
let $_ := xdmp:log(text { 'managed', $testDocUri })
return dls:document-delete($testDocUri, fn:false(), fn:false())
return ()
Finally you should also be able to simplify the query a bit. Since you're deleting everything, you can just ignore DLS.
xdmp:document-delete(
cts:uris(
(), (),
cts:element-query(xs:QName('forTestOnly'), cts:and-query(())))
This would be even simpler and more efficient if you set a collection on every test document: xdmp:collection-delete('test-docs').