What is the difference between the getPrimaryResults and getSecondaryResults methods of the RunResults class in JMH?
I haven't verified it on code basis but as I understand it primary results are figures that are produced solely by JMH (the time measurements) whereas secondary results are obtained by means of additional tools like profilers.
UPDATE: I just have seen that the StackProfiler is part of JMH meaning my definition is not completely valid.
UPDATE 2: Meanwhile, Aleksey Shipilev provided a much clearer definition in his comment.
Below you find some lines extracted from a benchmark result. I started the benchmark with '-prof stack'
[java] Iteration 10: 104439194.250 ±(99.9%) 7883662.439 ns/op
[java] ·stack: <delayed till summary>
...
[java] 118162555.108 ±(99.9%) 8317229.030 ns/op [Average]
[java] (min, avg, max) = (102325413.750, 118162555.108, 149303159.000), stdev = 12448837.807
[java] CI (99.9%): [109845326.078, 126479784.138] (assumes normal distribution)
...
[java] Secondary result "·stack":
[java] Stack profiler:
[java] ....[Thread state distributions]....................................................................
[java] 87.3% RUNNABLE
[java] 12.7% WAITING
[java] ....[Thread state: RUNNABLE].................................................................... ....
[java] 82.3% 94.3% sun.management.ThreadImpl.getThreadTotalCpuTime0
As you can see, the results from the stack profiler are marked as Secondary results. Hope that helps a bit.
Related
I've been trying to do stm in R, with codes like,
`kresult <- searchK(myout$documents, myout$vocab,
K=20:25,
prevalence = ~ created_at,
data=myout$meta,
)`
then I face to the error.
Beginning Spectral Initialization
Calculating the gram matrix...
Finding anchor words...
......................
Recovering initialization...
.................................
Initialization complete.
.......................................................................
Completed E-Step (15 seconds).
Error in .sortCsparse(.Call(dtCMatrix_sparse_solve, a, b)) :
'R_Realloc' could not re-allocate memory (2632458240 bytes)
Please tell me how can I fix this problem...
Thanks.
I closed every other applications and rebooted but there was no change.
I am working on extracting graph embeddings with training GraphSage algorihm. I am working on a large graph consisting of (82,339,589) nodes and (219,521,164) edges. When I checked with ":queries" command the query is listed as running. Algorithm started in 6 days ago. When I look the logs with "docker logs xxx" the last logs listed as
2021-12-01 12:03:16.267+0000 INFO Relationship Store Scan (RelationshipScanCursorBasedScanner): Imported 352,492,468 records and 0 properties from 16247 MiB (17,036,668,320 bytes); took 59.057 s, 5,968,663.57 Relationships/s, 275 MiB/s (288,477,487 bytes/s) (per thread: 1,492,165.89 Relationships/s, 68 MiB/s (72,119,371 bytes/s))
2021-12-01 12:03:16.269+0000 INFO [neo4j.BoltWorker-3 [bolt] [/10.0.0.6:56143] ] LOADING
INFO [neo4j.BoltWorker-3 [bolt] [/10.0.0.6:56143] ] LOADING Actual
memory usage of the loaded graph: 8602 MiB
INFO [neo4j.BoltWorker-3 [bolt] [/10.0.0.6:64076] ] GraphSageTrain ::
Start
There is a way to see detailed logs about training process. Is it normal for taking 6 days for graphs with shared sizes ?
It is normal for GraphSAGE to take a long time compared to FastRP or Node2Vec. Starting in GDS 1.7, you can use
CALL gds.beta.listProgress(jobId: String)
YIELD
jobId,
taskName,
progress,
progressBar,
status,
timeStarted,
elapsedTime
If you call without passing in a jobId, it will return a list of all running jobs. If you call with a jobId, it will give you details about a running job.
This query will summarize the details for job 03d90ed8-feba-4959-8cd2-cbd691d1da6c.
CALL gds.beta.listProgress("03d90ed8-feba-4959-8cd2-cbd691d1da6c")
YIELD taskName, status
RETURN taskName, status, count(*)
Here's the documentation for progress logging. The system monitoring procedures might also be helpful to you.
EDIT*: After all it turned out that this is not causing the slow import. Nevertheless the answer given explains a better way to implement different densities with one material. So I'll let the question exist. (Slow import was caused by running the scripts from the abaqus PDE and not using 'Run script' from the file menu. special thanks to droooze for finding the problem)
I'm trying to optimize the porosity distribution of a certain material. Therefor I'm performing abaqus FEA simulations with +-500 different materials in one part. The simulation itself only takes about 40 seconds, but reading the input file takes more than 3 minutes. (I used a python script to generate the inp file)
I'm using these commands to generate my materials in the input file:
*SOLID SECTION, ELSET = ES_Implant_MAT0 ,MATERIAL=Implant_material0
*ELSET, ELSET=ES_Implant_MAT336
6,52,356,376,782,1793,1954,1984,3072
*MATERIAL, NAME = Implant_material0
*DENSITY
4.43
*ELASTIC
110000, 0.3
Any idea why this is so slow and is there a more efficient way to do this to reduce the load input file time?
If your ~500 materials are all of the same kind (e.g. all linear elastic isotropic mass density), then you can collapse it all into one material then define a distribution table which distributes these materials directly onto the instance element label.
Syntax:
(somewhere in the Part definition, under section)
*SOLID SECTION, ELSET = ES_Implant_MAT0 ,MATERIAL=Implant_material0
(somewhere in the Assembly definition; part= should reference the name of the part above)
**
**
** ASSEMBLY
**
*Assembly, name=Assembly
**
*Instance, name=myinstance, part=mypart
*End Instance
**
*Elset, elset=ES_Implant_MAT0, instance=myinstance
1,2,...
(somewhere in the Materials definition; see Abaqus Keywords Reference Guide for the keywords *DISTRIBUTION TABLE and *DISTRIBUTION)
**
** MATERIALS
**
*DISTRIBUTION TABLE, NAME=IMPLANT_MATERIAL0_ELASTIC_TABLE
MODULUS,RATIO
*DISTRIBUTION, NAME=Implant_material0_elastic, LOCATION=element, TABLE=IMPLANT_MATERIAL0_ELASTIC_TABLE
,110000,0.3 # First line is some default value
myinstance.1,110000,0.3 # syntax: instance name [dot] instance element label
myinstance.2,110000,0.3 # these elements currently use the material properties assigned to `ELSET = ES_Implant_MAT0`. You can define the material properties belonging to other element sets in this same table, making sure you reference the element label correctly.
...
*DISTRIBUTION TABLE, NAME=IMPLANT_MATERIAL0_DENSITY_TABLE
DENSITY
*DISTRIBUTION, NAME=Implant_material0_density, LOCATION=element, TABLE=IMPLANT_MATERIAL0_DENSITY_TABLE
,4.43 # Default value
myinstance.1,4.43
myinstance.2,4.43
...
*Material, name=Implant_material0
*Elastic
Implant_material0_elastic # Distribution name
*Density
Implant_material0_density # Distribution name
In the aerospike set we have four bins userId, adId, timestamp, eventype and the primary key is userId:timestamp. Secondary Index is created on userId to get all the records for a particular user and the resulted records are passed to stream udf. On our client side till 500 qps the aerospike query latency is reasonable and mean latency is in microseconds but as soon as we increase the qps above 500 the aerospike query latency shoots up (around ~ 10 ms)
message that we see on the client side is attached below:
Name: Aerospike-13780
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#41aa29a4
Total blocked: 0 Total waited: 554,450
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
com.aerospike.client.lua.LuaInputStream.read(LuaInputStream.java:38)
com.aerospike.client.lua.LuaStreamLib$read.call(LuaStreamLib.java:60)
org.luaj.vm2.LuaClosure.execute(Unknown Source)
org.luaj.vm2.LuaClosure.onInvoke(Unknown Source)
org.luaj.vm2.LuaClosure.invoke(Unknown Source)
org.luaj.vm2.LuaClosure.execute(Unknown Source)
org.luaj.vm2.LuaClosure.onInvoke(Unknown Source)
org.luaj.vm2.LuaClosure.invoke(Unknown Source)
org.luaj.vm2.LuaValue.invoke(Unknown Source)
com.aerospike.client.lua.LuaInstance.call(LuaInstance.java:128)
com.aerospike.client.query.QueryAggregateExecutor.runThreads(QueryAggregateExecutor.java:104)
com.aerospike.client.query.QueryAggregateExecutor.run(QueryAggregateExecutor.java:77)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
below is the lua file:
function ad_count(stream)
local function map_function(record)
local result = map()
result["adId"] = record["adId"]
result["timestamp"] = record["timestamp"]
return result
end
local function add_fn(aggregate, record)
local ad_id = record["adId"]
local map_result = aggregate[ad_id]
local l = list()
if map_result == null then
map_result = l
end
list.append(map_result, record["timestamp"])
aggregate[ad_id] = map_result
return aggregate
end
local m = map()
return stream:map(map_function):aggregate(m, add_fn)
end
There are 2 nodes and the server is hosted in AWS with T2.large instance type.
transaction-queues=8;transaction-threads-per-queue=8;transaction-duplicate-threads=0;transaction-pending-limit=20;migrate-threads=1;migrate-xmit-priority=40;migrate-xmit-sleep=500;migrate-read-priority=10;migrate-read-sleep=500;migrate-xmit-hwm=10;migrate-xmit-lwm=5;migrate-max-num-incoming=256;migrate-rx-lifetime-ms=60000;proto-fd-max=15000;proto-fd-idle-ms=60000;proto-slow-netio-sleep-ms=1;transaction-retry-ms=1000;transaction-max-ms=1000;transaction-repeatable-read=false;dump-message-above-size=134217728;ticker-interval=10;microbenchmarks=false;storage-benchmarks=false;ldt-benchmarks=false;scan-max-active=100;scan-max-done=100;scan-max-udf-transactions=32;scan-threads=4;batch-index-threads=4;batch-threads=4;batch-max-requests=5000;batch-max-buffers-per-queue=255;batch-max-unused-buffers=256;batch-priority=200;nsup-delete-sleep=100;nsup-period=120;nsup-startup-evict=true;paxos-retransmit-period=5;paxos-single-replica-limit=1;paxos-max-cluster-size=32;paxos-protocol=v3;paxos-recovery-policy=manual;write-duplicate-resolution-disable=false;respond-client-on-master-completion=false;replication-fire-and-forget=false;info-threads=16;allow-inline-transactions=true;use-queue-per-device=false;snub-nodes=false;fb-health-msg-per-burst=0;fb-health-msg-timeout=200;fb-health-good-pct=50;fb-health-bad-pct=0;auto-dun=false;auto-undun=false;prole-extra-ttl=0;max-msgs-per-type=-1;service-threads=40;fabric-workers=16;pidfile=/var/run/aerospike/asd.pid;memory-accounting=false;udf-runtime-gmax-memory=18446744073709551615;udf-runtime-max-memory=18446744073709551615;sindex-builder-threads=4;sindex-data-max-memory=18446744073709551615;query-threads=6;query-worker-threads=15;query-priority=10;query-in-transaction-thread=0;query-req-in-query-thread=0;query-req-max-inflight=100;query-bufpool-size=256;query-batch-size=100;query-priority-sleep-us=1;query-short-q-max-size=500;query-long-q-max-size=500;query-rec-count-bound=18446744073709551615;query-threshold=10;query-untracked-time-ms=1000;pre-reserve-qnodes=false;service-address=0.0.0.0;service-port=3000;mesh-address=10.0.1.80;mesh-port=3002;reuse-address=true;fabric-port=3001;fabric-keepalive-enabled=true;fabric-keepalive-time=1;fabric-keepalive-intvl=1;fabric-keepalive-probes=10;network-info-port=3003;enable-fastpath=true;heartbeat-mode=mesh;heartbeat-protocol=v2;heartbeat-address=10.0.1.80;heartbeat-port=3002;heartbeat-interval=150;heartbeat-timeout=10;enable-security=false;privilege-refresh-period=300;report-authentication-sinks=0;report-data-op-sinks=0;report-sys-admin-sinks=0;report-user-admin-sinks=0;report-violation-sinks=0;syslog-local=-1;enable-xdr=false;xdr-namedpipe-path=NULL;forward-xdr-writes=false;xdr-delete-shipping-enabled=true;xdr-nsup-deletes-enabled=false;stop-writes-noxdr=false;reads-hist-track-back=1800;reads-hist-track-slice=10;reads-hist-track-thresholds=1,8,64;writes_master-hist-track-back=1800;writes_master-hist-track-slice=10;writes_master-hist-track-thresholds=1,8,64;proxy-hist-track-back=1800;proxy-hist-track-slice=10;proxy-hist-track-thresholds=1,8,64;udf-hist-track-back=1800;udf-hist-track-slice=10;udf-hist-track-thresholds=1,8,64;query-hist-track-back=1800;query-hist-track-slice=10;query-hist-track-thresholds=1,8,64;query_rec_count-hist-track-back=1800;query_rec_count-hist-track-slice=10;query_rec_count-hist-track-thresholds=1,8,64
We even changed the following config parameters but it further increased the latency:
query-batch-size=1000
query-short-q-max-size=100000
query-long-q-max-size=100000
query-threads=28
query-worker-threads=400
query-req-max-inflight=1000
Linking back to the same question on the Aerospike community forum: https://discuss.aerospike.com/t/high-query-latency/2769/3
You're mainly bumping against the limits of that specific instance. We do not recommend using the t2 family. Please review the recommendations section of Aerospike's Amazon deployment guide. You should consider instances in the m3, c3, c4, r3, or i2 instance families.
A quick note on your configuration:
transaction-queues should be tuned to the number of cores. You have it set to 8, and a t2.large has 2 cores.
transaction-threads-per-queue is set too high for this instance type.
query-threads and query-worker-threads are both set too high for this instance.
Further, your queries involves a stream UDF so they will be slower than a regular query. UDFs are useful but you need to consider the context for which they're a good fit.
My suggestion is that you can probably model this differently, and skip the secondary index and UDF.
Does anyone know if this is possible with SpecFlow? In the spirit of not having more than one assertion per test, I was hoping SpecFlow would treat each "Then" as a separate test.
I have a Scenario with multiple "Then" steps that looks something like this (snippet):
When a summary for day <day number> is requested
Then the summary well id should be "134134"
And the summary well name should be "My Well --oops!"
And the summary violated rules should be <violations>
And the summary variance should be <variance>
Examples:
| day number | violations | variance |
| 0 | BadTotal,HpOnLp | -33 |
| 3 | BadTotal | -133.33 |
| 5 | | 0 |
The second assertion "My Well --oops!" should fail. What I want is for SpecFlow to test the assertions that follow
I get:
When a summary for day 0 is requested
-> done: DprSummarySteps.WhenASummaryForDayIsRequested(0) (0.0s)
Then the summary well id should be "134134"
-> done: DprSummarySteps.ThenTheSummaryWellIdShouldBe("134134") (0.0s)
And the summary well name should be "My Well --oops!"
-> error: Assert.AreEqual failed. Expected:<My Well --oops!>. Actual:<My Well>.
And the summary violated rules should be BadTotal,HpOnLp
-> skipped because of previous errors
And the summary variance should be -33
-> skipped because of previous errors
Assert.AreEqual failed. Expected:<My Well --oops!>. Actual:<My Well>.
What I want:
When a summary for day 0 is requested
-> done: DprSummarySteps.WhenASummaryForDayIsRequested(0) (0.0s)
Then the summary well id should be "134134"
-> done: DprSummarySteps.ThenTheSummaryWellIdShouldBe("134134") (0.0s)
And the summary well name should be "My Well --oops!"
-> error: Assert.AreEqual failed. Expected:<My Well --oops!>. Actual:<My Well>.
And the summary violated rules should be BadTotal,HpOnLp
-> done: DprSummarySteps.ThenTheViolatedRulesShouldBe("BadTotal,HpOnLp") (0.0s)
And the summary variance should be -33
-> done: DprSummarySteps.ThenTheVarianceShouldBe(-33) (0.0s)
Assert.AreEqual failed. Expected:<My Well --oops!>. Actual:<My Well>.
I don't believe that you can do this but my question to you would be why do you want that? With any unit test it will fail at the first assertion. you wouldn't expect a test to continue execution after an assertion failed and this is no different. Surely knowing that the test failed for some reason is enough. In this specific case you might be able to make separate independent assertions which provide useful info, but in the general case assertions that come after one that has failed might be completely meaningless.
If you want to have each assertion independent of the others then you need to break it into several scenarios with each of you current And steps as its own Then step. You might be able to use a Background step to do the common setup.
I'm not sure that this will help you though as it seems like your assertions are all related to the examples so you end up needing to repeat the examples.
You can do this now if you upgrade to Specflow 2.4 and are using Nunit, see something like https://github.com/nunit/docs/wiki/Multiple-Asserts
Assert.Multiple(() =>
{
Assert.AreEqual(5.2, result.RealPart, "Real part");
Assert.AreEqual(3.9, result.ImaginaryPart, "Imaginary part");
});
I have identified one way to do this. So what happens internally in specflow, before running any step, the TestExecutionEngine checks the LastExecutionStatus of Scenario, and it does not proceed if it is not OK.. So what we can do is, in [AfterStep] hook, add the following line :
typeof(ScenarioContext).GetProperty("ScenarioExecutionStatus").SetValue(this.ScenarioContext, ScenarioExecutionStatus.OK);
Replace this.ScenarioContext with whatever object with type ScenarioContext exist. This will make the current status as OK and will allow you to proceed to next step.
Keep in mind this won't allow you to catch all assert failures in a single step.