I am having trouble generating a useful call-graph using opreport. The results all come out completely flat with each function being called only by itself and then calling only itself, somewhat confusingly.
-------------------------------------------------------------------------------
17875174 17875174 99.9325 99.9325 myProg.x FTL::ReclaimWriteStateMachine()
17875173 17875173 9.1289 9.1289 myProg.x FTL::ReclaimWriteStateMachine()
17875174 17875174 49.9925 49.9925 myProg.x FTL::ReclaimWriteStateMachine()
17875173 35750347 49.9925 99.9850 myProg.x FTL::ReclaimWriteStateMachine() [self]
-------------------------------------------------------------------------------
14133500 14133500 100.000 100.000 myProg.x Superblock::PageDataStore::ValidUnits() const
14133500 32008673 7.2180 16.3469 myProg.x Superblock::PageDataStore::ValidUnits() const
14133500 14133500 49.9739 49.9739 myProg.x Superblock::PageDataStore::ValidUnits() const
14133500 28267000 49.9739 99.9479 myProg.x Superblock::PageDataStore::ValidUnits() const [self]
-------------------------------------------------------------------------------
e.t.c
I have read this documentation, to no avail.
I am running operf -g ./myProg.x to generate the profiling information, and opreport -lact 1 to view the callgraph data.
I am also seeing this error a lot warning: dropping hyperspace sample at offset, but as far as I am aware that is not related to my call-graph issue.
I have checked the output for the profiling of ls by operf -g ls -la resulting in
-------------------------------------------------------------------------------
1 1 4.1667 4.1667 ld-2.17.so _dl_lookup_symbol_x
1 2 4.1667 8.3333 ld-2.17.so _dl_setup_hash
1 3 4.1667 12.5000 ld-2.17.so _dl_load_cache_lookup
1 4 4.1667 16.6667 ld-2.17.so access
1 5 4.1667 20.8333 libc-2.17.so brk
1 6 4.1667 25.0000 libc-2.17.so _itoa_word
1 7 4.1667 29.1667 libc-2.17.so __GI_strncmp
1 8 4.1667 33.3333 libc-2.17.so __xstat
1 9 4.1667 37.5000 libc-2.17.so __lxstat
1 10 4.1667 41.6667 libc-2.17.so mmap64
1 11 4.1667 45.8333 libc-2.17.so xdr_u_long
1 12 4.1667 50.0000 libpthread-2.17.so __pthread_initialize_minimal
2 14 8.3333 58.3333 ld-2.17.so mprotect
2 16 8.3333 66.6667 libc-2.17.so __sendto_nocancel
3 19 12.5000 79.1667 libc-2.17.so __getdents
5 24 20.8333 100.000 ld-2.17.so mmap
31 31 60.7843 60.7843 no-vmlinux /no-vmlinux
31 31 100.000 100.000 no-vmlinux /no-vmlinux [self]
-------------------------------------------------------------------------------
3 34 5.8824 66.6667 ls /usr/bin/ls
3 3 100.000 100.000 ls /usr/bin/ls [self]
-------------------------------------------------------------------------------
e.t.c
leading me to believe that the issue isn't with my installation of oprofile, will try to tweak how I build my excecutable
Related
I'm trying to detect a memory leak location for a certain process via Windbg, and have come across a strange problem.
Using windbg, I've created 2 memory dump snapshots - one before and one after the leak, that showed an increase of around 20MBs (detected via Performance Monitor - private bytes). it shows that there is indeed a similar size difference in one of the heaps before and after the leak (Used with the command !heap -s):
Before:
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-----------------------------------------------------------------------------
03940000 08000002 48740 35312 48740 4372 553 9 5 2d LFH
External fragmentation 12 % (553 free blocks)
03fb0000 08001002 7216 3596 7216 1286 75 4 8 0 LFH
External fragmentation 35 % (75 free blocks)
05850000 08001002 60 16 60 5 2 1 0 0
...
After:
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-----------------------------------------------------------------------------
03940000 08000002 64928 55120 64928 6232 1051 26 5 51 LFH
External fragmentation 11 % (1051 free blocks)
03fb0000 08001002 7216 3596 7216 1236 73 4 8 0 LFH
External fragmentation 34 % (73 free blocks)
05850000 08001002 60 16 60 5 2 1 0 0
...
See the first Heap (03940000) - there is a difference in committed KBs of around 55120 - 35312 = 19808 KB = 20.2 MB.
However, when I inspected that heap with (!heap -stat -h 03940000), it displays the following for both dump files:
size #blocks total ( %) (percent of total busy bytes)
3b32 1 - 3b32 (30.94)
1d34 1 - 1d34 (15.27)
880 1 - 880 (4.44)
558 1 - 558 (2.79)
220 1 - 220 (1.11)
200 2 - 400 (2.09)
158 1 - 158 (0.70)
140 2 - 280 (1.31)
...(rest of the lines show no difference)
size #blocks total ( %) (percent of total busy bytes)
3b32 1 - 3b32 (30.95)
1d34 1 - 1d34 (15.27)
880 1 - 880 (4.44)
558 1 - 558 (2.79)
220 1 - 220 (1.11)
200 2 - 400 (2.09)
158 1 - 158 (0.70)
140 2 - 280 (1.31)
...(rest of the lines show no difference)
As you can see, there is hardly a difference between the two, despite the abovementioned 20MB size difference.
Is there an explanation for that?
Note: I have also inspected the Unmanaged memory using UMDH - there wasn't a noticeable size difference there.
I have an iOS app developed in Xcode. When running my app in the simulator, I'm getting some random hexadecimal output in the console log when navigating to one particular view controller (example below). It does not happen when navigating to other controllers, and I don't see anything jumping out at me that I'm doing differently in this instance, and there are no apparent effects in the simulator of anything being wrong.
Has anyone seen this, can you lend any direction as to what is causing it, or at the very least tell if this is cause for alarm?
The output does not have a timestamp or my app name as a prefix like an NSLog output such as 2020-03-17 14:50:54.387749-0400 AppName. The output is consistent every time with the same information being logged.
0000000A: 0100 4 4 800
00000016: 0101 4 4 800
00000022: 0102 3 6 110
0000002E: 011A 5 8 116
0000003A: 011B 5 8 124
00000046: 0128 3 2 2
00000052: 0131 2 12 132
0000005E: 0132 2 20 144
000000A6: 0100 4 4 256
000000B2: 0101 4 4 256
000000BE: 0102 3 6 266
000000CA: 0103 3 2 6
000000D6: 0106 3 2 6
000000E2: 0115 3 2 3
000000EE: 0201 4 4 272
000000FA: 0202 4 4 4506
0000000A: 0100 4 4 800
00000016: 0101 4 4 800
00000022: 0102 3 6 110
0000002E: 011A 5 8 116
0000003A: 011B 5 8 124
00000046: 0128 3 2 2
00000052: 0131 2 12 132
0000005E: 0132 2 20 144
000000A6: 0100 4 4 256
000000B2: 0101 4 4 256
000000BE: 0102 3 6 266
000000CA: 0103 3 2 6
000000D6: 0106 3 2 6
000000E2: 0115 3 2 3
000000EE: 0201 4 4 272
000000FA: 0202 4 4 4506
0000000A: 0100 4 4 800
00000016: 0101 4 4 800
00000022: 0102 3 6 110
0000002E: 011A 5 8 116
0000003A: 011B 5 8 124
00000046: 0128 3 2 2
00000052: 0131 2 12 132
0000005E: 0132 2 20 144
000000A6: 0100 4 4 256
000000B2: 0101 4 4 256
000000BE: 0102 3 6 266
000000CA: 0103 3 2 6
000000D6: 0106 3 2 6
000000E2: 0115 3 2 3
000000EE: 0201 4 4 272
000000FA: 0202 4 4 4506
0000000A: 0100 4 4 800
00000016: 0101 4 4 800
00000022: 0102 3 6 110
0000002E: 011A 5 8 116
0000003A: 011B 5 8 124
00000046: 0128 3 2 2
00000052: 0131 2 12 132
0000005E: 0132 2 20 144
000000A6: 0100 4 4 256
000000B2: 0101 4 4 256
000000BE: 0102 3 6 266
000000CA: 0103 3 2 6
000000D6: 0106 3 2 6
000000E2: 0115 3 2 3
000000EE: 0201 4 4 272
000000FA: 0202 4 4 4506
I basically want the same thing as this OP:
Is there a J idiom for adding to a list until a certain condition is met?
But I cant get the answers to work with OP's function or my own.
I will rephrase the question and write about the answers at the bottom.
I am trying to create a function that will return a list of fibonacci numbers less than 2.000.000. (without writing "while" inside the function).
Here is what i have tried:
First, i picked a way to culculate fibonacci numbers from this site:
https://code.jsoftware.com/wiki/Essays/Fibonacci_Sequence
fib =: (i. +/ .! i.#-)"0
echo fib i.10
0 1 1 2 3 5 8 13 21 34
Then I made an arbitrary list I knew was larger than what I needed. :
fiblist =: (fib i.40) NB. THIS IS A BAD SOLUTION!
Finally, I removed the numbers that were greater than what I needed:
result =: (fiblist < 2e6) # fiblist
echo result
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765 10946 17711 28657 46368 75025 121393 196418 317811 514229 832040 1.34627e6
This gets the right result, but is there a way to avoid using some arbitrary number like
40 in "fib i.40" ?
I would like to write a function, such that "func 2e6" returns the list of fibonacci numbers below 2.000.000. (without writing "while" inside the function).
echo func 2e6
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765 10946 17711 28657 46368 75025 121393 196418 317811 514229 832040 1.34627e6
here are the answers from the other question:
first answer:
2 *^:(100&>#:])^:_"0 (1 3 5 7 9 11)
128 192 160 112 144 176
second answer:
+:^:(100&>)^:(<_) ] 3
3 6 12 24 48 96 192
As I understand it, I just need to replace the functions used in the answers, but i dont see how
that can work. For example, if I try:
echo (, [: +/ _2&{.)^:(100&>#:])^:_ i.2
I get an error.
I approached it this way. First I want to have a way of generating the nth Fibonacci number, and I used f0b from your link to the Jsoftware Essays.
f0b=: (-&2 +&$: -&1) ^: (1&<) M.
Once I had that I just want to put it into a verb that will check to see if the result of f0b is less than a certain amount (I used 1000) and if it was then I incremented the input and went through the process again. This is the ($:#:>:) part. $: is Self-Reference. The right 0 argument is the starting point for generating the sequence.
($:#:>: ^: (1000 > f0b)) 0
17
This tells me that the 17th Fibonacci number is the largest one less than my limit. I use that information to generate the Fibonacci numbers by applying f0b to each item in i. ($:#:>: ^: (1000 > f0b)) 0 by using rank 0 (fob"0)
f0b"0 i. ($:#:>: ^: (1000 > f0b)) 0
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987
In your case you wanted the ones under 2000000
f0b"0 i. ($:#:>: ^: (2000000 > f0b)) 0
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765 10946 17711 28657 46368 75025 121393 196418 317811 514229 832040 1346269
... and then I realized that you wanted a verb to be able to answer your original question. I went with dyadic where the left argument is the limit and the right argument generates the sequence. Same idea but I was able to make use of some hooks when I went to the tacit form. (> f0b) checks if the result of f0b is under the limit and ($: >:) increments the right argument while allowing the left argument to remain for $:
2000000 (($: >:) ^: (> f0b)) 0
32
fnum=: (($: >:) ^: (> f0b))
2000000 fnum 0
32
f0b"0 i. 2000000 fnum 0
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765 10946 17711 28657 46368 75025 121393 196418 317811 514229 832040 1346269
I have little doubt that others will come up with better solutions, but this is what I cobbled together tonight.
I have been trying to run Cox PH model on a sample data set of 10k customers (randomly taken from 32 million customer base) for predicting probability of survival in time t (which is month in my case). I am using recurrent event survival analysis using counting process for e-commerce. For this...
1. Observation starting point: right after a customer makes first purchase
2. Start/Stop times: Months of two consecutive purchases (as in the data)
I have a few independent variables as in the sample data below:
id start stop status tenure orders revenue Quantity
A 0 20 0 0 1 $89.0 1
B 0 17 0 0 1 $556.0 2
C 0 17 0 0 1 $900.0 2
D 32 33 0 1679 9 $357.8 9
D 26 32 1 1497 7 $326.8 7
D 23 26 1 1405 4 $142.9 4
D 17 23 1 1219 3 $63.9 3
D 9 17 1 978 2 $50.0 2
D 0 9 1 694 1 $35.0 1
E 0 15 0 28 2 $156.0 2
F 0 15 0 0 1 $348.0 1
F 12 14 0 375 2 $216.8 3
F 0 12 1 0 1 $67.8 2
G 9 15 0 277 2 $419.0 2
G 0 9 1 0 1 $359.0 1
While running cox PH using the following code:
fit10=coxph(Surv(start,stop,status)~orders+tenure+Quantity+revenue,data=test)
I keep getting the following error:
Warning: X matrix deemed to be singular; variable 1 2 3 4
I tried searching for the same error online but the answers I found said this could be because of interacting independent variables, whereas my variables are individual and continuous.
I am training a model using vowpal wabbit and notice something very strange. During training, the average loss reported is very low somewhere around 0.06. However I notice that when I asked the model to predict labels on the same training data, the average loss is high around ~0.66 and the model performs poorly on predicting labels for even the training data. My initial conjecture was that the model suffered a high bias, and hence I increased the complexity of the model to use 300 hidden nodes in the first layer, but still the problem persists.
I would greatly appreciate pointers on what could be going on
The tutorial slides for VW mentions:
"If you test on the train set, does it work? (no
=> something crazy)"
So something very crazy seems to be happening and I am trying to understand where I should dig deeper possibly.
More details:
I am using vowpal wabbit for a named entity recognition task where features are word representations. I am trying several models using neural networks with multiple hidden units and trying to evaluate the model. However all of my trained models exhibit high average loss when tested on the training data itself which I find very odd.
Here is one way to reproduce the problem:
Output of training:
vw -d ~/embeddings/eng_train_4.vw --loss_function logistic --oaa 6 --nn 32 -l 10 --random_weights 1 -f test_3.model --passes 4 -c
final_regressor = test_3.model
Num weight bits = 18
learning rate = 10
initial_t = 0
power_t = 0.5
decay_learning_rate = 1
using cache_file = /home/vvkulkarni/embeddings/eng_train_4.vw.cache
ignoring text input in favor of cache input
num sources = 1
average since example example current current current
loss last counter weight label predict features
0.666667 0.666667 3 3.0 1 1 577
0.833333 1.000000 6 6.0 1 2 577
0.818182 0.800000 11 11.0 4 4 577
0.863636 0.909091 22 22.0 1 4 577
0.636364 0.409091 44 44.0 1 1 577
0.390805 0.139535 87 87.0 1 1 577
0.258621 0.126437 174 174.0 1 1 577
0.160920 0.063218 348 348.0 1 1 577
0.145115 0.129310 696 696.0 1 1 577
0.138649 0.132184 1392 1392.0 1 1 577
0.122486 0.106322 2784 2784.0 1 1 577
0.097522 0.072557 5568 5568.0 1 1 577
0.076875 0.056224 11135 11135.0 1 1 577
0.058647 0.040417 22269 22269.0 1 1 577
0.047803 0.036959 44537 44537.0 1 1 577
0.038934 0.030066 89073 89073.0 1 1 577
0.036768 0.034601 178146 178146.0 1 1 577
0.032410 0.032410 356291 356291.0 1 1 577 h
0.029782 0.027155 712582 712582.0 1 1 577 h
finished run
number of examples per pass = 183259
passes used = 4
weighted example sum = 733036
weighted label sum = 0
average loss = 0.0276999
best constant = 0
total feature number = 422961744
Now when I evaluate the model above using the same data (used for training)
vw -t ~/embeddings/eng_train_4.vw -i test_3.model -p test_3.pred
only testing
Num weight bits = 18
learning rate = 10
initial_t = 1
power_t = 0.5
predictions = test_3.pred
using no cache
Reading datafile = /home/vvkulkarni/embeddings/eng_train_4.vw
num sources = 1
average since example example current current current
loss last counter weight label predict features
0.333333 0.333333 3 3.0 1 1 577
0.500000 0.666667 6 6.0 1 4 577
0.636364 0.800000 11 11.0 6 3 577
0.590909 0.545455 22 22.0 1 1 577
0.500000 0.409091 44 44.0 4 1 577
0.482759 0.465116 87 87.0 1 1 577
0.528736 0.574713 174 174.0 1 3 577
0.500000 0.471264 348 348.0 1 3 577
0.517241 0.534483 696 696.0 6 1 577
0.536638 0.556034 1392 1392.0 4 4 577
0.560345 0.584052 2784 2784.0 1 5 577
0.560884 0.561422 5568 5568.0 6 2 577
0.586349 0.611820 11135 11135.0 1 1 577
0.560914 0.535477 22269 22269.0 1 1 577
0.557200 0.553485 44537 44537.0 1 1 577
0.568938 0.580676 89073 89073.0 1 2 577
0.560568 0.552199 178146 178146.0 1 1 577
finished run
number of examples per pass = 203621
passes used = 1
weighted example sum = 203621
weighted label sum = 0
average loss = 0.557428 <<< This is what is tricky.
best constant = -4.91111e-06
total feature number = 117489309
Things I have tried:
1.I tried increasing the number of hidden nodes to 600 but to no avail.
2.I also tried using quadratic features with 300 hidden nodes but that did not help either.
The rationale behind trying 1.) and 2.) was to increase model complexity assuming that high training error was due to high bias.
Update:
Even more intrestingsly, if I however specify the number of passes to be 4 in the testing phase (even though I assumed the model would have learnt a decision boundary), then the problem goes away. I am trying to understand why ?
vvkulkarni#einstein:/scratch1/vivek/test$ vw -t ~/embeddings/eng_train_4.vw -i test_3.model -p test_3_1.pred --passes 4 -c
only testing
Num weight bits = 18
learning rate = 10
initial_t = 1
power_t = 0.5
decay_learning_rate = 1
predictions = test_3_1.pred
using cache_file = /home/vvkulkarni/embeddings/eng_train_4.vw.cache
ignoring text input in favor of cache input
num sources = 1
average since example example current current current
loss last counter weight label predict features
0.333333 0.333333 3 3.0 1 1 577
0.166667 0.000000 6 6.0 1 1 577
0.090909 0.000000 11 11.0 4 4 577
0.045455 0.000000 22 22.0 1 1 577
0.022727 0.000000 44 44.0 1 1 577
0.011494 0.000000 87 87.0 1 1 577
0.017241 0.022989 174 174.0 1 1 577
0.022989 0.028736 348 348.0 1 1 577
0.020115 0.017241 696 696.0 1 1 577
0.043822 0.067529 1392 1392.0 1 1 577
0.031968 0.020115 2784 2784.0 1 1 577
0.031968 0.031968 5568 5568.0 1 1 577
0.032959 0.033950 11135 11135.0 1 1 577
0.029952 0.026944 22269 22269.0 1 1 577
0.029212 0.028471 44537 44537.0 1 1 577
0.030481 0.031750 89073 89073.0 1 1 577
0.028673 0.026866 178146 178146.0 1 1 577
0.034001 0.034001 356291 356291.0 1 1 577 h
0.034026 0.034051 712582 712582.0 1 1 577 h
You have hash collisions because you have many more features than you have spaces in the hash.
The default hash size is 18 bits or 262144 spaces. According to your first printout, there are 422961744 features which at a minimum requires 27 bits so you should add -b27 (or more) to your command line.
I don't have your input file so I cannot try it and see.. but here is one way to check for collisions:
Run your learning phase and add --invert_hash final
then check collisions with these lines:
tail -n +13 final | sort -n -k 2 -t ':' | wc -l
tail -n +13 final | sort -nu -k 2 -t ':' | wc -l
The values output should be the same. I got this tip from John Langford, creator of Vowpal Wabbit.