%PDF-1.5
...
10737 0 obj
<</MarkInfo<</Marked true>>/Metadata 161 0 R/PageLayout/OneColumn/Pages 10732 0 R/StructTreeRoot 206 0 R/Type/Catalog>>
endobj
10738 0 obj
<</Contents[10740 0 R 10741 0 R 10747 0 R 10748 0 R 10749 0 R 10750 0 R 10751 0 R 10752 0 R]/CropBox[0.0 0.0 516.0 728.64]/MediaBox[0.0 0.0 516.0 728.64]/Parent 10733 0 R/Resources<</ColorSpace<</CS0 10771 0 R/CS1 10772 0 R>>/ExtGState<</GS0 10773 0 R>>/Font<</C2_0 10778 0 R/C2_1 10783 0 R/C2_2 10788 0 R/C2_3 10793 0 R/C2_4 10798 0 R/TT0 10800 0 R/TT1 10802 0 R/TT2 10804 0 R/TT3 10806 0 R/TT4 10808 0 R>>/XObject<</Im0 10769 0 R>>>>/Rotate 0/StructParents 0/Tabs/S/Type/Page>>
endobj
10739 0 obj
<</Filter/FlateDecode/First 410/Length 3756/N 38/Type/ObjStm>>stream
10771 0 10772 21 10773 42 10774 138 10775 190 10776 442 10777 741 10778 752 10779 869 10780 921 10781 1190 10782 2050 10783 2061 10784 2192 10785 2244 10786 2504 10787 3456 10788 3467 10789 3587 10790 3639 10791 3903 10792 6058 10793 6069 10794 6196 10795 6248 10796 6507 10797 8153 10798 8164 10799 8284 10800 8496 10801 9662 10802 9894 10803 11072 10804 11325 10805 11779 10806 11985 10807 13147 10808 13395
[/ICCBased 10753 0 R][/ICCBased 10754 0 R]
<</AIS false/BM/Normal/CA 1.0/OP false/OPM 1/SA true/SMask/None/Type/ExtGState/ca 1.0/op false>>
<</Ordering(Identity)/Registry(Adobe)/Supplement 0>><</Ascent 858/CIDSet 10757 0 R/CapHeight 719/Descent -148/Flags 4/FontBBox[-16 -148 1008 858]/FontFamily(\xfe\xff\x00H\x00Y\xc9\x11\xac\xe0\xb5\x15)/FontFile2 10758 0 R/FontName/YDRADB+H2gtrM/FontStretch/Normal/FontWeight 400/ItalicAngle 0/StemV 60/Type/FontDescriptor/XHeight 520>>
...
endstream
endobj
...
No. - Type
10732 - Pages
206 - StructTreeRoot
10771, 10772, 10773, 10778 ... - Font
Many indirect objects including 10732, 206, 10771 and 10772 do not exist in the pdf file.
But I think I found objects 10771~10808 in object 10739 stream.
Q1. Why are there no object 10732(Pages) and 206(StructTreeRoot) in the pdf file?
Q2. Why are indirect objects in stream?
I would be grateful if you would suggest any explanations or resources for reference.
Starting with version 1.5 PDF supports so called object streams, i.e. stream objects which contain other non-stream objects.
Your object 10739 is such an object stream as you can see in its Type ObjStm.
This allows those other objects to be compressed. In particular structure tree objects which otherwise can substantially increase the size of a PDF, can be compressed fairly well, reducing their impact on the document size.
For details please study the PDF specification, section 7.5.7 – Object Streams, in either the current PDF specification ISO 32000-2 or its predecessor ISO 32000-1.
Adobe has shared a copy of ISO 32000-1 on their web site which merely has its ISO page headers replaced. Simply google for "PDF32000_2008"; currently it is located at https://www.adobe.com/content/dam/acom/en/devnet/pdf/pdfs/PDF32000_2008.pdf but as far as I know this isn't a permalink.
I'm running a load test with wrk2 as a job on Jenkins. I'd like to send the results of the load test to Graylog but I only want to store the Requests/Sec and average latency.
Here's what the output looks like:
Running 30s test # https://example.com
1 threads and 100 connections
Thread calibration: mean lat.: 8338.285ms, rate sampling interval: 19202ms
Thread Stats Avg Stdev Max +/- Stdev
Latency 16.20s 6.17s 29.64s 65.74%
Req/Sec 5.00 0.00 5.00 100.00%
Latency Distribution (HdrHistogram - Recorded Latency)
50.000% 15.72s
75.000% 20.81s
90.000% 24.58s
99.000% 29.13s
99.900% 29.66s
99.990% 29.66s
99.999% 29.66s
100.000% 29.66s
Detailed Percentile spectrum:
Value Percentile TotalCount 1/(1-Percentile)
4497.407 0.000000 1 1.00
7561.215 0.100000 11 1.11
11100.159 0.200000 22 1.25
12582.911 0.300000 33 1.43
14565.375 0.400000 44 1.67
15720.447 0.500000 54 2.00
16416.767 0.550000 60 2.22
17301.503 0.600000 65 2.50
18464.767 0.650000 71 2.86
19185.663 0.700000 76 3.33
20807.679 0.750000 81 4.00
21479.423 0.775000 84 4.44
22347.775 0.800000 87 5.00
22527.999 0.825000 90 5.71
23216.127 0.850000 93 6.67
23478.271 0.875000 95 8.00
23805.951 0.887500 96 8.89
24723.455 0.900000 98 10.00
25067.519 0.912500 99 11.43
25395.199 0.925000 101 13.33
26525.695 0.937500 102 16.00
26525.695 0.943750 102 17.78
26705.919 0.950000 103 20.00
28065.791 0.956250 104 22.86
28065.791 0.962500 104 26.67
28377.087 0.968750 105 32.00
28377.087 0.971875 105 35.56
28475.391 0.975000 106 40.00
28475.391 0.978125 106 45.71
28475.391 0.981250 106 53.33
29130.751 0.984375 107 64.00
29130.751 0.985938 107 71.11
29130.751 0.987500 107 80.00
29130.751 0.989062 107 91.43
29130.751 0.990625 107 106.67
29655.039 0.992188 108 128.00
29655.039 1.000000 108 inf
#[Mean = 16199.756, StdDeviation = 6170.105]
#[Max = 29638.656, Total count = 108]
#[Buckets = 27, SubBuckets = 2048]
----------------------------------------------------------
130 requests in 30.02s, 13.44MB read
Socket errors: connect 0, read 0, write 0, timeout 1192
Requests/sec: 4.33
Transfer/sec: 458.47KB
Does anyone know how I could go about extracting Requests/sec (at the bottom) and the latency average to send as JSON parameters?
The expected output would be: "latency": 16.2, "requests_per_second": 4.33
You didn't provide the expected output so your question isn't clear but is this what you want?
$ awk 'BEGIN{a["Latency"]; a["Requests/sec:"]} ($1 in a) && ($2 ~ /[0-9]/){print $1, $2}' file
Latency 16.20s
Requests/sec: 4.33
Updated based on you adding expected output to your question:
$ awk '
BEGIN { map["Latency"]="latency"; map["Requests/sec:"]="requests_per_second" }
($1 in map) && ($2 ~ /[0-9]/) { printf "%s\"%s\": %s", ofs, map[$1], $2+0; ofs=", " }
END { print "" }
' file
"latency": 16.2, "requests_per_second": 4.33
I'm relatively new to lua and programming in general (self taught), so please be gentle!
Anyway, I wrote a lua script to read a UDP message from a game. The structure of the message is:
DATAxXXXXaaaaBBBBccccDDDDeeeeFFFFggggHHHH
DATAx = 4 letter ID and x = control character
XXXX = integer shows the group of the data (groups are known)
aaaa...HHHHH = 8 single-precision floating point numbers
The last ones is those numbers I need to decode.
If I print the message as received, it's something like:
DATA*{V???A?A?...etc.
Using string.byte(), I'm getting a stream of bytes like this (I have "formatted" the bytes to reflect the structure above.
68 65 84 65/42/20 0 0 0/237 222 28 66/189 59 182 65/107 42 41 65/33 173 79 63/0 0 128 63/146 41 41 65/0 0 30 66/0 0 184 65
The first 5 bytes are of course the DATA*. The next 4 are the 20th group of data. The next bytes, the ones I need to decode, and are equal to those values:
237 222 28 66 = 39.218
189 59 182 65 = 22.779
107 42 41 65 = 10.573
33 173 79 63 = 0.8114
0 0 128 63 = 1.0000
146 41 41 65 = 10.573
0 0 30 66 = 39.500
0 0 184 65 = 23.000
I've found C# code that does the decode with BitConverter.ToSingle(), but I haven't found any like this for Lua.
Any idea?
What Lua version do you have?
This code works in Lua 5.3
local str = "DATA*\20\0\0\0\237\222\28\66\189\59\182\65..."
-- Read two float values starting from position 10 in the string
print(string.unpack("<ff", str, 10)) --> 39.217700958252 22.779169082642 18
-- 18 (third returned value) is the next position in the string
For Lua 5.1 you have to write special function (or steal it from François Perrad's git repo )
local function binary_to_float(str, pos)
local b1, b2, b3, b4 = str:byte(pos, pos+3)
local sign = b4 > 0x7F and -1 or 1
local expo = (b4 % 0x80) * 2 + math.floor(b3 / 0x80)
local mant = ((b3 % 0x80) * 0x100 + b2) * 0x100 + b1
local n
if mant + expo == 0 then
n = sign * 0.0
elseif expo == 0xFF then
n = (mant == 0 and sign or 0) / 0
else
n = sign * (1 + mant / 0x800000) * 2.0^(expo - 0x7F)
end
return n
end
local str = "DATA*\20\0\0\0\237\222\28\66\189\59\182\65..."
print(binary_to_float(str, 10)) --> 39.217700958252
print(binary_to_float(str, 14)) --> 22.779169082642
It’s little-endian byte-order of IEEE-754 single-precision binary:
E.g., 0 0 128 63 is:
00111111 10000000 00000000 00000000
(63) (128) (0) (0)
Why that equals 1 requires that you understand the very basics of IEEE-754 representation, namely its use of an exponent and mantissa. See here to start.
See #Egor‘s answer above for how to use string.unpack() in Lua 5.3 and one possible implementation you could use in earlier versions.
I'm trying to predict time series data for the next few days looking at past few days, using Keras. My label data is target values for multiple future days, regression model has multiple output neurons (the "direct approach" for time series).
Here is test data with predictions for 10 days, using 60 days history.
10 days prediction for test data
As you can see, future values for all days are about the same. I've spent quite some time on it, and must admit that I'm probably missing something with respect to LSTM...
Here is training data with prediction:
10 days prediction for training data
In order to confirm that I'm preparing data properly, I've created a "tracking data set" which I used to visualize data transformations. Here it is...
Data set:
Open,High,Low,Close,Volume,OpenInt
111,112,113,114,115,0
121,122,123,124,125,0
131,132,133,134,135,0
141,142,143,144,145,0
151,152,153,154,155,0
161,162,163,164,165,0
171,172,173,174,175,0
181,182,183,184,185,0
191,192,193,194,195,0
201,202,203,204,205,0
211,212,213,214,215,0
221,222,223,224,225,0
231,232,233,234,235,0
241,242,243,244,245,0
251,252,253,254,255,0
261,262,263,264,265,0
271,272,273,274,275,0
281,282,283,284,285,0
291,292,293,294,295,0
Training set using 2 days history, predicting 3 days future values (I used different values of history days and future days, and it all makes sense to me), without feature scaling in order to visualize data transformations:
X train (6, 2, 5)
[[[111 112 113 114 115]
[121 122 123 124 125]]
[[121 122 123 124 125]
[131 132 133 134 135]]
[[131 132 133 134 135]
[141 142 143 144 145]]
[[141 142 143 144 145]
[151 152 153 154 155]]
[[151 152 153 154 155]
[161 162 163 164 165]]
[[161 162 163 164 165]
[171 172 173 174 175]]]
Y train (6, 3)
[[131 141 151]
[141 151 161]
[151 161 171]
[161 171 181]
[171 181 191]
[181 191 201]]
Test set
X test (6, 2, 5)
[[[201 202 203 204 205]
[211 212 213 214 215]]
[[211 212 213 214 215]
[221 222 223 224 225]]
[[221 222 223 224 225]
[231 232 233 234 235]]
[[231 232 233 234 235]
[241 242 243 244 245]]
[[241 242 243 244 245]
[251 252 253 254 255]]
[[251 252 253 254 255]
[261 262 263 264 265]]]
Y test (6, 3)
[[221 231 241]
[231 241 251]
[241 251 261]
[251 261 271]
[261 271 281]
[271 281 291]]
Model:
def CreateRegressor(self,
optimizer='adam',
activation='tanh', # RNN activation
init_mode='glorot_uniform',
hidden_neurons=50,
dropout_rate=0.0,
weight_constraint=0,
stateful=False,
# SGD parameters
learn_rate=0.01,
momentum=0):
kernel_constraint = maxnorm(weight_constraint) if weight_constraint > 0 else None
model = Sequential()
model.add(LSTM(units=hidden_neurons, activation=activation, kernel_initializer=init_mode, kernel_constraint=kernel_constraint,
return_sequences=True, input_shape=(self.X_train.shape[1], self.X_train.shape[2]), stateful=stateful))
model.add(Dropout(dropout_rate))
model.add(LSTM(units=hidden_neurons, activation=activation, kernel_initializer=init_mode, kernel_constraint=kernel_constraint,
return_sequences=True, stateful=stateful))
model.add(Dropout(dropout_rate))
model.add(LSTM(units=hidden_neurons, activation=activation, kernel_initializer=init_mode, kernel_constraint=kernel_constraint,
return_sequences=True, stateful=stateful))
model.add(Dropout(dropout_rate))
model.add(LSTM(units=hidden_neurons, activation=activation, kernel_initializer=init_mode, kernel_constraint=kernel_constraint,
return_sequences=False, stateful=stateful))
model.add(Dropout(dropout_rate))
model.add(Dense(units=self.y_train.shape[1]))
if (optimizer == 'SGD'):
optimizer = SGD(lr=learn_rate, momentum=momentum)
model.compile(optimizer=optimizer, loss='mean_squared_error')
return model
...which I create with these params:
self.CreateRegressor(optimizer = 'adam', hidden_neurons = 100)
... and then fit like this:
self.regressor.fit(self.X_train, self.y_train, epochs=100, batch_size=32)
... and predict:
y_pred = self.regressor.predict(X_test)
... or
y_pred_train = self.regressor.predict(X_train)
What am I missing?
I tried to create a neural network to estimate y = x ^ 2. So I created a fitting neural network and gave it some samples for input and output. I tried to build this network in C++. But the result is different than I expected.
With the following inputs:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 -1
-2 -3 -4 -5 -6 -7 -8 -9 -10 -11 -12 -13 -14 -15 -16 -17 -18 -19 -20 -21 -22 -23 -24 -25 -26 -27 -28 -29 -30 -31 -32 -33 -34 -35 -36 -37 -38 -39 -40 -41 -42 -43 -44 -45 -46 -47 -48 -49 -50 -51 -52 -53 -54 -55 -56 -57 -58 -59 -60 -61 -62 -63 -64 -65 -66 -67 -68 -69 -70 -71
and the following outputs:
0 1 4 9 16 25 36 49 64 81 100 121 144 169 196 225 256 289 324 361 400
441 484 529 576 625 676 729 784 841 900 961 1024 1089 1156 1225 1296
1369 1444 1521 1600 1681 1764 1849 1936 2025 2116 2209 2304 2401 2500
2601 2704 2809 2916 3025 3136 3249 3364 3481 3600 3721 3844 3969 4096
4225 4356 4489 4624 4761 4900 5041 1 4 9 16 25 36 49 64 81 100 121 144
169 196 225 256 289 324 361 400 441 484 529 576 625 676 729 784 841
900 961 1024 1089 1156 1225 1296 1369 1444 1521 1600 1681 1764 1849
1936 2025 2116 2209 2304 2401 2500 2601 2704 2809 2916 3025 3136 3249
3364 3481 3600 3721 3844 3969 4096 4225 4356 4489 4624 4761 4900 5041
I used fitting tool network. with matrix rows. Training is 70%, validation is 15% and testing is 15% as well. The number of hidden neurons is two. Then in command lines I wrote this:
purelin(net.LW{2}*tansig(net.IW{1}*inputTest+net.b{1})+net.b{2})
Other information :
My net.b[1] is: -1.16610230053776 1.16667147712026
My net.b[2] is: 51.3266249426358
And net.IW(1) is: 0.344272596370387 0.344111217766824
net.LW(2) is: 31.7635369693519 -31.8082184881063
When my inputTest is 3, the result of this command is 16, while it should be about 9. Have I made an error somewhere?
I found the Stack Overflow post Neural network in MATLAB that contains a problem like my problem, but there is a little difference, and the differences is in that problem the ranges of input and output are same, but in my problem is no. That solution says I need to scale out the results, but how can I scale out my result?
You are right about scaling. As was mentioned in the linked answer, the neural network by default scales the input and output to the range [-1,1]. This can be seen in the network processing functions configuration:
>> net = fitnet(2);
>> net.inputs{1}.processFcns
ans =
'removeconstantrows' 'mapminmax'
>> net.outputs{2}.processFcns
ans =
'removeconstantrows' 'mapminmax'
The second preprocessing function applied to both input/output is mapminmax with the following parameters:
>> net.inputs{1}.processParams{2}
ans =
ymin: -1
ymax: 1
>> net.outputs{2}.processParams{2}
ans =
ymin: -1
ymax: 1
to map both into the range [-1,1] (prior to training).
This means that the trained network expects input values in this range, and outputs values also in the same range. If you want to manually feed input to the network, and compute the output yourself, you have to scale the data at input, and reverse the mapping at the output.
One last thing to remember is that each time you train the ANN, you will get different weights. If you want reproducible results, you need to fix the state of the random number generator (initialize it with the same seed each time). Read the documentation on functions like rng and RandStream.
You also have to pay attention that if you are dividing the data into training/testing/validation sets, you must use the same split each time (probably also affected by the randomness aspect I mentioned).
Here is an example to illustrate the idea (adapted from another post of mine):
%%# data
x = linspace(-71,71,200); %# 1D input
y_model = x.^2; %# model
y = y_model + 10*randn(size(x)).*x; %# add some noise
%%# create ANN, train, simulate
net = fitnet(2); %# one hidden layer with 2 nodes
net.divideFcn = 'dividerand';
net.trainParam.epochs = 50;
net = train(net,x,y);
y_hat = net(x);
%%# plot
plot(x, y, 'b.'), hold on
plot(x, x.^2, 'Color','g', 'LineWidth',2)
plot(x, y_hat, 'Color','r', 'LineWidth',2)
legend({'data (noisy)','model (x^2)','fitted'})
hold off, grid on
%%# manually simulate network
%# map input to [-1,1] range
[~,inMap] = mapminmax(x, -1, 1);
in = mapminmax('apply', x, inMap);
%# propagate values to get output (scaled to [-1,1])
hid = tansig( bsxfun(#plus, net.IW{1}*in, net.b{1}) ); %# hidden layer
outLayerOut = purelin( net.LW{2}*hid + net.b{2} ); %# output layer
%# reverse mapping from [-1,1] to original data scale
[~,outMap] = mapminmax(y, -1, 1);
out = mapminmax('reverse', outLayerOut, outMap);
%# compare against MATLAB output
max( abs(out - y_hat) ) %# this should be zero (or in the order of `eps`)
I opted to use the mapminmax function, but you could have done that manually as well. The formula is a pretty simply linear mapping:
y = (ymax-ymin)*(x-xmin)/(xmax-xmin) + ymin;