My table:
Report_Period
Entity
Tag
Users Count
Report_Period_M-1
Report_Period_Q-1
...
2017-06-30
entity 1
X
471
2017-05-31
2017-03-31
...
2020-12-31
entity 2
A
135
2020-11-30
2020-09-30
...
2020-11-30
entity 3
X
402
2020-10-31
2020-08-31
...
What I want:
Report_Period
Entity
Tag
Users Count
Users_Count_M-1
Users_Count_Q-1
...
2017-06-30
entity 1
X
471
450
438
...
2020-12-31
entity 2
A
135
122
118
...
2020-11-30
entity 3
X
402
380
380
...
I have have tried this code but it duplicate records! How can I avoid it?
SELECT M."Entity",M."Tag",M."Report_Period",M."Users Count",
M."Report_Period_M-1",M1."Users Count" AS "Users Count M1",
FROM "DB"."SCHEMA"."PERIOD" M, "DB"."SCHEMA"."PERIOD" M1
WHERE M."Report_Period_M-1"= M1."Report_Period"
Your join clause should include the entity column and tag (I suspect)
SELECT M."Entity",
M."Tag",
M."Report_Period",
M."Users Count",
M."Report_Period_M-1",
M1."Users Count" AS "Users Count M1",
FROM "DB"."SCHEMA"."PERIOD" M,
"DB"."SCHEMA"."PERIOD" M1
WHERE M."Report_Period_M-1"= M1."Report_Period"
AND M."Entity" = M1."Entity"
AND M."Tag" = M1."Tag"
I'm trying to predict time series data for the next few days looking at past few days, using Keras. My label data is target values for multiple future days, regression model has multiple output neurons (the "direct approach" for time series).
Here is test data with predictions for 10 days, using 60 days history.
10 days prediction for test data
As you can see, future values for all days are about the same. I've spent quite some time on it, and must admit that I'm probably missing something with respect to LSTM...
Here is training data with prediction:
10 days prediction for training data
In order to confirm that I'm preparing data properly, I've created a "tracking data set" which I used to visualize data transformations. Here it is...
Data set:
Open,High,Low,Close,Volume,OpenInt
111,112,113,114,115,0
121,122,123,124,125,0
131,132,133,134,135,0
141,142,143,144,145,0
151,152,153,154,155,0
161,162,163,164,165,0
171,172,173,174,175,0
181,182,183,184,185,0
191,192,193,194,195,0
201,202,203,204,205,0
211,212,213,214,215,0
221,222,223,224,225,0
231,232,233,234,235,0
241,242,243,244,245,0
251,252,253,254,255,0
261,262,263,264,265,0
271,272,273,274,275,0
281,282,283,284,285,0
291,292,293,294,295,0
Training set using 2 days history, predicting 3 days future values (I used different values of history days and future days, and it all makes sense to me), without feature scaling in order to visualize data transformations:
X train (6, 2, 5)
[[[111 112 113 114 115]
[121 122 123 124 125]]
[[121 122 123 124 125]
[131 132 133 134 135]]
[[131 132 133 134 135]
[141 142 143 144 145]]
[[141 142 143 144 145]
[151 152 153 154 155]]
[[151 152 153 154 155]
[161 162 163 164 165]]
[[161 162 163 164 165]
[171 172 173 174 175]]]
Y train (6, 3)
[[131 141 151]
[141 151 161]
[151 161 171]
[161 171 181]
[171 181 191]
[181 191 201]]
Test set
X test (6, 2, 5)
[[[201 202 203 204 205]
[211 212 213 214 215]]
[[211 212 213 214 215]
[221 222 223 224 225]]
[[221 222 223 224 225]
[231 232 233 234 235]]
[[231 232 233 234 235]
[241 242 243 244 245]]
[[241 242 243 244 245]
[251 252 253 254 255]]
[[251 252 253 254 255]
[261 262 263 264 265]]]
Y test (6, 3)
[[221 231 241]
[231 241 251]
[241 251 261]
[251 261 271]
[261 271 281]
[271 281 291]]
Model:
def CreateRegressor(self,
optimizer='adam',
activation='tanh', # RNN activation
init_mode='glorot_uniform',
hidden_neurons=50,
dropout_rate=0.0,
weight_constraint=0,
stateful=False,
# SGD parameters
learn_rate=0.01,
momentum=0):
kernel_constraint = maxnorm(weight_constraint) if weight_constraint > 0 else None
model = Sequential()
model.add(LSTM(units=hidden_neurons, activation=activation, kernel_initializer=init_mode, kernel_constraint=kernel_constraint,
return_sequences=True, input_shape=(self.X_train.shape[1], self.X_train.shape[2]), stateful=stateful))
model.add(Dropout(dropout_rate))
model.add(LSTM(units=hidden_neurons, activation=activation, kernel_initializer=init_mode, kernel_constraint=kernel_constraint,
return_sequences=True, stateful=stateful))
model.add(Dropout(dropout_rate))
model.add(LSTM(units=hidden_neurons, activation=activation, kernel_initializer=init_mode, kernel_constraint=kernel_constraint,
return_sequences=True, stateful=stateful))
model.add(Dropout(dropout_rate))
model.add(LSTM(units=hidden_neurons, activation=activation, kernel_initializer=init_mode, kernel_constraint=kernel_constraint,
return_sequences=False, stateful=stateful))
model.add(Dropout(dropout_rate))
model.add(Dense(units=self.y_train.shape[1]))
if (optimizer == 'SGD'):
optimizer = SGD(lr=learn_rate, momentum=momentum)
model.compile(optimizer=optimizer, loss='mean_squared_error')
return model
...which I create with these params:
self.CreateRegressor(optimizer = 'adam', hidden_neurons = 100)
... and then fit like this:
self.regressor.fit(self.X_train, self.y_train, epochs=100, batch_size=32)
... and predict:
y_pred = self.regressor.predict(X_test)
... or
y_pred_train = self.regressor.predict(X_train)
What am I missing?
Could someone please explain to me why in the following code (using r25630 Windows), the value of iInsertTot at line 241 is null, or more to the point, why is line 234 ("return iInsertTot;") not executed and therefore at line 241, iInsertTot is null. The value of iInsertTot at lines 231/232 is an integer. While I can and probably should code this differently, I thought that I would try and see if it worked, because my understanding of Futures and Chaining was that it would work. I have used “return” in a similar way before and it worked, but I was returning null in those cases (eg. line 201 below).
/// The problem lines are :
233 fUpdateTotalsTable().then((_) {
234 return iInsertTot;
235 });
While running in the debugger, it appears that line 234 “return iInsertTot;” is never actually executed. Running from command line has the same result.
The method being called on line 233 (fUpdateTotalsTable) is something I am just in the process of adding, and it consists basically of sync code at this stage. However, the debugger appears to go through it correctly.
I have included the method “fUpdateTotalsTable()” (line 1076) just in case that is causing a problem.
Lines 236 to 245 have just been added, however just in case that code is invalid I have commented those lines out and run with the same problem occurring.
218 /*
219 * Process Inserts
220 */
221 }).then((_) {
222 sCheckpoint = "fProcessMainInserts";
223 ogPrintLine.fPrintForce ("Processing database ......");
224 int iMaxInserts = int.parse(lsInput[I_MAX_INSERTS]);
225 print ("");
226 return fProcessMainInserts(iMaxInserts, oStopwatch);
227 /*
228 * Update the 'totals' table with the value of Inserts
229 */
230 }).then((int iReturnVal) {
231 int iInsertTot = iReturnVal;
232 sCheckpoint = "fUpdateTotalsTable (insert value)";
233 fUpdateTotalsTable().then((_) {
234 return iInsertTot;
235 });
236 /*
237 * Display totals for inserts
238 */
239 }).then((int iInsertTot) {
240 ogTotals.fPrintTotals(
241 "${iInsertTot} rows inserted - Inserts completed",
242 iInsertTot, oStopwatch.elapsedMilliseconds);
243
244 return null;
245 /*
192 /*
193 * Clear main table if selected
194 */
195 }).then((tReturnVal) {
196 if (tReturnVal)
197 ogPrintLine.fPrintForce("Random Keys Cleared");
198 sCheckpoint = "Clear Table ${S_TABLE_NAME}";
199 bool tClearTable = (lsInput[I_CLEAR_YN] == "y");
200 if (!tFirstInstance)
201 return null;
202 return fClearTable(tClearTable, S_TABLE_NAME);
203
204 /*
205 * Update control row to increment count of instances started
206 */
207 }).then((_) {
1073 /*
1074 * Update totals table with values from inserts and updates
1075 */
1076 async.Future<bool> fUpdateTotalsTable() {
1077 async.Completer<bool> oCompleter = new async.Completer<bool>();
1078
1079 String sCcyValue = ogCcy.fCcyIntToString(ogTotals.iTotAmt);
1080
1081 print ("\n********* Total = ${sCcyValue} \n");
1082
1083 oCompleter.complete(true);
1084 return oCompleter.future;
1085 }
Your function L230-235 does not return anything and that's why your iInsertTot is null L239. To make it work you have to add a return at line 233.
231 int iInsertTot = iReturnVal;
232 sCheckpoint = "fUpdateTotalsTable (insert value)";
233 return fUpdateTotalsTable().then((_) {
234 return iInsertTot;
235 });
I tried to create a neural network to estimate y = x ^ 2. So I created a fitting neural network and gave it some samples for input and output. I tried to build this network in C++. But the result is different than I expected.
With the following inputs:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 -1
-2 -3 -4 -5 -6 -7 -8 -9 -10 -11 -12 -13 -14 -15 -16 -17 -18 -19 -20 -21 -22 -23 -24 -25 -26 -27 -28 -29 -30 -31 -32 -33 -34 -35 -36 -37 -38 -39 -40 -41 -42 -43 -44 -45 -46 -47 -48 -49 -50 -51 -52 -53 -54 -55 -56 -57 -58 -59 -60 -61 -62 -63 -64 -65 -66 -67 -68 -69 -70 -71
and the following outputs:
0 1 4 9 16 25 36 49 64 81 100 121 144 169 196 225 256 289 324 361 400
441 484 529 576 625 676 729 784 841 900 961 1024 1089 1156 1225 1296
1369 1444 1521 1600 1681 1764 1849 1936 2025 2116 2209 2304 2401 2500
2601 2704 2809 2916 3025 3136 3249 3364 3481 3600 3721 3844 3969 4096
4225 4356 4489 4624 4761 4900 5041 1 4 9 16 25 36 49 64 81 100 121 144
169 196 225 256 289 324 361 400 441 484 529 576 625 676 729 784 841
900 961 1024 1089 1156 1225 1296 1369 1444 1521 1600 1681 1764 1849
1936 2025 2116 2209 2304 2401 2500 2601 2704 2809 2916 3025 3136 3249
3364 3481 3600 3721 3844 3969 4096 4225 4356 4489 4624 4761 4900 5041
I used fitting tool network. with matrix rows. Training is 70%, validation is 15% and testing is 15% as well. The number of hidden neurons is two. Then in command lines I wrote this:
purelin(net.LW{2}*tansig(net.IW{1}*inputTest+net.b{1})+net.b{2})
Other information :
My net.b[1] is: -1.16610230053776 1.16667147712026
My net.b[2] is: 51.3266249426358
And net.IW(1) is: 0.344272596370387 0.344111217766824
net.LW(2) is: 31.7635369693519 -31.8082184881063
When my inputTest is 3, the result of this command is 16, while it should be about 9. Have I made an error somewhere?
I found the Stack Overflow post Neural network in MATLAB that contains a problem like my problem, but there is a little difference, and the differences is in that problem the ranges of input and output are same, but in my problem is no. That solution says I need to scale out the results, but how can I scale out my result?
You are right about scaling. As was mentioned in the linked answer, the neural network by default scales the input and output to the range [-1,1]. This can be seen in the network processing functions configuration:
>> net = fitnet(2);
>> net.inputs{1}.processFcns
ans =
'removeconstantrows' 'mapminmax'
>> net.outputs{2}.processFcns
ans =
'removeconstantrows' 'mapminmax'
The second preprocessing function applied to both input/output is mapminmax with the following parameters:
>> net.inputs{1}.processParams{2}
ans =
ymin: -1
ymax: 1
>> net.outputs{2}.processParams{2}
ans =
ymin: -1
ymax: 1
to map both into the range [-1,1] (prior to training).
This means that the trained network expects input values in this range, and outputs values also in the same range. If you want to manually feed input to the network, and compute the output yourself, you have to scale the data at input, and reverse the mapping at the output.
One last thing to remember is that each time you train the ANN, you will get different weights. If you want reproducible results, you need to fix the state of the random number generator (initialize it with the same seed each time). Read the documentation on functions like rng and RandStream.
You also have to pay attention that if you are dividing the data into training/testing/validation sets, you must use the same split each time (probably also affected by the randomness aspect I mentioned).
Here is an example to illustrate the idea (adapted from another post of mine):
%%# data
x = linspace(-71,71,200); %# 1D input
y_model = x.^2; %# model
y = y_model + 10*randn(size(x)).*x; %# add some noise
%%# create ANN, train, simulate
net = fitnet(2); %# one hidden layer with 2 nodes
net.divideFcn = 'dividerand';
net.trainParam.epochs = 50;
net = train(net,x,y);
y_hat = net(x);
%%# plot
plot(x, y, 'b.'), hold on
plot(x, x.^2, 'Color','g', 'LineWidth',2)
plot(x, y_hat, 'Color','r', 'LineWidth',2)
legend({'data (noisy)','model (x^2)','fitted'})
hold off, grid on
%%# manually simulate network
%# map input to [-1,1] range
[~,inMap] = mapminmax(x, -1, 1);
in = mapminmax('apply', x, inMap);
%# propagate values to get output (scaled to [-1,1])
hid = tansig( bsxfun(#plus, net.IW{1}*in, net.b{1}) ); %# hidden layer
outLayerOut = purelin( net.LW{2}*hid + net.b{2} ); %# output layer
%# reverse mapping from [-1,1] to original data scale
[~,outMap] = mapminmax(y, -1, 1);
out = mapminmax('reverse', outLayerOut, outMap);
%# compare against MATLAB output
max( abs(out - y_hat) ) %# this should be zero (or in the order of `eps`)
I opted to use the mapminmax function, but you could have done that manually as well. The formula is a pretty simply linear mapping:
y = (ymax-ymin)*(x-xmin)/(xmax-xmin) + ymin;
I'm using the entity framework 4.0 and I'm having some issues with the syntax of my query. I'm trying to join 2 tables and pass a parameter to find the value at the same time.I would like find all of the products in table 2 by finding the correlating value in table 1.
Can someone help me out with syntax please?
Thanks in advance.
sample data
table 1
ID productID categoryID
361 571 16
362 572 17
363 573 16
364 574 19
365 575 26
table 2
productID productCode
571 sku
572 sku
573 sku
574 sku
575 sku
var q = from i in context.table1
from it in context.table2
join <not sure>
where i.categoryID == it.categoryID and < parameter >
select e).Skip(value).Take(value));
foreach (var g in q)
{
Response.Write(g.productID);
}
var q = from i in context.table1
join it in context.table2 on i.productID equals it.productID
where i.categoryID == it.categoryID and it.productCode = xyz
select i).Skip(value).Take(value));