Given any general float torch.Tensor, possibly containing some NaN values, I am looking for an efficient method to either replace all the NaN values in it with zero, or remove them altogether and filter out the "useful" values in another new Tensor.
I am aware that a trivial way to do this is to manually iterate through all the values in the given tensor (and correspondingly replace them with zero or reject them for the new tensor).
Is there some pre-defined Torch function or a combination of functions which can achieve this more efficiently in terms of performance, which relies on the inherent CPU-GPU optimisations of Torch?
Well, it looks like there is no function in torch checking tensor for NaNs. But since NaN != NaN, there's a work around:
a = torch.rand(4, 5)
a[2][3] = tonumber('nan')
nan_mask = a:ne(a)
notnan_mask = a:eq(a)
print(a)
0.2434 0.1731 0.3440 0.3340 0.0519
0.0932 0.4067 nan 0.1827 0.5945
0.3020 0.1035 0.5415 0.3329 0.7881
0.6108 0.9498 0.0406 0.9335 0.3582
[torch.DoubleTensor of size 4x5]
print(nan_mask)
0 0 0 0 0
0 0 1 0 0
0 0 0 0 0
0 0 0 0 0
[torch.ByteTensor of size 4x5]
Having these masks, you can efficiently extract NaN/not NaN values and replace them with whatever you want:
print(a[notnan_mask])
...
[torch.DoubleTensor of size 19]
a[nan_mask] = 42
print(a)
0.2434 0.1731 0.3440 0.3340 0.0519
0.0932 0.4067 42.0000 0.1827 0.5945
0.3020 0.1035 0.5415 0.3329 0.7881
0.6108 0.9498 0.0406 0.9335 0.3582
[torch.DoubleTensor of size 4x5]
Related
could anyone explain how to write a custom multiclass metrics for Keras? I tried to write custom metric but encountered some issue. Main problem is I am not familiar with how tensor works during training (I think it is called Graph mode?). I am able to create confusion matrix and derived F1 score using NumPy or Python list.
I printed out the y-true and y_pred and tried to understand them, but the output was not what I expected:
Below is the function I used:
def f1_scores(y_true,y_pred):
y_true = K.print_tensor(y_true, message='y_true = ')
y_pred = K.print_tensor(y_pred, message='y_pred = ')
print(f"y_true_shape:{K.int_shape(y_true)}")
print(f"y_pred_shape:{K.int_shape(y_pred)}")
y_true_f = K.flatten(y_true)
y_pred_f = K.flatten(y_pred)
gt = K.argmax(y_true_f)
pred = K.argmax(y_pred_f)
print(f"pred_print:{pred}")
print(f"gt_print:{gt}")
pred = K.print_tensor(pred, message='pred= ')
gt = K.print_tensor(gt, message='gt =')
print(f"pred_shape:{K.int_shape(pred)}")
print(f"gt_shape:{K.int_shape(gt)}")
pred_f = K.flatten(pred)
gt_f = K.flatten(gt)
pred_f = K.print_tensor(pred_f, message='pred_f= ')
gt_f = K.print_tensor(gt_f, message='gt_f =')
print(f"pred_f_shape:{K.int_shape(pred_f)}")
print(f"gt_f_shape:{K.int_shape(gt_f)}")
conf_mat = tf.math.confusion_matrix(y_true_f,y_pred_f, num_classes = 14)
"""
add codes to find F1 score for each class
"""
# return an arbitrary number, as F1 scores not found yet.
return 1
The output at when epoch 1 just started:
y_true_shape:(None, 256, 256, 14)
y_pred_shape:(None, 256, 256, 14)
pred_print:Tensor("ArgMax_1:0", shape=(), dtype=int64)
gt_print:Tensor("ArgMax:0", shape=(), dtype=int64)
pred_shape:()
gt_shape:()
pred_f_shape:(1,)
gt_f_shape:(1,)
Then for the rest of the steps and epochs were similar as below:
y_true = [[[[1 0 0 ... 0 0 0]
[1 0 0 ... 0 0 0]
[1 0 0 ... 0 0 0]
...
y_pred = [[[[0.0889623 0.0624801107 0.0729747042 ... 0.0816219151 0.0735477135 0.0698677748]
[0.0857798532 0.0721047595 0.0754121244 ... 0.0723947287 0.0728530064 0.0676521733]
[0.0825942457 0.0670698211 0.0879610255 ... 0.0721599609 0.0845924541 0.0638583601]
...
pred= 1283828
gt = 0
pred_f= [1283828]
gt_f = [0]
Why is pred a number instead of a list of numbers with each number represents index of class? Similarly, why is pred_f is a list with only one number instead of list of indices?
And for gt (and gt_f), why is the value 0? I expect them to be list of indices.
I looks like argmax() simply uses the flattened y.
You need to specify which axis you want argmax() to reduce. Probably it's the last one, in your case 3. Then you'll get pred with a shape (None, 256, 256) containing integer between 0 and 13.
Try something like this: pred = K.argmax(y_pred, axis=3)
This is the documentation for tensorflow argmax. (But I'm not sure if you're using exactly that, since I can not see what K is imported as)
currently I have input X with shape (50, 25), where there are 50 feature vectors and each vector has 25 dimensions. The data of X is, for example, like follows:
X = [[0. 0. 0. ... 1. 1. 1.]
[0. 0. 0. ... 1. 1. 1.]
[0. 0. 0. ... 1. 1. 1.]
...
[0. 0. 0. ... 1. 1. 1.]
[0. 0. 0. ... 1. 1. 1.]
[0. 0. 0. ... 1. 1. 1.]]
And the output label y is [0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0], of length 50. I.e. each feature vector has a label which corresponds to an element in y.
how can I construct a pytorch LSTM, reshape the input object to 3 dimensions, and properly interpret the output object? Thanks so much for the help in advance.
Currently I have a template for LSTM like this, since my input is already numerical, I was thinking to get rid of the encoder / decoder part, is that correct?
class RNNModel(nn.Module):
"""Container module with an encoder, a recurrent module, and a decoder."""
def __init__(self, rnn_type, ntoken, ninp, nhid, nlayers, dropout=0, tie_weights=False):
super(RNNModel, self).__init__()
self.drop = nn.Dropout(dropout)
self.ntoken = ntoken
self.decoder = nn.Linear(nhid, self.ntoken)
if rnn_type in ['LSTM', 'GRU']:
self.rnn = getattr(nn, rnn_type)(ninp, nhid, nlayers, dropout=dropout)
else:
try:
nonlinearity = {'RNN_TANH': 'tanh', 'RNN_RELU': 'relu'}[rnn_type]
except KeyError:
raise ValueError( """An invalid option for `--model` was supplied,
options are ['LSTM', 'GRU', 'RNN_TANH' or 'RNN_RELU']""")
self.rnn = nn.RNN(ninp, nhid, nlayers, nonlinearity=nonlinearity, dropout=dropout)
self.init_weights()
self.rnn_type = rnn_type
self.nhid = nhid
self.nlayers = nlayers
def init_weights(self):
initrange = 0.1
nn.init.zeros_(self.decoder.weight)
nn.init.uniform_(self.decoder.weight, -initrange, initrange)
def forward(self, input, hidden):
emb = self.drop(input)
emb = emb.transpose(1, 0)
output, hidden = self.rnn(emb, hidden) #output of shape (length, batchsize, nhid)
output = self.drop(output)
output = output[-1, :, :] #shape (batchsize, nhid)
decoded = self.decoder(output) #shape (batchsize, ntoken)
return F.log_softmax(decoded, dim=1), hidden
def init_hidden(self, bsz):
weight = next(self.parameters())
if self.rnn_type == 'LSTM':
return (weight.new_zeros(self.nlayers, bsz, self.nhid),
weight.new_zeros(self.nlayers, bsz, self.nhid))
else:
return weight.new_zeros(self.nlayers, bsz, self.nhid)
Currently the train I wrote is
X = X.reshape((1, 50, 25))
hidden = self.model.init_hidden(1)
for iter in range(0, self.epochs):
data = torch.from_numpy(X)
target = torch.LongTensor(y.reshape((1, torch.LongTensor(y).size(0))))
self.model.zero_grad()
self.optimizer.zero_grad()
hidden = self.repackage_hidden(hidden)
output, hidden = self.model(data.float(), hidden)
loss = self.criterion(output, target)
loss.backward()
torch.nn.utils.clip_grad_norm_(self.model.parameters(), 0.25)
self.optimizer.step()
self.model.train()
But I got the error: RuntimeError: multi-target not supported at /tmp/pip-req-build-4baxydiv/aten/src/THNN/generic/ClassNLLCriterion.c:22
Output of rnn take shape of (length, batchsize, nhid), base on your label (1 number per sample) I assume you're doing classification, so usually we will give the classifier (self.decoder) output features of the last timestep. Here I changed your forward method to this and got output of shape (batchsize, ntoken) which fit the shape of your label.
def forward(self, input, hidden):
emb = self.drop(self.encoder(input))
emb = emb.transpose(1, 0) #(batchsize, length, ninp) => (length, batchsize, ninp)
output, hidden = self.rnn(emb, hidden) #output of shape (length, batchsize, nhid)
output = self.drop(output)
output = output[-1, :, :] #shape (batchsize, nhid)
decoded = self.decoder(output) #shape (batchsize, ntoken)
return F.log_softmax(decoded, dim=1), hidden
About getting rid of self.encoder, it is a embedding layer which take a array of indices and replace each of them with a vector. If your input includes indices (int/long) of something, you may use it, otherwise (it is not index but some number as float like temperature,...) you should get rid of it (because it's wrong). Sorry if my English is confusing.
Following a subsampling inside of resampling procedure, as exemplified here https://topepo.github.io/caret/subsampling-for-class-imbalances.html#subsampling-during-resampling my question is simply how to extract the actual data-set resulting from this procedure when the caret method = “rf” and the sampling method is “smote”.
If, for example, method= glm is used then the data can be extracted with model$finalModel$data; if the method = “rpart” the data can be similarly extracted with model$finalModel$call$data.
Using subsampling inside of resampling and the method = rpart the smote data-set can be extrated as follows:
library(caret)
library(DMwR)
data("GermanCredit")
set.seed(122)
index1<-createDataPartition(GermanCredit$Class, p=.7, list = FALSE)
training<-GermanCredit[index1, ]
#testing<-GermanCredit[-index1,]
colnames(training)
metric <- "ROC"
ctrl1<- trainControl(
method = "repeatedcv",
number = 10,
repeats = 5,
search = "random",
classProbs = TRUE, # note class probabilities included
savePredictions = T, #"final"
returnResamp = "final",
allowParallel = TRUE,
summaryFunction = twoClassSummary,
sampling = "smote")
set.seed(1)
mod_fit<-train(Class ~ Age +
ForeignWorker +
Property.RealEstate +
Housing.Own +
CreditHistory.Critical, data=training, method="rpart",
metric = metric,
trControl= ctrl1)
mod_fit # ROC 0.5951215
dat_smote<- mod_fit$finalModel$call$data
table(dat_smote$.outcome)
# Bad Good
# 630 840
head(dat_smote)
# Age ForeignWorker Property.RealEstate Housing.Own CreditHistory.Critical .outcome
# 40 1 0 1 1 Good
# 29 1 0 0 0 Good
# 37 1 1 0 1 Good
# 47 1 0 0 0 Good
# 53 1 0 1 0 Good
# 29 1 0 1 0 Good
I simply would like to be able to perform the same data-set extraction when the method = "rf". The code might look like this
dat<- mod_fit$trainingData[mod_fit$trainingData == mod_fit$finalModel$x,]
I think that the only way to do it is to write a custom model that saves the data object in the fit module (that's pretty unsatisfying though).
I have a vector of integers. I want to add this vector to a particular row/column in the Mat object. Is this possible. I have been looking at the API and am unable to find anything.
Try cv::Mat::row() and cv::Mat::col().
there 's a constructor for Mat(and an assignment operator, too), that takes a vector as input, so the same applies to rows of a Mat ( which are Mat's again )
Mat big(5,5,CV_32S); // 5x5, 1channel, int mat
vector<int> vec(5); vec[0]=1; vec[2] = 17; vec[4]=13; // make a row vec
big.row(1) = vec; // careful, does not work for col(), since that returns a copy
0 0 0 0 0
1 0 17 0 13
0 0 0 0 0
0 0 0 0 0
0 0 0 0 0
i was wrong here, sorry. both row() and col() make a copy of the original data, so assigning to that is useless.
I am trying to do image compression using DCT (Discrete Cosine Transform). Can someone please help me understand how masking affects bit per pixel in DCT? How is the bit allocation done in the masking?
PS: By masking, I mean multiplying the DCT coefficients with a matrix like the one below (element wise multiplication, not matrix multiplication).
mask = [1 1 1 1 0 0 0 0
1 1 1 0 0 0 0 0
1 1 0 0 0 0 0 0
1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0]
Background on "Masking"
Compression using the DCT calculates the DCT of blocks of an image, in this case 8x8 pixels. High frequency components of an image are less important for the human perception of an image and thus can be discarded in order to save space.
The mask Matrix selects which DCT coefficients are kept and which ones get discarded in order to save space. Coefficients towards the top left corner represent low frequencies.
For more information visit Discrete Cosine Transform.
This looks like a variation of quantization matrix.
Low frequencies are in top left, high frequencies are in bottom right. Eye is more sensitive to low frequencies, so removal of high frequency coefficients will remove less important details of the image.