I noticed an unusual behavior with torch7. I know a little about torch7. So I don't know how this behavior can be explained or corrected.
So, I am using CIFAR-10 dataset. Simply I fetched data for an image from CIFAR-10 and then saved it in my directory. When I loaded that saved image, it was different.
Here is my code -
require 'image'
i1 = testData.data[2] --fetching data from CIFAR-10
image.save("1.png", i) --saving the data as image
i2 = image.load("1.png") --loading the saved image
if(i1 == i2) then --checking if image1(i1) and image2(i2) are different
print("same")
end
Is this behavior expected? I thought png was supposed to be lossless.
If so how this can be corrected?
Code for loading CIFAR-10 dataset-
-- load dataset
trainData = {
data = torch.Tensor(50000, 3072),
labels = torch.Tensor(50000),
size = function() return trsize end
}
for i = 0,4 do
local subset = torch.load('cifar-10-batches-t7/data_batch_' .. (i+1) .. '.t7', 'ascii')
trainData.data[{ {i*10000+1, (i+1)*10000} }] = subset.data:t()
trainData.labels[{ {i*10000+1, (i+1)*10000} }] = subset.labels
end
trainData.labels = trainData.labels + 1
local subset = torch.load('cifar-10-batches-t7/test_batch.t7', 'ascii')
testData = {
data = subset.data:t():double(),
labels = subset.labels[1]:double(),
size = function() return tesize end
}
testData.labels = testData.labels + 1
testData.data = testData.data:reshape(10000,3,32,32)
== operator compares pointers to two tensors, not contents:
a = torch.Tensor(3, 5):fill(1)
b = torch.Tensor(3, 5):fill(1)
print(a == b)
> false
print(a:eq(b):all())
> true
Related
I would like to add information to my current dataset. At the moment, I have six-frame sequences in folders. The DataLoader reads all 6 and uses the first 3 for predicting the last 1/2/3 (depending on how many I tell him to). This is the function for the DataLoader.
class TrainFeeder(Dataset):
def init(self, data_set):
super(TrainFeeder, self).init()
self.input_data = data_set
#print(torch.cuda.current_device())
if torch.cuda.current_device() ==0:
print('There are total %d sequences in trainset' % len(self.input_data))
def getitem(self, index):
path = self.input_data[index]
imgs_path = sorted(glob.glob(path + '/*.png'))
imgs = []
for img_path in imgs_path:
img = cv2.imread(img_path)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img = cv2.resize(img, (256,448))
img = cv2.resize(img, (0, 0), fx=0.5, fy=0.5, interpolation=cv2.INTER_CUBIC) #has been 0.5 for official data, new is fx = 2.63 and fy = 2.84
img_tensor = ToTensor()(img).float()
imgs.append(img_tensor)
imgs = torch.stack(imgs, dim=0)
return imgs
def len(self):
return len(self.input_data)
Now I'd like to add one value to these images. It is a boolean, I have stored in a list in a .json in the same folder, like the six-frame-sequences. But I don't know how to add the values of the list in the .json to the tensor. Which dimension should I use? Will the system work at all, if I change the shape of the input?
The function getitem can return anything, so you can return a tuple instead of just images :
def __getitem__(self, index):
path = ...
# load your 6 images
imgs = torch.stack( ... )
# load your boolean metadata
metadata = load_json_data( ... )
# return them both
return (imgs, metadata)
You will need to make metadata a tensor before returning it, otherwise I expect that pytorch will complain about not being able to collate (i.e stack) them to make batches
"Will the system work" is a question only you can answer, since you did not provide the code of your ML model. I would bet on : "no but it won't require significant changes to work". Most likely you currently have a loop like
for imgs in dataloader:
# do some training
output = model(imgs)
...
And you will have to make it like
for imgs, metadata in dataloader:
# do some training
output = model(imgs)
...
i am doing covid-19 classification.i took dataset from kaggle. it has folder named dataset which contain 3 folders normal pnuemonia and covid-19 each contaning images for these classes i am stucked in writting getitem in pytorch custom dataloader ?
Dataset has 189 covid images but by this get item i get 920 images of covid kindly help
class_names = ['normal', 'viral', 'covid']
root_dir = 'COVID-19 Radiography Database'
source_dirs = ['NORMAL', 'Viral Pneumonia', 'COVID-19']
if os.path.isdir(os.path.join(root_dir, source_dirs[1])):
os.mkdir(os.path.join(root_dir, 'test'))
for i, d in enumerate(source_dirs):
os.rename(os.path.join(root_dir, d), os.path.join(root_dir, class_names[i]))
for c in class_names:
os.mkdir(os.path.join(root_dir, 'test', c))
for c in class_names:
images = [x for x in os.listdir(os.path.join(root_dir, c)) if x.lower().endswith('png')]
selected_images = random.sample(images, 30)
for image in selected_images:
source_path = os.path.join(root_dir, c, image)
target_path = os.path.join(root_dir, 'test', c, image)
shutil.move(source_path, target_path)
Above code is used to create test dataset which has 30 images of each class
class ChestXRayDataset(torch.utils.data.Dataset):
def __init__(self, image_dirs, transform):
def get_images(class_name):
images = [x for x in os.listdir(image_dirs[class_name]) if
x[-3:].lower().endswith('png')]
print(f'Found {len(images)} {class_name} examples')
return images
self.images = {}
self.class_names = ['normal', 'viral', 'covid']
for class_name in self.class_names:
self.images[class_name] = get_images(class_name)
self.image_dirs = image_dirs
self.transform = transform
def __len__(self):
return sum([len(self.images[class_name]) for class_name in self.class_names])
def __getitem__(self, index):
class_name = random.choice(self.class_names)
index = index % len(self.images[class_name])
image_name = self.images[class_name][index]
image_path = os.path.join(self.image_dirs[class_name], image_name)
image = Image.open(image_path).convert('RGB')
return self.transform(image), self.class_names.index(class_name)
**Stucked in get item of this **
images in folder are arranged as follows
Dataset is as follows
**Code for confusion matrix is **
nb_classes = 3
confusion_matrix = torch.zeros(nb_classes, nb_classes)
with torch.no_grad():
for data in tqdm_notebook(dl_train,total=len(dl_train),unit='batch'):
img,lab = data
print(lab)
img,lab = img.to(device),lab.to(device)
_,output = torch.max(model(img),1)
print(output)
for t, p in zip(lab.view(-1), output.view(-1)):
confusion_matrix[t.long(), p.long()] += 1
output for confusion matrix only one class is getting trained
confusio matrix image
Putting you images in a dictionary complicates the manipulation, rather use a list. Also you Dataset should not have any randomness, shuffling of the data should happen from the DataLoader not from the Dataset.
Use something like below:
class ChestXRayDataset(torch.utils.data.Dataset):
def __init__(self, image_dirs, transform):
def get_images(class_name):
images = [x for x in os.listdir(image_dirs[class_name]) if
x[-3:].lower().endswith('png')]
print(f'Found {len(images)} {class_name} examples')
return images
self.images = []
self.labels = []
self.class_names = ['normal', 'viral', 'covid']
for class_name in self.class_names:
images = get_images(class_name)
# This is a list containing all the images
self.images.extend(images)
# This is a list containing all the corresponding image labels
self.labels.extend([class_name]*len(images))
self.image_dirs = image_dirs
self.transform = transform
def __len__(self):
return len(self.images)
# Will return the image and its label at the position `index`
def __getitem__(self, index):
# image at index position of all the images
image_name = self.images[index]
# Its label
class_name = self.labels[index]
image_path = os.path.join(self.image_dirs[class_name], image_name)
image = Image.open(image_path).convert('RGB')
return self.transform(image), self.class_names.index(class_name)
If you enumerate it say using
ds = ChestXRayDataset(image_dirs, transform)
for x, y in ds:
print (x.shape, y)
You should see all the images and the labels in the sequential order.
However in real case you would rather use a Torch DataLoader and pass it the ds object with shuffle parameter set to True. So the DataLoader will take care of shuffling the Dataset by calling the __getitem__ with shuffled index values.
How do I get the full path to the required value in the table? I want to track changes in another table through a proxy table.
I understand that I need to use metatables and __index in it. But I haven't been able to come up with a tracker yet.
Sample table structure:
Objects = {
Panel = { layer = 1, x = 600, y = 328, w = 331, h = 491;
objects = {
label = { layer = 1, x = 0, y = 0, text = 'header' };
Window = { layer = 2, x = 400, y = 100, w = 100, h = 100;
objects = {
label = { layer = 1, x = 0, y = 0, text = 'lorem ipsum dorem' };
};
};
};
};
};
Path: Objects.Panel.objects.Window.objects.label.text
I tried to create a metatable for each of the tables and collect the result of each call to __index into a table in order to roughly understand which key and value were retrieved or changed in order to synchronize these values with other tables.
This will prove itself to be horrendously slow and memory inefficient. Anyway, you were right on the track: proxy and handle __index and __newindex metamethods to your liking. This being said you also need to track the state of the proxy somehow.
You can try to hide it with some closures and upvalues but the easy way is to store the information directly in the proxy tables:
function make_tracker (o, name)
local mt = {}
mt.__index = function (proxy, key)
local path = {unpack(rawget(proxy, "__path"))} -- Stupid shallow copy
local object = rawget(proxy, "__to")
table.insert(path, key)
if type(object[key]) == "table" then
return setmetatable({__to = object[key], __path = path}, mt)
else
return table.concat(path, ".") .. " = " .. tostring(object[key])
end
end
return setmetatable({__to = o, __path = {name}}, mt)
end
__to fields indicates what proxy should point to and __path is there to cover fields we have trespassed so far. It does a shallow copy, so that one can use subproxies with local variables. name parameter is there to initialize the name of the first table, as you just simply can't know that. You use it like this:
local tObjects = make_tracker(Objects, "Objects")
local subproxy = tObjects.Panel.objects.Window
print(subproxy.objects.label.text)
print(tObjects.Panel.objects.label.text)
print(subproxy.x)
-- prints:
-- Objects.Panel.objects.Window.objects.label.text = lorem ipsum dorem
-- Objects.Panel.objects.label.text = header
-- Objects.Panel.objects.Window.x = 400
Of course, I doubt that appending the path to the original value is what you want. Modify insides of else block:
return table.concat(path, ".") .. " = " .. tostring(object[key])
according to your needs, e.g:
register_tracked_path(table.concat(path, "."))
return object[key]
If you want to handle modification of values you need to extend the metatable with similar __newindex.
I'm a novice with Lua/Torch. I have an existing model that includes a max pooling layer. I want to take the input into that layer and split it into chunks, feeding each chunk into a new max pooling layer.
I have written a stand-alone Lua script that can split a tensor into two chunks and forward the two chunks into a network with two max-pooling layers.
But trying to integrate that back into the existing model I can't figure out how to amend the data "mid-flow", as it were, to do the tensor split. I've read the docs and can't see any function or example of architecture that somewhere along the line splits a tensor into two and forwards each part separately.
Any ideas? Thanks!
you want define a layer yourself.
The layer will be like below, if your layer input is one dimension:
CSplit, parent = torch.class('nn.CSplit', 'nn.Module')
function CSplit:__init(firstCount)
self.firstCount = firstCount
parent.__init(self)
end
function CSplit:updateOutput(input)
local inputSize = input:size()[1]
local firstCount = self.firstCount
local secondCount = inputSize - firstCount
local first = torch.Tensor(self.firstCount)
local second = torch.Tensor(secondCount)
for i=1, inputSize do
if i <= firstCount then
first[i] = input[i]
else
second[i - firstCount] = input[i]
end
end
self.output = {first, second}
return self.output
end
function CSplit:updateGradInput(input, gradOutput)
local inputSize = input:size()[1]
self.gradInput = torch.Tensor(input)
for i=1, inputSize do
if i <= self.firstCount then
self.gradInput[i] = gradOutput[1][i]
else
self.gradInput[i] = gradOutput[2][i-self.firstCount]
end
end
return self.gradInput
end
How to use it? you need to specify the first chunk size like the code below.
testNet = nn.CSplit(4)
input = torch.randn(10)
output = testNet:forward(input)
print(input)
print(output[1])
print(output[2])
testNet:backward(input, {torch.randn(4), torch.randn(6)})
you can see runnable iTorch notebook code here
Following code works quite well when used without scaling, but when scaling is applied results are too far from actual. Here is the code:
data =(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63).
model = SVR(kernel='poly', C=1e3, degree=3)
data_min = min(data)
data_max = max(data)
diff = data_max - data_min
data_scaled = []
for i in range(0,len(data)):
data_scaled.append((data[i]-data_min)/diff)
data_scaled = np.matrix(data_scaled)
data_scaled = data_scaled.reshape(-1,1)
y = (1,8,27,64,125,216,343,512,729,1000,1331,1728,2197,2744,3375,4096,4913,5832,6859,8000,9261,10648,12167,13824,15625,17576,19683,21952,24389,27000,29791,32768,35937,39304,42875,46656,50653,54872,59319,64000,68921,74088,79507,85184,91125,97336,103823,110592,117649,125000,132651,140608,148877,157464,166375,175616,185193,195112,205379,216000,226981,238328,250047)
model.fit(data_scaled, y)
predicted = model.predict(data_scaled)