Torch - repeat tensor like numpy repeat - lua

I am trying to repeat a tensor in torch in two ways. For example repeating the tensor {1,2,3,4} 3 times both ways to yield;
{1,2,3,4,1,2,3,4,1,2,3,4}
{1,1,1,2,2,2,3,3,3,4,4,4}
There is a built in torch:repeatTensor function which will generate the first of the two (like numpy.tile()) but I can't find one for the latter (like numpy.repeat()). I'm sure that I could call sort on the first to give the second but I think this might be computationally expensive for larger arrays?
Thanks.

Try torch.repeat_interleave() method: https://pytorch.org/docs/stable/torch.html#torch.repeat_interleave
>>> x = torch.tensor([1, 2, 3])
>>> x.repeat_interleave(2)
tensor([1, 1, 2, 2, 3, 3])

Quoting https://discuss.pytorch.org/t/how-to-tile-a-tensor/13853 -
z = torch.FloatTensor([[1,2,3],[4,5,6],[7,8,9]])
1 2 3
4 5 6
7 8 9
z.transpose(0,1).repeat(1,3).view(-1, 3).transpose(0,1)
1 1 1 2 2 2 3 3 3
4 4 4 5 5 5 6 6 6
7 7 7 8 8 8 9 9 9
This will give you a intuitive feel of how it works.

a = torch.Tensor([1,2,3,4])
To get [1., 2., 3., 4., 1., 2., 3., 4., 1., 2., 3., 4.] we repeat the tensor thrice in the 1st dimension:
a.repeat(3)
To get [1,1,1,2,2,2,3,3,3,4,4,4] we add a dimension to the tensor and repeat it thrice in the 2nd dimension to get a 4 x 3 tensor, which we can flatten.
b = a.reshape(4,1).repeat(1,3).flatten()
or
b = a.reshape(4,1).repeat(1,3).view(-1)

Here's a generic function that repeats elements in tensors.
def repeat(tensor, dims):
if len(dims) != len(tensor.shape):
raise ValueError("The length of the second argument must equal the number of dimensions of the first.")
for index, dim in enumerate(dims):
repetition_vector = [1]*(len(dims)+1)
repetition_vector[index+1] = dim
new_tensor_shape = list(tensor.shape)
new_tensor_shape[index] *= dim
tensor = tensor.unsqueeze(index+1).repeat(repetition_vector).reshape(new_tensor_shape)
return tensor
If you have
foo = tensor([[1, 2],
[3, 4]])
By calling repeat(foo, [2,1]) you get
tensor([[1, 2],
[1, 2],
[3, 4],
[3, 4]])
So you duplicated every element along dimension 0 and left elements as they are on dimension 1.

Use einops:
from einops import repeat
repeat(x, 'i -> (repeat i)', repeat=3)
# like {1,2,3,4,1,2,3,4,1,2,3,4}
repeat(x, 'i -> (i repeat)', repeat=3)
# like {1,1,1,2,2,2,3,3,3,4,4,4}
This code works identically for any framework (numpy, torch, tf, etc.)

Can you try something like:
import torch as pt
#1 work as numpy tile
b = pt.arange(10)
print(b.repeat(3))
#2 work as numpy tile
b = pt.tensor(1).repeat(10).reshape(2,-1)
print(b)
#3 work as numpy repeat
t = pt.tensor([1,2,3])
t.repeat(2).reshape(2,-1).transpose(1,0).reshape(-1)

Related

Batch wise batch normalization in TensorFlow

What is the correct way of performing batch wise batch normalization in TensorFlow? (I.e. I don't want to compute a running mean and variance). My current implementation is based on tf.nn.batch_normalization, where xis the output of a convolutional layer with shape [batch_size, width, height, num_channels]. I want to perform batch norm channel wise.
batch_mean, batch_var = tf.nn.moments(x, axes=[0, 1, 2])
x = tf.nn.batch_normalization(x, batch_mean, batch_var, offset=0, scale=0, variance_epsilon=1e-6)
But the results of this implementation are very bad. Comparison with tensorflow.contrib.slim.batch_norm shows that it is fare inferior (similarly bad training performance).
What am I doing wrong, and what can explain this bad performance?
You may consider tf.contrib.layers.layer_norm. You may want to reshape x to [batch, channel, width, height] and set begin_norm_axis=2 for channel wise normalization (each batch and each channel will be normalized independently).
Here is example how to reshape from your original order to [batch, channel, width, height]:
import tensorflow as tf
sess = tf.InteractiveSession()
batch = 2
height = 2
width = 2
channel = 3
tot_size = batch * height * channel * width
ts_4D_bhwc = tf.reshape(tf.range(tot_size), [batch, height, width, channel])
ts_4D_bchw = tf.transpose(ts_4D_bhwc, perm=[0,3,1,2])
print("Original tensor w/ order bhwc\n")
print(ts_4D_bhwc.eval())
print("\nTransormed tensor w/ order bchw\n")
print(ts_4D_bchw.eval())
Outputs:
Original tensor w/ order bhwc
[[[[ 0 1 2]
[ 3 4 5]]
[[ 6 7 8]
[ 9 10 11]]]
[[[12 13 14]
[15 16 17]]
[[18 19 20]
[21 22 23]]]]
Transormed tensor w/ order bchw
[[[[ 0 3]
[ 6 9]]
[[ 1 4]
[ 7 10]]
[[ 2 5]
[ 8 11]]]
[[[12 15]
[18 21]]
[[13 16]
[19 22]]
[[14 17]
[20 23]]]]
The solution by #Maosi works, but I found that it is slow. The following is simple and fast.
batch_mean, batch_var = tf.nn.moments(x, axes=[0, 1, 2])
x = tf.subtract(x, batch_mean)
x = tf.div(x, tf.sqrt(batch_var) + 1e-6)

Torch: How to shuffle a tensor by its rows?

I am currently working in torch to implement a random shuffle (on the rows, the first dimension in this case) on some input data. I am new to torch, so I have some troubles figuring out how permutation works..
The following is supposed to shuffle the data:
if argshuffle then
local perm = torch.randperm(sids:size(1)):long()
print("\n\n\nSize of X and y before")
print(X:view(-1, 1000, 128):size())
print(y:size())
print(sids:size())
print("\nPerm size is: ")
print(perm:size())
X = X:view(-1, 1000, 128)[{{perm},{},{}}]
y = y[{{perm},{}}]
print(sids[{{1}, {}}])
sids = sids[{{perm},{}}]
print(sids[{{1}, {}}])
print(X:size())
print(y:size())
print(sids:size())
os.exit(69)
end
This prints out
Size of X and y before
99
1000
128
[torch.LongStorage of size 3]
99
1
[torch.LongStorage of size 2]
99
1
[torch.LongStorage of size 2]
Perm size is:
99
[torch.LongStorage of size 1]
5
[torch.LongStorage of size 1x1]
5
[torch.LongStorage of size 1x1]
99
1000
128
[torch.LongStorage of size 3]
99
1
[torch.LongStorage of size 2]
99
1
[torch.LongStorage of size 2]
Out of the value, I can imply that the function did not shuffle the data. How can I make it shuffle correctly, and what is the common solution in lua/torch?
I also faced a similar issue. In the documentation, there is no shuffle function for tensors (there are for dataset loaders). I found a workaround to the problem using torch.randperm.
>>> a=torch.rand(3,5)
>>> print(a)
tensor([[0.4896, 0.3708, 0.2183, 0.8157, 0.7861],
[0.0845, 0.7596, 0.5231, 0.4861, 0.9237],
[0.4496, 0.5980, 0.7473, 0.2005, 0.8990]])
>>> # Row shuffling
...
>>> a=a[torch.randperm(a.size()[0])]
>>> print(a)
tensor([[0.4496, 0.5980, 0.7473, 0.2005, 0.8990],
[0.0845, 0.7596, 0.5231, 0.4861, 0.9237],
[0.4896, 0.3708, 0.2183, 0.8157, 0.7861]])
>>> # column shuffling
...
>>> a=a[:,torch.randperm(a.size()[1])]
>>> print(a)
tensor([[0.2005, 0.7473, 0.5980, 0.8990, 0.4496],
[0.4861, 0.5231, 0.7596, 0.9237, 0.0845],
[0.8157, 0.2183, 0.3708, 0.7861, 0.4896]])
I hope it answers the question!
dim = 0
idx = torch.randperm(t.shape[dim])
t_shuffled = t[idx]
If your tensor is e.g. of shape CxNxF (channels by rows by features), then you can shuffle along the second dimension like so:
dim=1
idx = torch.randperm(t.shape[dim])
t_shuffled = t[:,idx]
A straightforward solution is to use permutation matrices (those that are usual in linear algebra). Since you seem to be interested in the 3d case, we will have to flatten your 3d tensor first. So, here's an example code (ready-to use) that I came up with
data=torch.floor(torch.rand(5,3,2)*100):float()
reordered_data=data:view(5,-1)
perm=torch.randperm(5);
perm_rep=torch.repeatTensor(perm,5,1):transpose(1,2)
indexes=torch.range(1,5);
indexes_rep=torch.repeatTensor(indexes,5,1)
permutation_matrix=indexes_rep:eq(perm_rep):float()
permuted=permutation_matrix*reordered_data
print("perm")
print(perm)
print("before permutation")
print(data)
print("after permutation")
print(permuted:view(5,3,2))
As you will see from one execution, it reorders the tensor data according to the row indexes given in perm.
Based on your syntax, I assume you're using to torch with lua rather than PyTorch.
torch.Tensor.index is your function, it works like below:
x = torch.rand(4, 4)
p = torch.randperm(4)
print(x)
print(p)
print(x:index(1,p:long())

Hashfunction to map combinations of 5 to 7 cards

Referring to the original problem: Optimizing hand-evaluation algorithm for Poker-Monte-Carlo-Simulation
I have a list of 5 to 7 cards and want to store their value in a hashtable, which should be an array of 32-bit-integers and directly accessed by the hashfunctions value as index.
Regarding the large amount of possible combinations in a 52-card-deck, I don't want to waste too much memory.
Numbers:
7-card-combinations: 133784560
6-card-combinations: 20358520
5-card-combinations: 2598960
Total: 156.742.040 possible combinations
Storing 157 million 32-bit-integer values costs about 580MB. So I would like to avoid increasing this number by reserving memory in an array for values that aren't needed.
So the question is: How could a hashfunction look like, that maps each possible, non duplicated combination of cards to a consecutive value between 0 and 156.742.040 or at least comes close to it?
Paul Senzee has a great post on this for 7 cards (deleted link as it is broken and now points to a NSFW site).
His code is basically a bunch of pre-computed tables and then one function to look up the array index for a given 7-card hand (represented as a 64-bit number with the lowest 52 bits signifying cards):
inline unsigned index52c7(unsigned __int64 x)
{
const unsigned short *a = (const unsigned short *)&x;
unsigned A = a[3], B = a[2], C = a[1], D = a[0],
bcA = _bitcount[A], bcB = _bitcount[B], bcC = _bitcount[C], bcD = _bitcount[D],
mulA = _choose48x[7 - bcA], mulB = _choose32x[7 - (bcA + bcB)], mulC = _choose16x[bcD];
return _offsets52c[bcA] + _table4[A] * mulA +
_offsets48c[ (bcA << 4) + bcB] + _table [B] * mulB +
_offsets32c[((bcA + bcB) << 4) + bcC] + _table [C] * mulC +
_table [D];
}
In short, it's a bunch of lookups and bitwise operations powered by pre-computed lookup tables based on perfect hashing.
If you go back and look at this website, you can get the perfect hash code that Senzee used to create the 7-card hash and repeat the process for 5- and 6-card tables (essentially creating a new index52c7.h for each). You might be able to smash all 3 into one table, but I haven't tried that.
All told that should be ~628 MB (4 bytes * 157 M entries). Or, if you want to split it up, you can map it to 16-bit numbers (since I believe most poker hand evaluators only need 7,462 unique hand scores) and then have a separate map from those 7,462 hand scores to whatever hand categories you want. That would be 314 MB.
Here's a different answer based on the colex function concept. It works with bitsets that are sorted in descending order. Here's a Python implementation (both recursive so you can see the logic and iterative). The main concept is that, given a bitset, you can always calculate how many bitsets there are with the same number of set bits but less than (in either the lexicographical or mathematical sense) your given bitset. I got the idea from this paper on hand isomorphisms.
from math import factorial
def n_choose_k(n, k):
return 0 if n < k else factorial(n) // (factorial(k) * factorial(n - k))
def indexset_recursive(bitset, lowest_bit=0):
"""Return number of bitsets with same number of set bits but less than
given bitset.
Args:
bitset (sequence) - Sequence of set bits in descending order.
lowest_bit (int) - Name of the lowest bit. Default = 0.
>>> indexset_recursive([51, 50, 49, 48, 47, 46, 45])
133784559
>>> indexset_recursive([52, 51, 50, 49, 48, 47, 46], lowest_bit=1)
133784559
>>> indexset_recursive([6, 5, 4, 3, 2, 1, 0])
0
>>> indexset_recursive([7, 6, 5, 4, 3, 2, 1], lowest_bit=1)
0
"""
m = len(bitset)
first = bitset[0] - lowest_bit
if m == 1:
return first
else:
t = n_choose_k(first, m)
return t + indexset_recursive(bitset[1:], lowest_bit)
def indexset(bitset, lowest_bit=0):
"""Return number of bitsets with same number of set bits but less than
given bitset.
Args:
bitset (sequence) - Sequence of set bits in descending order.
lowest_bit (int) - Name of the lowest bit. Default = 0.
>>> indexset([51, 50, 49, 48, 47, 46, 45])
133784559
>>> indexset([52, 51, 50, 49, 48, 47, 46], lowest_bit=1)
133784559
>>> indexset([6, 5, 4, 3, 2, 1, 0])
0
>>> indexset([7, 6, 5, 4, 3, 2, 1], lowest_bit=1)
0
"""
m = len(bitset)
g = enumerate(bitset)
return sum(n_choose_k(bit - lowest_bit, m - i) for i, bit in g)

What does '**' in Ruby do?

What does the ** symbol mean in Ruby?
(1..5).map { |i| i**2 } # => [1, 4, 9, 16, 25]
Fixnum#** is the exponent operator. In your example, you are squaring i (raising it to the power of 2).
I have never used Ruby, but from the results I infer that i**2 means i^2 (that is, i*i):
1*1 = 1
2*2 = 4
3*3 = 9
4*4 = 16
5*5 = 25

How to split a matrix into several matrices based on another vector

If
A = [1 2 3; 4 5 6; 7 8 9]
B = [1 2 2]
I found that
A(B == 1, :) returns [1 2 3] and
A(B == 2, :) returns [4 5 6; 7 8 9]
because
B == 1 returns [1 0 0] and
B == 2 returns [0 1 1]
Given the above examples for A and B is there any easier way to get both the final matrices [1 2 3] and [4 5 6; 7 8 9] in one step i.e. without using a for loop.
Objective is to generate centroids of clusters to which each example(row in A) has been assigned in a K-means clustering problem. I am thinking of passing the resulting matrices to mean() function to generate the centroids.
You can get a cell array
A = [1 2 3; 4 5 6; 7 8 9];
B = [1 2 2];
arrayfun(#(lev) A(B==lev, :), unique(B), 'UniformOutput', false)
returns
ans =
{
[1,1] =
1 2 3
[1,2] =
4 5 6
7 8 9
}

Resources