I have a matrix defined mxn 128x128. And I have translated my 2d x,y positions onto this 1D matrix grid. My 2d coordinates accept positions using numbers 0->127 i.e. any combo in ranges {x=0,y=0}-->{x=127,y=127}. I'm implementing algorithms that take the neighboring positions of these nodes. Specifically the 8 surrounding positions of distance i (lets say i=1). So considering node={0,0}, my neighbours are generated by adding these vectors to said node:
two_d_nodes={
{0,i*1},{0,-i*1},{-i*1,0},{i*1,0},
{i*1,i*1},{i*1,-i*1},{-i*1,-i*1},{-i*1,i*1}
}
In terms of 2d though I am excluding neighbours outside the boundary. So in the above for node={0,0}, only neighours {0,1},{1,1}{1,0} are generated. Setting the boundary is basically just implementing some form of:
if x>=0 and y>=0 and x<=127 and y<=127 then...
The 1d translation of node={0,0} is node={0} and my vector additions translated to 1d are:
one_d_nodes={{128},{-128},{-1},{1},{129},{-127},{-129},{127}}
However the relationship with the 2d boundary expressions doesn't hold true here. Or at least I don't know how to translate it. In response I tried generating all the loose cases of the grid:
{0,127,16256,16383} --the 4 corner positions
node%128==0 --right-side boundary
node%128==1 --left-side boundary
node>1 and node<128 --top-side boundary
node>127*128 and node<128*128 --bottom-side boundary
Then tried implementing special cases....where I just ignored generating the specific out of bounds neighbours. That was messy, and didn't even work for some reason. Regardless I feel I am missing a much cleaner method.
So my question is: How do I translate my 2d boundaries onto my 1d grid for the purposes of only generating neighbours within the boundary?
The following is in regards to the answer below:
function newmatrix(node) --node={x=0,y=0}
local matrix={}
add(matrix,{(node.y<<8)+node.x}) --matrix= {{0},...}
--lets say [1 2 3] is a width=3; height=1 matrix,
--then the above line maps my 2d coord to a matrix of width=256, height=128
matrix.height, matrix.width = #node,#node[1] --1,1
return matrix
end
function indexmatrix(matrix, r,c)
if r > 1 and r <= matrix.height and c > 1 and c <= matrix.width then
return matrix[matrix.width * r + c]
else
return false
end
end
function getneighbors(matrix, r, c)
local two_d_nodes={
{0,1},{0,-1},{-1,0},{1,0},
{1,1},{1,-1},{-1,-1},{-1,1}
}
local neighbors = {}
for index, node in ipairs(two_d_nodes) do
table.insert(neighbors, indexmatrix(matrix, r + node[1], c + node[2]))
end
return neighbors
end
--Usage:
m={x=0,y=0}
matrix=newmatrix(m) --{{0}}
--here's where I'm stuck, cause idk what r and c are
--normally I'd grab my neighbors next....
neighbors=getneighbors(matrix)
--then I have indexmatrix for...?
--my understanding is that I am using indexmatrix to
--check if the nieghbors are within the bounds or not, is that right?
--can you illustrate how it would work for my code here, it should
--toss out anything with x less then 0 and y less than 0. Same as in OP's ex
indexmatrix(matrix) ---not sure what to do here
Attempt 2 in regards to the comment sections below:
function indexmatrix(matrix, x ,y)
if x > 1 and x <= matrix['height'] and y > 1 and y <= matrix['width'] then
return matrix[matrix['width'] * x + y]
else
return false
end
end
function getneighbors(matrix, pos_x, pos_y)
local two_d_nodes={
{0,1},{0,-1},{-1,0},{1,0},
{1,1},{1,-1},{-1,-1},{-1,1}
}
local neighbors = {}
for _, node in ipairs(two_d_nodes) do
add(neighbors, indexmatrix(matrix, pos_x + node[1], pos_y + node[2]))
end
return neighbors
end
matrix={} --128 columns/width, 128 rows/height
for k=1,128 do
add(matrix,{}) ----add() is same as table.insert()
for i=1,128 do
matrix[k][i]=i
end
end
id_matrix={{}} --{ {1...16k}}
for j=1,128*128 do
id_matrix[1][j]=j
end
id_matrix.height, id_matrix.width = 128,128
position={x=0,y=0}
neighbors = getNeighbors(matrix, position.x, position.y)
Attempt 3: A working dumbed down version of the code given. Not what I wanted at all.
function indexmatrix(x,y)
if x>=0 and y>=0 and x<127 and y<127 then
return 128 * x + y
else
return false
end
end
function getneighbors(posx,posy)
local two_d_nodes={
{0,1},{0,-1},{-1,0},{1,0},
{1,1},{1,-1},{-1,-1},{-1,1}
}
local neighbors = {}
for _, node in pairs(two_d_nodes) do
add(neighbors, indexmatrix(posx+node[1], posy + node[2]))
end
return neighbors
end
pos={x=0,y=10}
neighbors = getneighbors(pos.x,pos.y)
Edit: The equation to map 2D coordinates to 1D, y = mx + z, is a function of two variables. It is not possible for a multivariable equation to have a single solution unless a system of equations is given that gets x or z in terms of the other variable. Because x and z are independent of one another, the short answer to the question is: no
Instead, the constraints on x and z must be used to ensure integrity of the 1D coordinates.
What follows is an example of how to work with a 1D array as if it were a 2D matrix.
Let's say we have a constructor that maps a 2D table to a 1D matrix
local function newMatrix(m) -- m is 128x128 Matrix
local Matrix = {}
--logic to map m to 1D array
-- ...
return Matrix -- Matrix is m 1x16384 Array
end
The numeric indices are reserved, but we can add non-numeric keys to store information about the matrix. Let's store the number of rows and columns as height and width. We can do this in the constructor
local function newMatrix(m)
local Matrix = {}
--logic to map to 1D array
-- ...
-- Store row and column info in the matrix
Matrix.height, Matrix.width = #m, #m[1] -- Not the best way
return Matrix
end
Although the matrix is now a 1x16384 array, we can create a function that allows us to interact with the 1D array like it's still a 2D matrix. This function will get the value of a position in the matrix, but we return false/nil if the indices are out of bounds.
To be clear, the formula to map 2D coordinates to a 1D coordinate for a matrix, and can be found here:
1D position = 2D.x * Matrix-Width + 2D.y
And here's what that function could look like:
local function indexMatrix(Matrix, r,c)
if r >= 1 and r <= Matrix.height and c >= 1 and c <= Matrix.width then
return Matrix[Matrix.width * r + c] -- the above formula
else
return false -- out of bounds
end
end
We can now index our Matrix with any bounds without fear of returning an incorrect element.
Finally, we can make a function to grab the neighbors given a 2D position. In this function, we add vectors to the given 2D position to get surrounding positions, and then index the matrix using the indexMatrix function. Because indexMatrix checks if a 2D position is within the bounds of the original Matrix (before it was converted), we only get neighbors that exist.
local function getNeighbors(Matrix, r, c) -- r,c = row, column (2D position)
local two_d_nodes={
{0,1},{0,-1},{-1,0},{1,0},
{1,1},{1,-1},{-1,-1},{-1,1}
}
local neighbors = {}
for index, node in ipairs(two_d_nodes) do
-- Add each vector to the given position and get the node from the Matrix
table.insert(neighbors, indexMatrix(Matrix, r + node[1], c + node[2]))
end
return neighbors
end
You can either skip elements that return false from indexMatrix or remove them after the fact. Or anything else that sounds better to you (this code is not great, it's just meant to be an example). Wrap it in a for i ... do loop and you can go out an arbitrary distance.
I hope I haven't assumed too much and that this is helpful. Just know it's not foolproof (the # operator stops counting at the first nil, for instance)
Edit: Usage
Matrix = {
{1,2,3...128}, -- row 1
{1,2,3...128},
...
{1,2,3...128}, -- row 128
}
Array = newMatrix(Matrix) -- Convert to 1D Array ({1,2,3,...,16384})
--Array.width = 128, Array.height = 128
position = {x=0, y=0}
neighbors = getNeighbors(Array, position.x, position.y)
-- neighbors is: {{0,1}, false, false, {1,0}, {1,1}, false, false, false}
Related
I'm working on a problem where I have all the variables as categorical variables and applied MCA. When I visualize MCA results combined with clusters obtained through K-modes (applied independently of MCA), the clusters overlap with each other. I was wondering instead of applying k-modes, I should simply get MCA components and apply K-means or other clustering algorithm on those components. Does that make sense?
I don't think K-Means allows overlapping. The sample result is assigned to closest cluster, but not to all, so there is no overlapping. Check out the code sample below.
import numpy as np
import matplotlib.pyplot as plt
from scipy.spatial import Voronoi
def voronoi_finite_polygons_2d(vor, radius=None):
"""
Reconstruct infinite voronoi regions in a 2D diagram to finite
regions.
Parameters
----------
vor : Voronoi
Input diagram
radius : float, optional
Distance to 'points at infinity'.
Returns
-------
regions : list of tuples
Indices of vertices in each revised Voronoi regions.
vertices : list of tuples
Coordinates for revised Voronoi vertices. Same as coordinates
of input vertices, with 'points at infinity' appended to the
end.
"""
if vor.points.shape[1] != 2:
raise ValueError("Requires 2D input")
new_regions = []
new_vertices = vor.vertices.tolist()
center = vor.points.mean(axis=0)
if radius is None:
radius = vor.points.ptp().max()*2
# Construct a map containing all ridges for a given point
all_ridges = {}
for (p1, p2), (v1, v2) in zip(vor.ridge_points, vor.ridge_vertices):
all_ridges.setdefault(p1, []).append((p2, v1, v2))
all_ridges.setdefault(p2, []).append((p1, v1, v2))
# Reconstruct infinite regions
for p1, region in enumerate(vor.point_region):
vertices = vor.regions[region]
if all([v >= 0 for v in vertices]):
# finite region
new_regions.append(vertices)
continue
# reconstruct a non-finite region
ridges = all_ridges[p1]
new_region = [v for v in vertices if v >= 0]
for p2, v1, v2 in ridges:
if v2 < 0:
v1, v2 = v2, v1
if v1 >= 0:
# finite ridge: already in the region
continue
# Compute the missing endpoint of an infinite ridge
t = vor.points[p2] - vor.points[p1] # tangent
t /= np.linalg.norm(t)
n = np.array([-t[1], t[0]]) # normal
midpoint = vor.points[[p1, p2]].mean(axis=0)
direction = np.sign(np.dot(midpoint - center, n)) * n
far_point = vor.vertices[v2] + direction * radius
new_region.append(len(new_vertices))
new_vertices.append(far_point.tolist())
# sort region counterclockwise
vs = np.asarray([new_vertices[v] for v in new_region])
c = vs.mean(axis=0)
angles = np.arctan2(vs[:,1] - c[1], vs[:,0] - c[0])
new_region = np.array(new_region)[np.argsort(angles)]
# finish
new_regions.append(new_region.tolist())
return new_regions, np.asarray(new_vertices)
# make up data points
np.random.seed(1234)
points = np.random.rand(15, 2)
# compute Voronoi tesselation
vor = Voronoi(points)
# plot
regions, vertices = voronoi_finite_polygons_2d(vor)
print("--")
print(regions)
print("--")
print(vertices)
# colorize
for region in regions:
polygon = vertices[region]
plt.fill(*zip(*polygon), alpha=0.4)
plt.plot(points[:,0], points[:,1], 'ko')
plt.axis('equal')
plt.xlim(vor.min_bound[0] - 0.1, vor.max_bound[0] + 0.1)
plt.ylim(vor.min_bound[1] - 0.1, vor.max_bound[1] + 0.1)
I think some clustering algos actually do allow overlapping. Do a Google search and you will find what you are looking for.
Hope that helps.
I want to check the optimal number of k using the elbow method. I'm not using the scikit-learn library. I have my k-means coded from scratch and now I'm having a difficult time figuring out how to code the elbow method in python. I'm a total beginner.
This is my k-means code:
def cluster_init(array, k):
initial_assgnm = np.append(np.arange(k), np.random.randint(0, k, size=(len(array))))[:len(array)]
np.random.shuffle(initial_assgnm)
zero_arr = np.zeros((len(initial_assgnm), 1))
for indx, cluster_assgnm in enumerate(initial_assgnm):
zero_arr[indx] = cluster_assgnm
upd_array = np.append(array, zero_arr, axis=1)
return upd_array
def kmeans(array, k):
cluster_array = cluster_init(array, k)
while True:
unique_clusters = np.unique(cluster_array[:, -1])
centroid_dictonary = {}
for cluster in unique_clusters:
centroid_dictonary[cluster] = np.mean(cluster_array[np.where(cluster_array[:, -1] == cluster)][:, :-1], axis=0)
start_array = np.copy(cluster_array)
for row in range(len(cluster_array)):
cluster_array[row, -1] = unique_clusters[np.argmin(
[np.linalg.norm(cluster_array[row, :-1] - centroid_dictonary.get(cluster)) for cluster in unique_clusters])]
if np.array_equal(cluster_array, start_array):
break
return centroid_dictonary
This is what I have tried for the elbow method:
cost = []
K= range(1,239)
for k in K :
KM = kmeans(x,k)
print(k)
KM.fit(x)
cost.append(KM.inertia_)
But I get the following error
KM.fit(x)
AttributeError: 'dict' object has no attribute 'fit'
If you want to compute the elbow values from scratch, you need to compute the inertia for the current clustering assigment. To do this, you can compute the sum of the particle inertias. The particle inertia from a data point is the distance from its current position, to the closest center. If you have a function that computes this for you (in scikit-learn this function corresponds to pairwise_distances_argmin_min) you could do
labels, mindist = pairwise_distances_argmin_min(
X=X, Y=centers, metric='euclidean', metric_kwargs={'squared': True})
inertia = mindist.sum()
If you actually wanted to write this function what you would do is loop over every row x in X, find the minimum over all y in Y of dist(x,y), and this would be your inertia for x. This naive method of computing the particle inertias is O(nk), so you might consider using the library function instead.
I have a sfc_multipoint object and want to use st_buffer but with different distances for every single point in the multipoint object.
Is that possible?
The multipoint object are coordinates.
table = data
Every coordinate point (in the table in "lon" and "lat") should have a buffer with a different size. This buffer size is containt in the table in row "dist".
The table is called data.
This is my code:
library(sf)
coords <- matrix(c(data$lon,data$lat), ncol = 2)
tt <- st_multipoint(coords)
sfc <- st_sfc(tt, crs = 4326)
dt <- st_sf(data.frame(geom = sfc))
web <- st_transform(dt, crs = 3857)
geom <- st_geometry(web)
buf <- st_buffer(geom, dist = data$dist)
But it uses just the first dist of (0.100).
This is the result. Just really small buffers.
small buffer
For visualization see this picture. It´s just an example to show that the buffer should get bigger. example result
I think that he problem here is in how you are "creating" the points dataset.
Replicating your code with dummy data, doing this:
library(sf)
data <- data.frame(lat = c(0,1,2,3), lon = c(0,1,2,3), dist = c(0.1,0.2,0.3, 0.4))
coords <- matrix(c(data$lon,data$lat), ncol = 2)
tt <- st_multipoint(coords)
does not give you multiple points, but a single MULTIPOINT feature:
tt
#> MULTIPOINT (0 0, 1 1, 2 2, 3 3)
Therefore, only a single buffer distance can be "passed" to it and you get:
plot(sf::st_buffer(tt, data$dist))
To solve the problem, you need probably to build the point dataset differently. For example, using:
tt <- st_as_sf(data, coords = c("lon", "lat"))
gives you:
tt
#> Simple feature collection with 4 features and 1 field
#> geometry type: POINT
#> dimension: XY
#> bbox: xmin: 0 ymin: 0 xmax: 3 ymax: 3
#> epsg (SRID): NA
#> proj4string: NA
#> dist geometry
#> 1 0.1 POINT (0 0)
#> 2 0.2 POINT (1 1)
#> 3 0.3 POINT (2 2)
#> 4 0.4 POINT (3 3)
You see that tt is now a simple feature collection made of 4 points, on which buffering with multiple distances will indeed work:
plot(sf::st_buffer(tt, data$dist))
HTH!
I am trying to implement a custom Keras objective function:
in 'Direct Intrinsics: Learning Albedo-Shading Decomposition by Convolutional Regression', Narihira et al.
This is the sum of equations (4) and (6) from the previous picture. Y* is the ground truth, Y a prediction map and y = Y* - Y.
This is my code:
def custom_objective(y_true, y_pred):
#Eq. (4) Scale invariant L2 loss
y = y_true - y_pred
h = 0.5 # lambda
term1 = K.mean(K.sum(K.square(y)))
term2 = K.square(K.mean(K.sum(y)))
sca = term1-h*term2
#Eq. (6) Gradient L2 loss
gra = K.mean(K.sum((K.square(K.gradients(K.sum(y[:,1]), y)) + K.square(K.gradients(K.sum(y[1,:]), y)))))
return (sca + gra)
However, I suspect that the equation (6) is not correctly implemented because the results are not good. Am I computing this right?
Thank you!
Edit:
I am trying to approximate (6) convolving with Prewitt filters. It works when my input is a chunk of images i.e. y[batch_size, channels, row, cols], but not with y_true and y_pred (which are of type TensorType(float32, 4D)).
My code:
def cconv(image, g_kernel, batch_size):
g_kernel = theano.shared(g_kernel)
M = T.dtensor3()
conv = theano.function(
inputs=[M],
outputs=conv2d(M, g_kernel, border_mode='full'),
)
accum = 0
for curr_batch in range (batch_size):
accum = accum + conv(image[curr_batch])
return accum/batch_size
def gradient_loss(y_true, y_pred):
y = y_true - y_pred
batch_size = 40
# Direction i
pw_x = np.array([[-1,0,1],[-1,0,1],[-1,0,1]]).astype(np.float64)
g_x = cconv(y, pw_x, batch_size)
# Direction j
pw_y = np.array([[-1,-1,-1],[0,0,0],[1,1,1]]).astype(np.float64)
g_y = cconv(y, pw_y, batch_size)
gra_l2_loss = K.mean(K.square(g_x) + K.square(g_y))
return (gra_l2_loss)
The crash is produced in:
accum = accum + conv(image[curr_batch])
...and error description is the following one:
*** TypeError: ('Bad input argument to theano function with name "custom_models.py:836" at index 0 (0-based)', 'Expected an array-like
object, but found a Variable: maybe you are trying to call a function
on a (possibly shared) variable instead of a numeric array?')
How can I use y (y_true - y_pred) as a numpy array, or how can I solve this issue?
SIL2
term1 = K.mean(K.square(y))
term2 = K.square(K.mean(y))
[...]
One mistake spread across the code was that when you see (1/n * sum()) in the equations, it is a mean. Not the mean of a sum.
Gradient
After reading your comment and giving it more thought, I think there is a confusion about the gradient. At least I got confused.
There are two ways of interpreting the gradient symbol:
The gradient of a vector where y should be differentiated with respect to the parameters of your model (usually the weights of the neural net). In previous edits I started to write in this direction because that's the sort of approach used to trained the model (eg. gradient descent). But I think I was wrong.
The pixel intensity gradient in a picture, as you mentioned in your comment. The diff of each pixel with its neighbor in each direction. In which case I guess you have to translate the example you gave into Keras.
To sum up, K.gradients() and numpy.gradient() are not used in the same way. Because numpy implicitly considers (i, j) (the row and column indices) as the two input variables, while when you feed a 2D image to a neural net, every single pixel is an input variable. Hope I'm clear.
I've been working on a project that renders a Mandelbrot fractal. For those of you who know, it is generated by iterating through the following function where c is the point on a complex plane:
function f(c, z) return z^2 + c end
Iterating through that function produces the following fractal (ignore the color):
When you change the function to this, (z raised to the third power)
function f(c, z) return z^3 + c end
the fractal should render like so (again, the color doesn't matter):
(source: uoguelph.ca)
However, when I raised z to the power of 3, I got an image extremely similar as to when you raise z to the power of 2. How can I make the fractal render correctly? This is the code where the iterations are done: (the variables real and imaginary simply scale the screen from -2 to 2)
--loop through each pixel, col = column, row = row
local real = (col - zoomCol) * 4 / width
local imaginary = (row - zoomRow) * 4 / width
local z, c, iter = 0, 0, 0
while math.sqrt(z^2 + c^2) <= 2 and iter < maxIter do
local zNew = z^2 - c^2 + real
c = 2*z*c + imaginary
z = zNew
iter = iter + 1
end
So I recently decided to remake a Mandelbrot fractal generator, and it was MUCH more successful than my attempt last time, as my programming skills have increased with practice.
I decided to generalize the mandelbrot function using recursion for anyone who wants it. So, for example, you can do f(z, c) z^2 + c or f(z, c) z^3 + c
Here it is for anyone that may need it:
function raise(r, i, cr, ci, pow)
if pow == 1 then
return r + cr, i + ci
end
return raise(r*r-i*i, 2*r*i, cr, ci, pow - 1)
end
and it's used like this:
r, i = raise(r, i, CONSTANT_REAL_PART, CONSTANT_IMAG_PART, POWER)