I'm trying to implement a function that computes the Relu derivative for each element in a matrix, and then return the result in a matrix. I'm using Python and Numpy.
Based on other Cross Validation posts, the Relu derivative for x is
1 when x > 0, 0 when x < 0, undefined or 0 when x == 0
Currently, I have the following code so far:
def reluDerivative(self, x):
return np.array([self.reluDerivativeSingleElement(xi) for xi in x])
def reluDerivativeSingleElement(self, xi):
if xi > 0:
return 1
elif xi <= 0:
return 0
Unfortunately, xi is an array because x is an matrix. reluDerivativeSingleElement function doesn't work on array. So I'm wondering is there a way to map values in a matrix to another matrix using numpy, like the exp function in numpy?
Thanks a lot in advance.
That's an exercise in vectorization.
This code
if x > 0:
y = 1
elif xi <= 0:
y = 0
Can be reformulated into
y = (x > 0) * 1
This is something that will work for numpy arrays, since boolean expressions involving them are turned into arrays of values of these expressions for elements in said array.
I guess this is what you are looking for:
>>> def reluDerivative(x):
... x[x<=0] = 0
... x[x>0] = 1
... return x
>>> z = np.random.uniform(-1, 1, (3,3))
>>> z
array([[ 0.41287266, -0.73082379, 0.78215209],
[ 0.76983443, 0.46052273, 0.4283139 ],
[-0.18905708, 0.57197116, 0.53226954]])
>>> reluDerivative(z)
array([[ 1., 0., 1.],
[ 1., 1., 1.],
[ 0., 1., 1.]])
Basic function to return derivative of relu could be summarized as follows:
f'(x) = x > 0
So, with numpy that would be:
def relu_derivative(z):
return np.greater(z, 0).astype(int)
def dRelu(z):
return np.where(z <= 0, 0, 1)
Here z is a ndarray in my case.
def reluDerivative(self, x):
return 1 * (x > 0)
You are on a good track: thinking on vectorized operation. Where we define a function, and we apply this function to a matrix, instead of writing a for loop.
This threads answers your question, where it replace all the elements satisfy the condition. You can modify it into ReLU derivative.
https://stackoverflow.com/questions/19766757/replacing-numpy-elements-if-condition-is-met
In addition, python supports functional programming very well, try to use lambda function.
https://www.python-course.eu/lambda.php
This works:
def dReLU(x):
return 1. * (x > 0)
As mentioned by Neil in the comments, you can use heaviside function of numpy.
def reluDerivative(self, x):
return np.heaviside(x, 0)
If you want to use pure Python:
def relu_derivative(x):
return max(sign(x), 0)
If you want it with the derivative you can use:
def relu(neta):
relu = neta * (neta > 0)
d_relu = (neta > 0)
return relu, d_relu
When x is larger than 0, the slope is 1.
When x is smaller than or equal to 0, the slope is 0.
if (x > 0):
return 1
if (x <= 0):
return 0
This can be written more compact:
return 1 * (x > 0)
Related
could anyone explain how to write a custom multiclass metrics for Keras? I tried to write custom metric but encountered some issue. Main problem is I am not familiar with how tensor works during training (I think it is called Graph mode?). I am able to create confusion matrix and derived F1 score using NumPy or Python list.
I printed out the y-true and y_pred and tried to understand them, but the output was not what I expected:
Below is the function I used:
def f1_scores(y_true,y_pred):
y_true = K.print_tensor(y_true, message='y_true = ')
y_pred = K.print_tensor(y_pred, message='y_pred = ')
print(f"y_true_shape:{K.int_shape(y_true)}")
print(f"y_pred_shape:{K.int_shape(y_pred)}")
y_true_f = K.flatten(y_true)
y_pred_f = K.flatten(y_pred)
gt = K.argmax(y_true_f)
pred = K.argmax(y_pred_f)
print(f"pred_print:{pred}")
print(f"gt_print:{gt}")
pred = K.print_tensor(pred, message='pred= ')
gt = K.print_tensor(gt, message='gt =')
print(f"pred_shape:{K.int_shape(pred)}")
print(f"gt_shape:{K.int_shape(gt)}")
pred_f = K.flatten(pred)
gt_f = K.flatten(gt)
pred_f = K.print_tensor(pred_f, message='pred_f= ')
gt_f = K.print_tensor(gt_f, message='gt_f =')
print(f"pred_f_shape:{K.int_shape(pred_f)}")
print(f"gt_f_shape:{K.int_shape(gt_f)}")
conf_mat = tf.math.confusion_matrix(y_true_f,y_pred_f, num_classes = 14)
"""
add codes to find F1 score for each class
"""
# return an arbitrary number, as F1 scores not found yet.
return 1
The output at when epoch 1 just started:
y_true_shape:(None, 256, 256, 14)
y_pred_shape:(None, 256, 256, 14)
pred_print:Tensor("ArgMax_1:0", shape=(), dtype=int64)
gt_print:Tensor("ArgMax:0", shape=(), dtype=int64)
pred_shape:()
gt_shape:()
pred_f_shape:(1,)
gt_f_shape:(1,)
Then for the rest of the steps and epochs were similar as below:
y_true = [[[[1 0 0 ... 0 0 0]
[1 0 0 ... 0 0 0]
[1 0 0 ... 0 0 0]
...
y_pred = [[[[0.0889623 0.0624801107 0.0729747042 ... 0.0816219151 0.0735477135 0.0698677748]
[0.0857798532 0.0721047595 0.0754121244 ... 0.0723947287 0.0728530064 0.0676521733]
[0.0825942457 0.0670698211 0.0879610255 ... 0.0721599609 0.0845924541 0.0638583601]
...
pred= 1283828
gt = 0
pred_f= [1283828]
gt_f = [0]
Why is pred a number instead of a list of numbers with each number represents index of class? Similarly, why is pred_f is a list with only one number instead of list of indices?
And for gt (and gt_f), why is the value 0? I expect them to be list of indices.
I looks like argmax() simply uses the flattened y.
You need to specify which axis you want argmax() to reduce. Probably it's the last one, in your case 3. Then you'll get pred with a shape (None, 256, 256) containing integer between 0 and 13.
Try something like this: pred = K.argmax(y_pred, axis=3)
This is the documentation for tensorflow argmax. (But I'm not sure if you're using exactly that, since I can not see what K is imported as)
I have a brute force optimization algorithm with the objective function of the form:
np.clip(x # M, a_min=0, a_max=1) # P
where x is a Boolean decision vector, M is a Boolean matrix/tensor and P is a probability vector. As you can guess, x # M as an inner product can have values higher than 1 where is not allowed as the obj value should be a probability scalar or vector (if M is a tensor) between 0 to 1. So, I have used numpy.clip to fix the x # M to 0 and 1 values. How can I set up a mechanism like clip in cvxpy to achieve the same result? I have spent ours on internet with no lock so I appreciate any hint. I have been trying to use this to replicate clip but it raises Exception: Cannot evaluate the truth value of a constraint or chain constraints, e.g., 1 >= x >= 0. As a side note, since cvxpy cannot handle tensors, I loop through tensor slices with M[s].
n = M.shape[0]
m = M.shape[1]
w = M.shape[2]
max_budget_of_decision_variable = 7
x = cp.Variable(n, boolean=True)
obj = 0
for s in range(m):
for w in range(w):
if (x # M[s])[w] >= 1:
(x # M[s])[w] = 1
obj += x # M[s] # P
objective = cp.Maximize(obj)
cst = []
cst += [cp.sum(y) <= max_budget_of_decision_variable ]
prob = cp.Problem(objective, constraints = cst)
As an example, consider M = np.array([ [1, 0, 0, 1, 1, 0], [0, 0, 1, 0, 1, 0], [1, 1, 1, 0, 1, 0]]) and P = np.array([0.05, 0.15, 0.1, 0.15, 0.5, 0.05]).
I have a convex optimization problem I am trying to solve with cvxpy. Given a 1 x n row vector y and an m x n matrix C, I want to find a scalar b and a 1 x m row vector a such that the sum of squares of y - (aC + b(aC # aC)) is as small as possible (the # denotes element wise multiplication). In addition, all entires in a must be nonnegative and sum to 1 and -100 <= b <= 100. Below is my attempt to solve this using cvxpy.
import numpy as np
import cvxpy as cvx
def find_a(y, C, b_min=-100, b_max=100):
b = cvx.Variable()
a = cvx.Variable( (1,C.shape[0]) )
aC = a * C # this should be matrix multiplication
x = (aC + cvx.multiply(b, cvx.square(aC)))
objective = cvx.Minimize ( cvx.sum_squares(y - x) )
constraints = [0. <= a,
a <= 1.,
b_min <= b,
b <= b_max,
cvx.sum(a) == 1.]
prob = cvx.Problem(objective, constraints)
result = prob.solve()
print a.value
print result
y = np.asarray([[0.10394265, 0.25867508, 0.31258457, 0.36452763, 0.36608997]])
C = np.asarray([
[0., 0.00169811, 0.01679245, 0.04075472, 0.03773585],
[0., 0.00892802, 0.03154158, 0.06091544, 0.07315024],
[0., 0.00962264, 0.03245283, 0.06245283, 0.07283019],
[0.04396226, 0.05245283, 0.12245283, 0.18358491, 0.23886792]])
find_a(y, C)
I keep getting a DCPError: Problem does not follow DCP rules. error when I try to solve for a. I am thinking that either my function is not really convex, or I do not understand how to construct the proper cvxpy Problem. Any help would be greatly appreciated.
I have a z3 Array:
x = Array('x', IntSort(), IntSort())
A fixed number n:
n = 10
And a filtering condition based on simple arithmetic:
lambda i: Or(i == 0, i > 2)
What I want is to know the total number of elements from index 0 to index n which satisfy this condition (something that, if this were a normal python list, would look like len(filter(lambda i: i == 0 or i > 2, x)).
I can't figure out how to do this in z3. I tried
y == Int('y')
solver.add(y == sum([1 if Or(x[i] == 0, x[i] > 2) else 0 for i in range(0,n)]))
but get the following error:
z3.z3types.Z3Exception: Symbolic expressions cannot be cast to concrete Boolean values.
Is there a way to proceed? Many thanks!
It'd be best to post the exact code you tried. For one thing, the if-then-else should be coded using the If construct and the sum is best expressed using Sum. The following works fine, for instance:
from z3 import *
x = Array ('x', IntSort(), IntSort())
solver = Solver()
n = 10
y = Int('y')
solver.add(y == Sum([If(Or(x[i] == 0, x[i] > 2), 1, 0) for i in range(0,n)]))
print solver.check()
print solver.model()
But this'll be satisfiable for all values of y since there is no constraint on the contents of the array x.
I created this plot through the GeoGebra spreadsheet:
I would be interested in getting the same plot, but only in range between the first (1;3) and the last (3;4.5) point. How can I do this?
Thanks in advance
Ok this is my firs answer , you can type that in the input line below : If(1 < x < 3, f(x))
where f(x) is your function
For example, for f(x) = x^2, you can input like this: x² / (x < -1 ∨ x > 1)
(x < -1 ∨ x > 1) is a boolean value, True == 1, and False == 0. 0 as a denominator is meaningless, and this can be used to limit the display range in Geogebra.