I would like to binary quantize vector in order to use it in loss function.
However I am not sure this is the right way.
def _binary_quantize1(h):
with tf.name_scope('binary_quantization'):
return tf.cast(h > tf.constant(0.5, shape=tf.shape(h)), tf.float32)
def _binary_quantize2(h):
with tf.name_scope('binary_quantization'):
return tf.sign(tf.round(h), name='quantized')
How would you approach such functionality? Will the gradient pass ok through such function?
EDIT:
I want to do something like this function in tensorflow https://github.com/kevinlin311tw/cvpr16-deepbit/blob/master/src/caffe/layers/K2_min_quantize_loss_layer.cpp
Related
I am trying to normalize my data(with shape (23687,7)), then I save the mean and std of the original dataset to "normalized_param.pkl"
After fitting the normalized data to my LSTM model, I will get an answer array (with shape (23687, 1))
Now what I gonna do is：
test_sc_path = os.path.join('normalized_standard', 'normalized_param.pkl')
test_scaler = load(test_sc_path)
test_denorm_value = test_scaler.inverse_transform(test_normalized_data)
ValueError: non-broadcastable output operand with shape (23687,1) doesn't match the broadcast shape (23687,7)
I think that's because the test_scaler object have 7 dim params inside, so if I want to de-normalize only 1 dim data, I should use
test_scaler.mean_[-1]and「test_scaler.scale_[-1]to get the last param I want to compute.
However, I think it's quite complicated, is there any sklearn method just like scaler.inverse_transform() I can easily use to solve this problem？
thanks
Yes, there is a method for it. See the documentation here.
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(data) # Basically fits the data, store means & standard deviations.
scaler.transform(data) # Standardize (Normalize) the data with the scaler parameters
scaler.fit_transform(data) # Fits & Transform
scaler.inverse_transform(data) # Apply inverse transformation for the input data.
I am new to keras and trying to modify the outputs of a layer during training. I want to write a function that takes the layer outputs and return the modeified outputs to the next layer during learning. I have tried using lambda functions but not really got hold of it.
def fun(x):
a = min(x)
y = np.round(x*(2**a))
return y
layer_1 = Dense(32, activation='relu')(input)
layer_2 = Dense(12, activation='relu')(layer_1)
lambda_layer = Lambda(fun, output_shape=(12,))(layer_2)
layer_3 = dense(32, activation='relu')(lambda_layer)
how can I get the layer outputs and modify them before passing it to next layer?
Using a lambda function is the right approach for your problem. However, keep in mind that the lambda function will be part of your computational graph and during training gradients have to be computed for the whole graph.
For example, you should not use the min() function as you did but rather use functions which are part of Keras Backend. Replacing all operations by their keras backend equivalent results in:
import keras.backend as K
def fun(x):
a = K.min(x)
y = K.round(K.dot(x, (K.pow(2, a))))
return y
Your final model (and so all Lambda layers) should only contain native Keras functions, in order to safely perform all calculations during training.
This fails because you are using non-native operations (like np.round) inside a Lambda function, which expects keras operations
Examine the keras.backend docs, and take the functions you want to use from there.
So your function should look something like this
from keras import backend as K
def fun(x):
a = K.min(x, axis=-1) # Specify the axis you need!
y = K.round(x*(2**a))
return y
I have a custom activation function and its derivative, although I can use the custom activation function I don't know how to tell keras what is its derivative.
It seems like it finds one itself but I have a parameter that has to be shared between the function and its derivative so how can I do that?
I know there is a relatively easy way to do this in tensorflow but I have no idea how to implement it in keras here is how you do it in tensorflow
Edit: based on the answer I got maybe I wasn't clear enough. What I want is to implement a custom derivative for my activation function so that it use my derivative during the backpropagation. I know how to implement a custom activation function.
Take a look at the source code where the activation functions of Keras are defined:
keras/activations.py
For example:
def relu(x, alpha=0., max_value=None):
"""Rectified Linear Unit.
# Arguments
x: Input tensor.
alpha: Slope of the negative part. Defaults to zero.
max_value: Maximum value for the output.
# Returns
The (leaky) rectified linear unit activation: `x` if `x > 0`,
`alpha * x` if `x < 0`. If `max_value` is defined, the result
is truncated to this value.
"""
return K.relu(x, alpha=alpha, max_value=max_value)
And also how does Keras layers call the activation functions: self.activation = activations.get(activation) the activation can be string or callable.
Thus, similarly, you can define your own activation function, for example:
def my_activ(x, p1, p2):
...
return ...
Suppose you want use this activation in Dense layer, you just put your function like this:
x = Dense(128, activation=my_activ(p1, p2))(input)
If you mean you want to implement your own derivative:
If your activation function is written in Tensorflow/Keras functions of which the operations are differentiable (e.g. K.dot(), tf.matmul(), tf.concat() etc.), then the derivatives will be obtained by automatic differentiation https://en.wikipedia.org/wiki/Automatic_differentiation. In that case you dont need to write your own derivative.
If you still want to re-write the derivatives, check this document https://www.tensorflow.org/extend/adding_an_op where you need to register your gradients using tf.RegisterGradient
what google told me is:
For keras, the ImageDataGenerator function seems to have a zca_whitening which can be used out of the box. But if this option been set, it requires to call the ImageDataGenerator.fit on the whole dataset X. So this is not an option.
For sklearn, the IncrementalPCA seems to work with a huge dataset, but I don't know how to rotate PCA to ZCA in an generator style.
Thanks for the help!
I have defined a function that might be helpful following the ZCA transformation:
def ZCAtransform(X,IPCA_model):
# get the Eigenvectors and Eigenvalues
U = IPCA_model.components_.transpose()
S = np.sqrt(IPCA_model.explained_variance_)
Xdemeand = (X-np.mean(X,0)).transpose()
#get the transformed data
# Xproj' = U * diag(1/(S+I*epsilon)) * U' * X_data
return (U.dot(np.diag(1/(S+IPCA_model.noise_variance_))).dot(U.transpose()).dot(Xdemeand)).transpose()
Xproj = ZCAtransform(X, ipca)
Following the given example at Scikit-learn, I was able to generate the ZCA of Iris dataset as shown below:
ZCA Whitened PCA
The Keras ImageDataGenerator class provides the two flow methods flow(X, y) and flow_from_directory(directory) (https://keras.io/preprocessing/image/).
Why is the parameter
target_size: tuple of integers, default: (256, 256). The dimensions to which all images found will be resized
Only provided by flow_from_directory(directory) ? And what is the most concise way to add reshaping of images to the preprocessing pipeline using flow(X, y) ?
flow_from_directory(directory) generates augmented images from directory with arbitrary collection of images. So there is need of parameter target_size to make all images of same shape.
While flow(X, y) augments images which are already stored in a sequence in X which is nothing but numpy matrix and can be easily preprocessed/resized before passing to flow. So no need for target_size parameter. As for resizing I prefer using scipy.misc.imresize over PIL.Image resize, or cv2.resize as it can operate on numpy image data.
import scipy
new_shape = (28,28,3)
X_train_new = np.empty(shape=(X_train.shape[0],)+new_shape)
for idx in xrange(X_train.shape[0]):
X_train_new[idx] = scipy.misc.imresize(X_train[idx], new_shape)
For large training dataset, performing transformations such as resizing on the entire training data is very memory consuming. As Keras did in ImageDataGenerator, it's better to do it batch by batch. As far as I know, there're 2 ways to achieve this other than operating the whole dataset:
You can use Lambda Layer to create a layer and then feed original training data to it. The output is the resized you need.
Here is the sample code if you use TensorFlow as the backend of Keras:
original_dim = (32, 32, 3)
target_size = (64, 64)
input = keras.layers.Input(original_dim)
x = tf.keras.layers.Lambda(lambda image: tf.image.resize(image, target_size))(input)
As #Retardust mentioned, maybe you can customize your own ImageDataGenerator as well as the preprocessing_function.
For anyone else who wants to do this, .flow method of ImageDataGenerator does not have a target_shape parameter and we cannot resize an image using preprocessing_function parameter as the documentation states The function will run after the image is resized and augmented. The function should take one argument: one image (Numpy tensor with rank 3), and should output a Numpy tensor with the same shape.
So in order to use .flow, you will have to pass resized images only otherwise use a custom generator that resizes them on the fly.
Here's a sample of custom generator in keras (can also be made using python generator or any other method)
class Custom_Generator(keras.utils.Sequence) :
def __init__(self,...,datapath, batch_size, ..) :
def __len__(self) :
#calculate data len, something like len(train_labels)
def load_and_preprocess_function(self, label_names, ...):
#do something...
#load data for the batch using label names with whatever library
def __getitem__(self, idx) :
batch_y = train_labels[idx:idx+batch_size]
batch_x = self.load_and_preprocess_function()
return ( batch_x, batch_y )
X_data_resized = numpy.asarray([skimage.transform.resize(image, new_shape) for image in X_data])
because of the above code is now depreciated...
There is also (newer) method flow_from_dataframe() which accepts a Pandas dataframe with file paths and y data as columns - and it also allows to specify the target size. Just in case your image data is not organized directory-wise!