if my code as follow:
main_input = Input(shape=(16), name='main_input')
act1= Dense(10, activation='tanh')(main_input )
now I have as symbol act1, the shape is (batch_size,10)
if i wanna matrix act1 to add 0.5 each point, or if i wanna to get the data,only include act1[:,0:5]
i am confused. Because in MXnet, there are some operation on Symbol variables but in keras, it looks like all in layer operation. How could I operate on a variable ....
Hope guys can help me . Thx
It seems that you want to use a so called Lambda function. Here e.g. the first case looks like this:
act1_new = Lambda(lambda x: x + 0.5, output_shape=(10,))(act_1)
and second one:
act1_new = Lambda(lambda x: x[:,:5], output_shape=(5,))(act1)
In this example a first dimension is skiped as it represents a batch dimension.
Related
I have the following data:
ar <- arima.sim(list(order=c(1,0,0), ar=0.9), n=M1) + 10
How to fit an AR(1) model to simulated data above with ar parameter=0.5?
EDIT:
I used:
fit <- arima(ar, fixed = 0.5, include.mean = T)
fit
Call:
arima(x = ar, include.mean = T, fixed = 0.5))
Coefficients:
intercept
0.5
This is not correct. I want my fitted model to have the mean (which should be approximately 10) and the ar_parameter=0.5.
Pls help
Both stats::arima and forecast::Arima accept a parameter fixed which will fix certain parameters to a given value, and fit the remaining parameters. You need to pass the value NA for any parameters that you don't want to fix.
Therefore, you can do this:
mod <- stats::arima(ar, c(1,0,0), include.mean = TRUE, fixed = c(0.5, NA))
This will fit an AR(1) where the the AR coefficient is fixed at 0.5 and the constant is fitted freely. See the documentation for the convention on the ordering of parameters in fixed.
I am implementing a custom connection between two different keras layers. The neural network begins something like below:
model = tf.keras.Sequential()
c1 = model.add(Conv2D(6, kernel_size=[5,5], strides=(stride,stride), padding="valid", input_shape=(32,32,1),
activation = 'tanh'))
s2 = model.add(AveragePooling2D(pool_size=2, strides=2, padding='valid'))
Now, the output of s2 has a size of 14*14*6
Here, I want to apply my custom connection to convolution layer c3 which has an output size of 10*10*16 (that is, 16 filters need to be applied on s2 of size 14*14*6 and get an output of 10*10*16). For this, I need to use kernal_size = 5*5, filers=16, stride = 1, and padding=valid.
However, all the 6 feature maps (of s2) are not connected to 16 feature maps of (c3). The connections are explained as given here.
For example (the explanation of given link above), to build your first feature map of C3, you convolve 3 of your input maps (of s2 of size 14*14*6) with 5x5 filters, which gives you 3 10x10 maps that are summed up to give your first feature map, which is then of size 10x10.
I read somewhere that, we need to use Functional API to build this.
But, I am not sure, how to proceed further. Can someone help on implementing this.
My initial approach of implementing this is as follows:
from keras.models import Model
from keras.layers import Conv2D, Input, Concatenate, Lambda, Add
inputTensor = Input(shape=(14, 14,6))
stride =1
group0_a = Lambda(lambda x: x[:,:,0])(inputTensor)
group0_b = Lambda(lambda x: x[:,:,1])(inputTensor)
group0_c = Lambda(lambda x: x[:,:,2])(inputTensor) # Take 0,1,2 feature map of s2
conv_group0_a = Conv2D(1, kernel_size=[5,5], strides=(stride,stride), padding="valid", activation = 'tanh')(group0_a)
conv_group0_b = Conv2D(1, kernel_size=[5,5], strides=(stride,stride), padding="valid", activation = 'tanh')(group0_b)
conv_group0_c = Conv2D(1, kernel_size=[5,5], strides=(stride,stride), padding="valid", activation = 'tanh')(group0_c) #Applying convolution on each of 0, 1, 2 feature maps of s2 with distinct kernals
added_0 = Add()([conv_group0_a, conv_group0_b, conv_group0_c]) #adding all the three to get one of the 10*10*16
#Repeat this for 16 neurons of c3 and then finally
output_layer = Concatenate()([]) #concatenate them
Mymodel = Model(inputTensor,output_layer)
I want to know, if my approach is correct (I know it is not because I am getting too many errors). So, I need help in recreating the custom connection as explained above. Any help is appreciated.
the above code is correct, the only change I made is group0_a = Lambda(lambda x: x[:,:,0:1])(inputTensor), that is instead of passing x as x[:,:,0] I passed it as x[:,:,0:1]
I am trying to create a list based on my neural network outputs and use it in Tensorflow as a loss function.
Assume that results is list of size [1, batch_size] that is output by a neural network. I check to see whether the first value of this list is in a specific range passed in as a placeholder called valid_range, and if it is add 1 to a list. If it is not, add -1. The goal is to make all predictions of the network in the range, so the correct predictions is a tensor of all 1, which I call correct_predictions.
values_list = []
for j in range(batch_size):
a = results[0, j] >= valid_range[0]
b = result[0, j] <= valid_range[1]
c = tf.logical_and(a, b)
if (c == 1):
values_list.append(1)
else:
values_list.append(-1.)
values_list_tensor = tf.convert_to_tensor(values_list)
correct_predictions = tf.ones([batch_size, ], tf.float32)
Now, I want to use this as a loss function in my network, so that I can force all the predictions to be in the specified range. I try to train like this:
loss = tf.reduce_mean(tf.squared_difference(values_list_tensor, correct_predictions))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
gradients, variables = zip(*optimizer.compute_gradients(loss))
gradients, _ = tf.clip_by_global_norm(gradients, gradient_clip_threshold)
optimize = optimizer.apply_gradients(zip(gradients, variables))
This, however, has a problem and throws an error on the last optimize line, saying:
ValueError: No gradients provided for any variable: ['<tensorflow.python.training.optimizer._RefVariableProcessor object at 0x7f0245d4afd0>',
'<tensorflow.python.training.optimizer._RefVariableProcessor object at 0x7f0245d66050>'
...
I tried to debug this in Tensorboard, and I notice that the list I am creating does not appear in the graph, so basically the x part of the loss function is not part of the network itself. Is there some way to accurately create a list based on the predictions of a neural network and use it in the loss function in Tensorflow to train the network?
Please help, I have been stuck on this for a few days now.
Edit:
Following what was suggested in the comments, I decided to use a l2 loss function, multiplying it by the binary vector I had from before values_list_tensor. The binary vector now has values 1 and 0 instead of 1 and -1. This way when the prediction is in the range the loss is 0, else it is the normal l2 loss. As I am unable to see the values of the tensors, I am not sure if this is correct. However, I can view the final loss and it is always 0, so something is wrong here. I am unsure if the multiplication is being done correctly and if values_list_tensor is calculated accurately? Can someone help and tell me what could be wrong?
loss = tf.reduce_mean(tf.nn.l2_loss(tf.matmul(tf.transpose(tf.expand_dims(values_list_tensor, 1)), tf.expand_dims(result[0, :], 1))))
Thanks
To answer the question in the comment. One way to write a piece-wise function is using tf.cond. For example, here is a function that returns 0 in [-1, 1] and x everywhere else:
sess = tf.InteractiveSession()
x = tf.placeholder(tf.float32)
y = tf.cond(tf.logical_or(tf.greater(x, 1.0), tf.less(x, -1.0)), lambda : x, lambda : 0.0)
y.eval({x: 1.5}) # prints 1.5
y.eval({x: 0.5}) # prints 0.0
I need to be able to plot e.g. the cost function values as a function of some parameter (for example the bias b below). If e.g. my graph is something like (pseudocode)
y = g(W x + b),
cost = sum(y ** 2),
where W and b are tf.Variables, I'd like to change b from, say 0 to 1 and plot the values of cost.
Please note that I do not want to call eval or sesssion.run after each change of b because of the overhead! E.g. for 100 plot points that would take forever.
I know of the existence of tf.assign, but doing something like [assign, cost, assign, cost, ...] and evaluating that doesn't seem to work
I guess I could update the value of b inside the graph and call cost after each update, but I wouldn't really want to change the graph
So how could I do this in an efficient manner? Thank you in advance!
EDIT: actually this is probably impossible to do without calling eval/run between the iterations... oh well...
In tensor-flow if you use variables you can only evaluate them only after an initialization. So you cannot probably evaluate them without a session.
but you can change the parameters like the following way
import tensorflow as tf
my_var = tf.Variable(10)
with tf.Session() as sess:
sess.run(my_var.initializer)
print(sess.run(my_var.assign_sub(2))) #>> 8
print(sess.run(my_var.assign_sub(2))) #>> 6
This sounds like a use case for feeding a different value at each step. Assuming b is a scalar variable, you could code your loop with something like the following:
import numpy as np
sess = tf.Session()
# Vary `b_val` from 0 to 1 in 100 steps.
for b_val in np.linspace(0, 1, 100):
# Evaluate `cost` using `b = b_val`.
cost_val = sess.run(cost, feed_dict={b: b_val})
# Do something with `cost_val`....
Given I have a linear model as the following I would like to get the gradient vector with regards to W and b.
# tf Graph Input
X = tf.placeholder("float")
Y = tf.placeholder("float")
# Set model weights
W = tf.Variable(rng.randn(), name="weight")
b = tf.Variable(rng.randn(), name="bias")
# Construct a linear model
pred = tf.add(tf.mul(X, W), b)
# Mean squared error
cost = tf.reduce_sum(tf.pow(pred-Y, 2))/(2*n_samples)
However if I try something like this where cost is a function of cost(x,y,w,b) and I only want to gradients with respect to w and b:
grads = tf.gradients(cost, tf.all_variable())
My placeholders will also be included (X and Y).
Even if I do get a gradient with [x,y,w,b] how do I know which element in the gradient that belong to each parameter since it is just a list without names to which parameter the derivative has be taken with regards to?
In this question I'm using parts of this code and I build on this question.
Quoting the docs for tf.gradients
Constructs symbolic partial derivatives of sum of ys w.r.t. x in xs.
So, this should work:
dc_dw, dc_db = tf.gradients(cost, [W, b])
Here, tf.gradients() returns the gradient of cost wrt each tensor in the second argument as a list in the same order.
Read tf.gradients for more information.