TensorFlow:Evaluate test set multiple times but get different accuracy - machine-learning

Question

I have trained the model of MNIST using CNN, but when I check the model's accuracy with test data after the training, I find that my accuracy will improve. Here is the code.
BATCH_SIZE = 50
LR = 0.001 # learning rate
mnist = input_data.read_data_sets('./mnist', one_hot=True) # they has been normalized to range (0,1)
test_x = mnist.test.images[:2000]
test_y = mnist.test.labels[:2000]
def new_cnn(imageinput, inputshape):
weights = tf.Variable(tf.truncated_normal(inputshape, stddev = 0.1),name = 'weights')
biases = tf.Variable(tf.constant(0.05, shape = [inputshape[3]]),name = 'biases')
layer = tf.nn.conv2d(imageinput, weights, strides = [1, 1, 1, 1], padding = 'SAME')
layer = tf.nn.relu(layer)
return weights, layer
tf_x = tf.placeholder(tf.float32, [None, 28 * 28])
image = tf.reshape(tf_x, [-1, 28, 28, 1]) # (batch, height, width, channel)
tf_y = tf.placeholder(tf.int32, [None, 10]) # input y
# CNN
weights1, layer1 = new_cnn(image, [5, 5, 1, 32])
pool1 = tf.layers.max_pooling2d(
layer1,
pool_size=2,
strides=2,
) # -> (14, 14, 32)
weight2, layer2 = new_cnn(pool1, [5, 5, 32, 64]) # -> (14, 14, 64)
pool2 = tf.layers.max_pooling2d(layer2, 2, 2) # -> (7, 7, 64)
flat = tf.reshape(pool2, [-1, 7 * 7 * 64]) # -> (7*7*64, )
hide = tf.layers.dense(flat, 1024, name = 'hide') # hidden layer
output = tf.layers.dense(hide, 10, name = 'output')
loss = tf.losses.softmax_cross_entropy(onehot_labels=tf_y, logits=output) # compute cost
accuracy = tf.metrics.accuracy( labels=tf.argmax(tf_y, axis=1), predictions=tf.argmax(output, axis=1),)[1]
train_op = tf.train.AdamOptimizer(LR).minimize(loss)
sess = tf.Session()
init_op = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer()) # the local var is for accuracy
sess.run(init_op) # initialize var in graph
saver = tf.train.Saver()
for step in range(101):
b_x, b_y = mnist.train.next_batch(BATCH_SIZE)
_, loss_ = sess.run([train_op, loss], {tf_x: b_x, tf_y: b_y})
if step % 50 == 0:
print(loss_)
accuracy_, loss2 = sess.run([accuracy, loss], {tf_x: test_x, tf_y: test_y })
print('Step:', step, '| test accuracy: %f' % accuracy_)
To simplify the problem, I only use the 100 training iterations. And the final accuracy of test set is approximately 0.655000.
But when I run the following code:
for i in range(5):
accuracy2 = sess.run(accuracy, {tf_x: test_x, tf_y: test_y })
print(sess.run(weight2[1,:,0,0])) # To show that the model parameters won't update
print(accuracy2)
The output is
[-0.06928255 -0.13498515 0.01266837 0.05656774 0.09438231]
0.725875
[-0.06928255 -0.13498515 0.01266837 0.05656774 0.09438231]
0.7684
[-0.06928255 -0.13498515 0.01266837 0.05656774 0.09438231]
0.79675
[-0.06928255 -0.13498515 0.01266837 0.05656774 0.09438231]
0.817
[-0.06928255 -0.13498515 0.01266837 0.05656774 0.09438231]
0.832187
This makes me confused, can somebody tell me what's wrong?
Thanks for your patience!

Answer 1

tf.metrics.accuracy is not as trivial as you might think. Take a look at its documentation:
The accuracy function creates two local variables, total and
count that are used to compute the frequency with which
predictions matches labels. This frequency is ultimately
returned as accuracy: an idempotent operation that simply divides
total by count.
Internally, an is_correct operation computes a Tensor with
elements 1.0 where the corresponding elements of predictions and
labels match and 0.0 otherwise. Then update_op increments
total with the reduced sum of the product of weights and
is_correct, and it increments count with the reduced sum of
weights.
For estimation of the metric over a stream of data, the function
creates an update_op operation that updates these variables and
returns the accuracy.
...
Returns:
accuracy: A Tensor representing the accuracy, the value of total divided
by count.
update_op: An operation that increments the total and count variables
appropriately and whose value matches accuracy.
Note that it returns a tuple and you take the second item, namely update_op. Consecutive invocation of update_op is treated as streaming of data, which is not what you intend to do (because each evaluation during training will affect future evaluations). In fact, this running metric is pretty counter-intuitive.
The solution for you is to use plain simple accuracy calculation. Change this line to:
accuracy = tf.reduce_mean(tf.cast(tf.equal(tf.argmax(tf_y, axis=1), tf.argmax(output, axis=1)), tf.float32))
and you'll have a stable accuracy calculation.