Keras wrong accuracy using model.predict - machine-learning

Question

I am using this code for a CNN
train_batches = ImageDataGenerator().flow_from_directory('dice_sklearn/train', target_size=(IMG_WIDTH, IMG_HEIGHT),
classes=['1', '2', '3', '4', '5', '6'],
batch_size=cv_opt['batch'],
color_mode="grayscale")
test_batches = ImageDataGenerator().flow_from_directory('dice_sklearn/test', target_size=(IMG_WIDTH, IMG_HEIGHT),
class_mode='categorical',
batch_size=cv_opt['batch'],
shuffle=False)
train_num = len(train_batches)
test_num = len(test_batches)
model = Sequential([
Conv2D(32, (3, 3), padding='same', activation='relu', input_shape=(IMG_WIDTH, IMG_HEIGHT, 1)),
Conv2D(32, (3, 3), activation='relu'),
MaxPooling2D(pool_size=(2, 2)),
Dropout(0.30),
Conv2D(64, (3, 3), padding='same', activation='relu'),
Conv2D(64, (3, 3), activation='relu'),
MaxPooling2D(pool_size=(2, 2)),
Dropout(0.30),
Conv2D(64, (3, 3), padding='same', activation='relu'),
Conv2D(64, (3, 3), activation='relu'),
MaxPooling2D(pool_size=(2, 2)),
Dropout(0.30),
Flatten(),
Dense(512, activation='relu'),
Dropout(0.5),
Dense(6, activation='softmax'),
])
print(model.summary())
model.compile(Adam(lr=cv_opt['lr']), loss='categorical_crossentropy', metrics=['accuracy'])
history = model.fit(train_batches, steps_per_epoch=train_num,
epochs=cv_opt['epoch'], verbose=2)
model.save('cnn-keras.h5')
test_batches.reset()
prediction = model.predict(test_batches, steps=test_num, verbose=1)
predicted_class = np.argmax(prediction, axis=1)
classes = test_batches.classes[test_batches.index_array]
accuracy = (predicted_class == classes).mean()
print("Final accuracy:", accuracy * 100)
Where
cv_opt['batch'] is set to 50
cv_opt['lr'] is set to 0.0003
cv_opt['epoch'] is set to 50
The output from the training phase (with model.fit) on the last line (last epoch) returns:
192/192 [==============================] - 98s 510ms/step - loss: 0.0514 - accuracy: 0.9818 - val_loss: 0.0369 - val_accuracy: 0.9833
But when I run this part of code:
test_batches.reset()
prediction = model.predict(test_batches, steps=test_num, verbose=1)
predicted_class = np.argmax(prediction, axis=1)
classes = test_batches.classes[test_batches.index_array]
accuracy = (predicted_class == classes).mean()
print("Final accuracy:", accuracy * 100)
I get an accuracy score very very low: (0.16).
But if a plot the learning curves I can see that the test/validation curve (if in testing or in parameter tuning) both reach accuracies near 90%.
Am I using the model.predict in the wrong way?

Answer 1

Your model is not overfitting. Steps 1 and 2 do not have to be implemented at all in order to solve your problem. In fact, it is even more wrong since the author states that in case of overfitting you need to add more layers, which is strongly advised against: when one has an overfitting model, the model needs to be made simpler, not more complex.
The solution to your issue lies in #Dr.Snoopy's answer : the order of the classes do not match.
My recommendation is to iterate manually through the entire test set, get the ground truth, get the prediction (ensure the same exact preprocessing on images like in the training set is applied on your test set images) before you feed them to your model.
Then, calculate your metrics. This will solve your problem.
For example, you could use the idea below:
correctly_predicted = 0
for image in os.scandir(path_to_my_test_directory):
image_path = image.path
image = cv2.imread(image_path)
image = apply_the_same_preprocessing_like_in_training(image)
#transform from (H,W,3) to (1,H,W,3) because TF + Keras predict only on batches
image = np.expand_dims(image,axis=0)
prediction_label = np.argmax(model.predict(image))
if prediction_label == ground_truth_label:
correctly_predicted+=1