Is it normal for my test loss to reach millions - image-processing

Question

I'm training a model over several iterations (training, saving, and training again) on the second iteration my val_loss reached millions for some reason. is something wrong with how i'm importing the model?
This is how i saved my initial model after my first run
model.save('/content/drive/My Drive/Colab Notebooks/path/to/save/locaiton',save_format='tf')
and this is how i'm importing and overwriting it
def retrainmodel(model_path,tr_path,v_path):
image_size = 224
BATCH_SIZE_TRAINING = 10
BATCH_SIZE_VALIDATION = 10
BATCH_SIZE_TESTING = 1
EARLY_STOP_PATIENCE = 6
STEPS_PER_EPOCH_TRAINING = 10
STEPS_PER_EPOCH_VALIDATION = 10
NUM_EPOCHS = 20
model = tf.keras.models.load_model(model_path)
data_generator = ImageDataGenerator(preprocessing_function=preprocess_input)
train_generator = data_generator.flow_from_directory(tr_path,
target_size=(image_size, image_size),
batch_size=BATCH_SIZE_TRAINING,
class_mode='categorical')
validation_generator = data_generator.flow_from_directory(v_path,
target_size=(image_size, image_size),
batch_size=BATCH_SIZE_VALIDATION,
class_mode='categorical')
cb_early_stopper = EarlyStopping(monitor = 'val_loss', patience = EARLY_STOP_PATIENCE)
cb_checkpointer = ModelCheckpoint(filepath = 'path/to/checkpoint/folder', monitor = 'val_loss', save_best_only = True, mode = 'auto')
fit_history = model.fit(
train_generator,
steps_per_epoch=STEPS_PER_EPOCH_TRAINING,
epochs = NUM_EPOCHS,
validation_data=validation_generator,
validation_steps=STEPS_PER_EPOCH_VALIDATION,
callbacks=[cb_checkpointer, cb_early_stopper]
)
model.save('/content/drive/My Drive/Colab Notebooks/path/to/save/locaiton',save_format='tf')
this is my output after passing my directories onto this function
Found 1421 images belonging to 5 classes.
Found 305 images belonging to 5 classes.
Epoch 1/20
10/10 [==============================] - 233s 23s/step - loss: 2.3330 - acc: 0.7200 - val_loss: 4.6237 - val_acc: 0.4400
Epoch 2/20
10/10 [==============================] - 171s 17s/step - loss: 2.7988 - acc: 0.5900 - val_loss: 56996.6289 - val_acc: 0.6800
Epoch 3/20
10/10 [==============================] - 159s 16s/step - loss: 1.2776 - acc: 0.6800 - val_loss: 8396707.0000 - val_acc: 0.6500
Epoch 4/20
10/10 [==============================] - 144s 14s/step - loss: 1.4562 - acc: 0.6600 - val_loss: 2099639.7500 - val_acc: 0.7200
Epoch 5/20
10/10 [==============================] - 126s 13s/step - loss: 1.0970 - acc: 0.7033 - val_loss: 50811.5781 - val_acc: 0.7300
Epoch 6/20
10/10 [==============================] - 127s 13s/step - loss: 0.7326 - acc: 0.8000 - val_loss: 84781.5703 - val_acc: 0.7000
Epoch 7/20
10/10 [==============================] - 110s 11s/step - loss: 1.2356 - acc: 0.7100 - val_loss: 1000.2982 - val_acc: 0.7300
here is my optimizer:
sgd = optimizers.SGD(lr = 0.01, decay = 1e-6, momentum = 0.9, nesterov = True)
model.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = 'acc')
where do you think i'm wrong?
I am training my model in batches because i'm working on google colab with 22K images in total, so these results are after feeding the network 2800 training images. do you think it'll sort itself out if i feed it more images, or is something seriously wrong?

Answer 1

I think it's not good to have this loss. It's logical to have a higher loss during the initial few epochs as we load a model and retrain it. However, this loss value should not shoot to the stars, as in your case. If at the time of saving, the loss value was roughly 0.5, then when you load the same model for retraining, it should not be higher than 10x previous value, so, a value of 5 +- 1 would be expected. [NOTE: this is purely based on experience. There is no general method to know the loss beforehand.]
If your loss is too high, the following would be reasonable:
Varying dataset - Changing the dynamics of the training dataset could force the model for this behavior.
Model save might have altered weights
Solutions Suggested:
Try using save_weights instead of save method on model
model.save_weights('path/to/filename.h5')
also, use load_weights instead of load_model
model = call_cnn_function_to_build_model()
model.compile(... your args ...)
model = model.load_weights('path/to/filename.h5')
Since you have checkpoints, try using checkpoint saved models. So rather than the final model, try to load a model from checkpoint which is near to your last epochs.
PS: Corrections gratefully accepted.