Does model evaluation during training affect final accuracy in PyTorch? - machine-learning

Question

During a simple training loop for PyTorch a strange effect was observed.
If the evaluation function is called or not seems to have effects on the final performance of the model.
We train on the CIFAR10 using a very simple MLP model and Adam with 10 training epochs.
We try two Main loops:
After the end of each training epoch we measure the accuracy of validation set
We calculate the validation only once at the end of all training.
We show the difference in code here below:
Main Loop 1:
# Main Loop 1
num_epochs = 10
print(f"num_epochs: {num_epochs}")
for epoch in range(num_epochs): # loop over the dataset multiple times
print(f"\nStart Epoch {epoch}")
model.train()
train_loss, train_accuracy = training_epoch(trainloader,optimizer,model,criterion)
print(f"Training Loss: {train_loss:.3f} - Training Accuracy: {train_accuracy:.3f}")
model.eval()
with torch.no_grad():
val_loss, val_accuracy = val_epoch(testloader, model, criterion)
print(f"Val Loss: {val_loss:.3f} - Val Accuracy: {val_accuracy:.3f}")
print('Finished Training')
Main Loop 2:
# Main Loop 2
num_epochs = 10
print(f"num_epochs: {num_epochs}")
for epoch in range(num_epochs): # loop over the dataset multiple times
print(f"\nStart Epoch {epoch}")
model.train()
train_loss, train_accuracy = training_epoch(trainloader,optimizer,model,criterion)
print(f"Training Loss: {train_loss:.3f} - Training Accuracy: {train_accuracy:.3f}")
model.eval()
with torch.no_grad():
val_loss, val_accuracy = val_epoch(testloader, model, criterion)
print(f"Val Loss: {val_loss:.3f} - Val Accuracy: {val_accuracy:.3f}")
print('Finished Training')
Though there shouldn't be any change, the final performance of model change.
Val Loss: 1.526 - Val Accuracy: 0.523
Val Loss: 1.501 - Val Accuracy: 0.528
Of course for reproducibility, we set all seeds. Moreover, this effect can already be observed at the beginning of the second training epoch.
I share the entire code as a Colab notebook:
https://colab.research.google.com/drive/1BODeKHZmcT8lH3r2bxYVHNR2KOpT9O9Y?usp=sharing

Answer 1

The observed difference would be due to variance because of stochasticity in the optimization algorithm. The evaluation you perform has no effect on the model's weights.
Also in the link you provided, you are re-initializing a SimpleMLP on both experiments. Since the module's weights get instantiated randomly the inference will naturally yield different results.