Why does fully convolutional network plateau first and then learns? - machine-learning

Question

Why does fully convolutional network plateau first and then learns?
Im training a fully convolutional network to classify handwriting Chinese characters. The dev dataset I am using has 250 classes with 200 - 300 samples in each class.
And I found out no matter how I tweak the model, somehow the ones I've tried so far all has a similar behaviour, which they all plateau at first and then the accuracies start to shoot up while the losses decrease, as shown in the screenshot below:
I would love to know more about the reasons behind this behaviour.
Thanks a lot!
Edit:
Sorry for didn't provide more details before.
My best performing network so far is as below, using an Adadelta optimizer with LR at 0.1. My weights were initialised using xavier initialisation.
Input(shape=(30, 30, 1))
Lambda(
lambda image: tf.image.resize_images(
image, size=(resize_size, resize_size),
method=tf.image.ResizeMethod.BILINEAR
)
)
Conv2D(filters=96, kernel_size=(5, 5), strides=(1, 1), padding="same", "relu")
Conv2D(filters=96, kernel_size=(1, 1), strides=(1, 1), padding="same", "relu")
MaxPooling2D(pool_size=(3, 3), strides=2)
Conv2D(filters=192, kernel_size=(5, 5), strides=(2, 2), padding="same", "relu")
Conv2D(filters=192, kernel_size=(1, 1), strides=(1, 1), padding="same", "relu")
MaxPooling2D(pool_size=(3, 3), strides=2)
Conv2D(filters=192, kernel_size=(3, 3), "same", "relu")
Conv2D(filters=192, kernel_size=(1, 1), "same", "relu")
Conv2D(filters=10, kernel_size=(1, 1), "same", "relu")
AveragePooling2D(pool_size=(3, 3))
Flatten()
Dense(250, activation="softmax")
model = Model(inp, x)
model.compile(
loss=categorical_crossentropy,
optimizer=Adadelta(lr=0.1),
metrics=["accuracy"],
)
As to the input data, they are all handwriting chinese characters that had been transformed into a MNIST format by me, with size of 30x30x1 (the existence of the Lambda layer after the Input layer was because I was following the original FCN paper paper and they used 32x32 input size), which is as below:
And this is how the loss and accuracy charts above came about.
Hope this provides better intuition. Thanks.

Answer 1

We can't answer specifically, because you've neglected to identify your network and inputs sufficiently, let alone the training methods. To fully trace the high-level training characteristics, we'd need some detailed visualization of the kernels through the iterations in question.
In general, this is simply because a highly complex model usually needs a few iterations before it gets better than random results. We begin with random weights and kernels. In the first few iterations, the model has to work through the chaos, establish a few useful patterns in the early-level kernels, and find weights that correlate with enough output categories that the accuracy moves above 0.4% with statistical significance.
Part of the problem is that, in those first few iterations, the model also stumbles across patterns that are useful in chaos, but actually harm long-term learning. For instance, it may build a pattern for black dots, and guess right that this correlates to mammal eyes and vehicle wheels. All too soon, that generalization, that an airplane and an Airedale are structurally related, turns out to be a wrong assumption. It has to break down the second-level correlations between those categories and find something else.
This is the sort of learning that keeps the accuracy low for longer than you might think. The model spends the first few iterations jumping to hundreds of conclusions about classifications, anything that correlates with one or two correct guesses. Then it has to learn enough to separate valid ones from invalid ones. That is where the model starts making advances it can retain.