How do you build a CNN autoencoder? - machine-learning

Question

I have many images of documents and I would like to cluster them to create categories (invoices, receipts, etc.). I would like to explore the image approach (I know I can use text), so I decided to build a CNN auto-encoder to compress the dimensions to a lower space then run a clustering algorithm like DBSCAN.
My issue is that I have no idea how to select the network layers and the different activation functions etc. This is my current model, what do you think ?
model = Sequential()
model.add(Conv2D(16, (3, 3), strides=2, padding='same', kernel_regularizer = l2(), input_shape=image_rgb_dims_top))
model.add(LeakyReLU(alpha=0.2))
#model.add(AveragePooling2D(pool_size=(2,2), padding='same'))
model.add(Conv2D(32, (3, 3), strides=2, padding='same', kernel_regularizer = l2()))
model.add(LeakyReLU(alpha=0.2))
#model.add(AveragePooling2D(pool_size=(2,2), padding='same'))
model.add(Flatten())
model.add(Dense(96, activity_regularizer=l1(10e-6)))
model.add(Dense(np.prod(model.layers[-2].output_shape[1:]),activation='relu'))
model.add(Reshape(model.layers[-4].output_shape[1:]))
model.add(Conv2DTranspose(32,(3, 3), strides=(2,2), padding='same'))
model.add(LeakyReLU(alpha=0.2))
#model.add(UpSampling2D((2, 2)))
model.add(Conv2DTranspose(16,(3, 3), strides=(2,2), padding='same'))
model.add(LeakyReLU(alpha=0.2))
#model.add(UpSampling2D((2, 2)))
model.add(Conv2D(1,(3, 3), padding='same'))
model.add(Activation('sigmoid'))
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 20, 76, 16) 160
_________________________________________________________________
leaky_re_lu (LeakyReLU) (None, 20, 76, 16) 0
_________________________________________________________________
conv2d_1 (Conv2D) (None, 10, 38, 32) 4640
_________________________________________________________________
leaky_re_lu_1 (LeakyReLU) (None, 10, 38, 32) 0
_________________________________________________________________
flatten (Flatten) (None, 12160) 0
_________________________________________________________________
dense (Dense) (None, 96) 1167456
_________________________________________________________________
dense_1 (Dense) (None, 12160) 1179520
_________________________________________________________________
reshape (Reshape) (None, 10, 38, 32) 0
_________________________________________________________________
conv2d_transpose (Conv2DTran (None, 20, 76, 32) 9248
_________________________________________________________________
leaky_re_lu_2 (LeakyReLU) (None, 20, 76, 32) 0
_________________________________________________________________
conv2d_transpose_1 (Conv2DTr (None, 40, 152, 16) 4624
_________________________________________________________________
leaky_re_lu_3 (LeakyReLU) (None, 40, 152, 16) 0
_________________________________________________________________
conv2d_2 (Conv2D) (None, 40, 152, 1) 145
_________________________________________________________________
activation (Activation) (None, 40, 152, 1) 0
=================================================================
Total params: 2,365,793
Trainable params: 2,365,793
Non-trainable params: 0
_________________________________________________________________
I use MSE and adam optimizers.
The problems I encountered are :
The model overfits to the images most present in the dataset, so it creates many categories for the same type of that document when there is very little difference between them (just a small logo added, and it considers it a new cluster)
The images that are less present are not learned enough, I get a very blurry output, and most of them are clustered as -1 with DBSCAN.
Any idea how to make the model more effective ? I don't want it to overfit yet it's underfitting some images.
What are good layers/activation functions/regluarizers to use ? Should I increase the compressed representation size or decrease it ? It's very difficult to benchmark the effect of changes in the network, all I can do is run the dbscan clustering and look at the output classes, but this still depends on the dbscan epsilon parameter so I can't know if the model did well or not.

Answer 1

There are many things that you could try, and it is hard to know whether will it be effective before actually trying.
I will first address the problems which you explicitly stated.
"The model overfits to the images most present in the dataset"
You may try using bigger datasets, or if you can't, using the smaller model might work.
You may try using a pretrained model and run a Transfer Learning on it.
You may try Early Stopping.
"The images that are less present are not learned enough"
You may try using datasets that are not biased towards particular
classes.
"What are good layers/activation functions/regluarizers to use ?"
For activation function, ReLU and its variants mostly work well.
There are various layers and combination of them that you can use. Why don't you try using modern SOTA(State of the art) CNN network's architecture as a reference? (You can easily find some of them on Here )
"This is my current model, what do you think ?"
At least, the architecture seems old. If it works well, it would be fine. However, if needed, try using the modern SOTA architectures as mentioned earlier.
"Should I increase the compressed representation size or decrease it ?"
It is unclear. You should try both and pick the better performing method.
You could try different training methods too!
Hand-labeling all of the images would be a complete nightmare, so you might try labeling some of them and run a semi-supervised learning. Ex) SimCLR
Or else, you could search for researches about document-image-classification and use them as a reference.
Hope the answer helped!