How to calculate RAM memory needed for training? - machine-learning

Question

How can I calculate RAM memory needed for training a keras model? I want to calculate this because I sometimes encounter the exceeds system memory error when training models. Here is my model, for example:
Layer (type) Output Shape Param #
=================================================================
conv2d_1 (Conv2D) (None, 30, 30, 32) 320
_________________________________________________________________
conv2d_2 (Conv2D) (None, 28, 28, 32) 9248
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 14, 14, 32) 0
_________________________________________________________________
dropout_1 (Dropout) (None, 14, 14, 32) 0
_________________________________________________________________
conv2d_3 (Conv2D) (None, 12, 12, 64) 18496
_________________________________________________________________
conv2d_4 (Conv2D) (None, 10, 10, 64) 36928
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 5, 5, 64) 0
_________________________________________________________________
dropout_2 (Dropout) (None, 5, 5, 64) 0
_________________________________________________________________
flatten_1 (Flatten) (None, 1600) 0
_________________________________________________________________
dense_1 (Dense) (None, 128) 204928
_________________________________________________________________
dropout_3 (Dropout) (None, 128) 0
_________________________________________________________________
dense_2 (Dense) (None, 10) 1290
=================================================================
Total params: 271,210
Trainable params: 271,210
Non-trainable params: 0

Answer 1

Assuming each parameter is a 32-bit datatype (single-precision floating point, 4 bytes). Your memory usage should be somewhere around: (# of params) * 4B
In this case: 271,210 * 4B = 1084840B =~ 1MB
However, there is an important consideration to keep in mind. This is assuming a batch size of 1, i.e. you're loading in 1 input at a time. If you are using minibatch (typical batch size of 32 or 64) then you'll have to multiply that memory calculation by the size of the batch. If you are using batch gradient descent, then you may be using your entire dataset on each batch. In this case, your memory requirements could be enormous.
This analysis inspired by: https://datascience.stackexchange.com/questions/17286/cnn-memory-consumption