These notes explain how to train a neural network to classify images on lungs that are affected by pneumonia, COVID-19 disease or belong to an healthy person. It is part of a project for the OpenCV official course.
You can find the colab here : https://colab.research.google.com/drive/14T75yLa9gbT2BW33F5WMVLA6GHrhR_lu?usp=sharing
Neural network
The convolutional layer of an existing neural network, MobileNet, is used as a base, then a set of Dense layers has been added to complete the network, a processed called Transfer Learning.
The dataset is divided in 3 main classes on 3 directories, representing the test, the train and the validation data
- train
- Pneumonia
- Covid
- Normal
- test
- Pneumonia
- Covid
- Normal
- val
- Pneumonia
- Covid
- Normal
and it is available here https://www.dropbox.com/s/73s9n7nugqrv1h7/Dataset.zip?dl=1, so after importing the necessary directives
1import tensorflow as tf
2from keras.preprocessing.image import ImageDataGenerator
3tf.keras.backend.clear_session()
4from keras.layers import Input, Softmax, Dense, Dropout, BatchNormalization
5from keras.models import Model
6import numpy as np
7from sklearn.metrics import classification_report
8from sklearn.metrics import confusion_matrix
it is downloaded and unzipped in the colab environment.
!wget https://www.dropbox.com/s/73s9n7nugqrv1h7/Dataset.zip?dl=1 -O 'archive.zip'
!unzip -q '/content/archive.zip'
!rm -rf '/content/archive.zip'
Then the batch size, seed and main path are set:
BATCH_SIZE = 64
SEED = 21
dataset_path = '/content/Dataset'
Data augmentation
The data is augmented and rescaled through the ImageDataGenerator, which is fed by the images in the dataset
1train_datagen = train_val_gen.flow_from_directory(
2 directory = dataset_path + '/train',
3 target_size = (224, 224),
4 color_mode = "rgb",
5 classes = None,
6 class_mode = "categorical",
7 batch_size = BATCH_SIZE,
8 shuffle = True,
9 seed = SEED,
10 interpolation = "nearest")
11
12val_datagen = train_val_gen.flow_from_directory(
13 directory = dataset_path + '/val',
14 target_size = (224, 224),
15 color_mode = "rgb",
16 classes = None,
17 class_mode = "categorical",
18 batch_size = BATCH_SIZE,
19 shuffle = True,
20 seed = SEED,
21 interpolation = "nearest")
22
23test_datagen = test_gen.flow_from_directory(
24 directory = dataset_path + '/test',
25 target_size = (224, 224),
26 color_mode = "rgb",
27 classes = None,
28 class_mode = "categorical",
29 batch_size = 1,
30 shuffle = False,
31 seed = SEED,
32 interpolation = "nearest")
The important thing is to not shuffle the data in the test_datagen and use a batch size of 1.
The model is then initialized : it is based on MobileNet with a large number of classes that will be changed later with the transfer learning.
1pretrained_model = tf.keras.applications.MobileNet(
2 weights = 'imagenet',
3 classes = 1000,
4 input_shape = (224, 224, 3),
5 include_top = False,
6 pooling = 'max')
The summary of the model is
1print(pretrained_model.summary())
2...
3...
4conv_pw_13_relu (ReLU) (None, 7, 7, 1024) 0
5
6 global_max_pooling2d_1 (Glo (None, 1024) 0
7 balMaxPooling2D)
8
9========================================================
10Total params: 3,228,864
11Trainable params: 3,206,976
12Non-trainable params: 21,888
Transfer learning
After the convolutional layer, used as feature detector, has been set, it is time to add the Dense layers
1x = Dense(512, activation='relu')(pretrained_model.output)
2x = Dropout(0.5)(x)
3x = Dense(512, activation='relu')(x)
4x = BatchNormalization()(x)
5x = Dense(16, activation='relu')(x)
6predictions = Dense(3, activation = 'softmax')(x)
7
8
9model = Model(inputs = pretrained_model.input, outputs = predictions)
10
11
12model.compile(optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001),
13 loss = "categorical_crossentropy",
14 metrics = ["accuracy"])
15
In particular, the last layer ensure that we have the final classification of 3 values : penumonia, covid, normal
Callbacks
The callbacks to save on checkpoint the model and to reduce the learning rate on plateau (that is, when the gradient is not descending but remains on the same values) are added to the model
1model_filepath = '/content/best_model.h5'
2
3model_save = tf.keras.callbacks.ModelCheckpoint(
4 model_filepath,
5 monitor = "val_accuracy",
6 verbose = 0,
7 save_best_ony = True,
8 save_weightsonly = False,
9 mode = "max",
10 save_freq ="epoch")
11
12reduce_lr = tf.keras.callbacks.ReduceLROnPlateau(
13 monitor='val_loss',
14 factor=0.1,
15 patience=6,
16 verbose=1,
17 min_delta=5 * 1e-3,
18 min_lr=5 * 1e-9 )
19
20
21callback = [model_save, reduce_lr]
Let’s print the summary again
1print(model.summary())
2...
3...
4 batch_normalization_3 (Batc (None, 512) 2048
5 hNormalization)
6
7 dense_14 (Dense) (None, 16) 8208
8
9 dense_15 (Dense) (None, 3) 51
10
11=================================================================
12Total params: 4,026,627
13Trainable params: 4,003,715
14Non-trainable params: 22,912
We see that the number of total parameters have increased.
Train
It is time to train the model for 15 epochs.
1
2history = model.fit(train_datagen,
3 epochs = 15,
4 steps_per_epoch = (len(train_datagen)),
5 validation_data = val_datagen,
6 validation_steps = (len(val_datagen)),
7 shuffle = False,
8 callbacks = callback)
The output is
Epoch 1/15
177/177 [==============================] - 119s 651ms/step - loss: 0.5713 - accuracy: 0.7591 - val_loss: 0.6968 - val_accuracy: 0.7468 - lr: 1.0000e-04
Epoch 2/15
177/177 [==============================] - 113s 640ms/step - loss: 0.1800 - accuracy: 0.9353 - val_loss: 0.3375 - val_accuracy: 0.8700 - lr: 1.0000e-04
....
....
...
...
Epoch 15/15
177/177 [==============================] - 114s 643ms/step - loss: 0.0063 - accuracy: 0.9984 - val_loss: 0.1567 - val_accuracy: 0.9760 - lr: 1.0000e-05
The accuracy and the loss can be plotted from the history data structure
1mport matplotlib.pyplot as plt
2
3plt.figure(figsize = (15,7))
4
5tr_losses = history.history['loss']
6val_losses = history.history['val_loss']
7
8tr_accs = history.history['accuracy']
9val_accs = history.history['val_accuracy']
10
11plt.plot(tr_losses, label = "train_loss")
12plt.plot(val_losses, label = "val_loss")
13plt.xlabel("Number of epochs")
14plt.ylabel("Cost (J)")
15plt.grid()
16plt.legend()
17plt.show()
18
19plt.figure(figsize = (15,7))
20
21plt.plot(tr_accs, label = "acc_train")
22plt.plot(val_accs, label = "acc_val")
23plt.xlabel("Number of epochs")
24plt.ylabel("Accuracy")
25plt.grid()
26plt.legend()
27plt.show()
Evaluation and confusion matrix
The last step is to evaluate the model on a the test_datagen generator.
1predictions = model.predict(test_datagen,
2 verbose = 1,
3 steps = (len(test_datagen)))
4
5predictions.squeeze().argmax
6(axis-1)
The report shows a good precision and recall
1classification__report = classification_report(test_datagen.classes,
2 predictions.squeeze().argmax(axis = 1))
3print(classification__report)
1 precision recall f1-score support
2
3 0 0.98 0.99 0.99 491
4 1 0.96 0.98 0.97 545
5 2 0.99 0.97 0.98 527
6
7 accuracy 0.98 1563
8 macro avg 0.98 0.98 0.98 1563
9weighted avg 0.98 0.98 0.98 1563
Then the confusion matrix can be printed also. This matrix shows how many candidates are correctly identified and how manu aren’t
1confusion__matrix = confusion_matrix(test_datagen.classes,
2 predictions.squeeze().argmax(axis = 1))
3print(confusion__matrix)
4
5[[485 5 1]
6 [ 6 535 4]
7 [ 2 16 509]]
It can also be shown graphically (with a code snippet from scikit-learn web site https://scikit-learn.org/0.18/auto_examples/model_selection/plot_confusion_matrix.html)
1import itertools
2def plot_confusion_matrix(cm,
3 classes,
4 normalise = False,
5 title = 'Confusion matrix',
6 cmap = plt.cm.Reds):
7
8 plt.imshow(cm, interpolation = 'nearest', cmap = cmap)
9 plt.title(title)
10 plt.colorbar()
11 tick_marks = np.arange(len(classes))
12 plt.xticks(tick_marks, classes, rotation = 45)
13 plt.yticks(tick_marks, classes)
14
15 if normalise:
16 cm = cm.astype('float') / cm.sum(axis = 1)[:, np.newaxis]
17 cm = cm.round(2)
18
19 thresh = cm.max() / 2.
20
21 for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
22 plt.text(j, i, cm[i, j],
23 horizontalalignment = "center",
24 color = "white" if cm[i, j] > thresh else "black")
25
26 plt.tight_layout()
27 plt.ylabel('True label')
28 plt.xlabel('Predicted label')
It can be shown un-normalised
np.set_printoptions(precision = 2)
fig1 = plt.figure(figsize = (7, 6))
plot_confusion_matrix(confusion__matrix,
classes = np.unique(test_datagen.classes),
title = 'Confusion matrix without normalisation')
fig1.savefig('/content/cm_wo_norm.jpg')
plt.show()
and normalized
np.set_printoptions(precision = 2)
fig2 = plt.figure(figsize = (7,6))
plot_confusion_matrix(confusion__matrix,
classes = np.unique(test_datagen.classes),
normalise = True,
title = 'Normalised Confusion matrix')
fig2.savefig('/content/cm_norm.jpg')
plt.show()
