Transfer Learning with TensorFlow : Scaling Up
Notebook demonstrates scaling up the Transfer learning model from the previous post for the FoodVision dataset from kaggle
- Transfer Learning in TensorFlow : Scaling Up
- Creating helper functions
- 101 Food Classes : working with less data
- Train a large Model with transfer learning on 10% of 101 food classes
- Fine-tuning
- Saving and loading our Model
- Evaluating the performance of our model across all different classes
- Making predictions with our trained model
- Evluating our model's predictions
- Making a Confusion Matrix
- Making a Classification Report
- Visualizing predictions on custom images
- Finding the most wrong predictions
- Test our model on custom images
Transfer Learning in TensorFlow : Scaling Up
This Notebook is an account of my working for the Udemy course :TensorFlow Developer Certificate in 2022: Zero to Mastery.
Concepts covered in this Notebook:
- Dowloading and preparing 10% of all Food101 Classes (7500+ training images)
- Training a transfer learning feature extraction model.
- Fine-tuning feature extraction model to beat the original Food101 with only 10% of data.
- Evaluating Food Vison Mini's predictions
- Find the most wrong predictions(on test dataset)
- Making predictions with Food Vision mini on our own custom images
In this Notebook, we are scaling up our models for even larger dataset and with more Classes of data -- For all classes of the FOOD Vision Dataset(i.e. 101 classes of Food)
!wget https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/extras/helper_functions.py
from helper_functions import create_tensorboard_callback, plot_loss_curves, unzip_data, compare_historys, walk_through_dir
!wget https://storage.googleapis.com/ztm_tf_course/food_vision/101_food_classes_10_percent.zip
unzip_data("101_food_classes_10_percent.zip")
train_dir = "101_food_classes_10_percent/train/"
test_dir = "101_food_classes_10_percent/test/"
# how many image classes are there
walk_through_dir("101_food_classes_10_percent")
import tensorflow as tf
IMG_SIZE = (224,224)
train_data_all_10_percent = tf.keras.preprocessing.image_dataset_from_directory(train_dir,
label_mode = "categorical",
image_size = IMG_SIZE)
test_data = tf.keras.preprocessing.image_dataset_from_directory(test_dir,
label_mode = "categorical",
image_size = IMG_SIZE,
shuffle = False) # don't shuffle test data for prediction analysis
Train a large Model with transfer learning on 10% of 101 food classes
Here are the steps we're going to take :
- Create a ModelCheckpoint callback
- Create a data augmentation layer to build data augmentation right into the layer
- Build a headless(no top layers) Functional EfficientNetB0 backboned-model
- Compile our model
- Feature extract for 5 full passes(5 epochs on the traindataset and validate on 15% of the test data, to save epoch time)
checkpoint_path = "101_classes_10_percent_data_model_checkpoint"
checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(checkpoint_path,
save_weights_only = True,
monitor = "val_accuracy",
save_best_only = True)
from tensorflow.keras import layers
from tensorflow.keras.layers.experimental import preprocessing
from tensorflow.keras.models import Sequential
# Setup data augmentation
data_augmentation = Sequential([
preprocessing.RandomFlip("horizontal"),
preprocessing.RandomRotation(0.2),
preprocessing.RandomHeight(0.2),
preprocessing.RandomWidth(0.2),
preprocessing.RandomZoom(0.2)
# preprocessing.Rescaling(1/255.) # rescale inputs of images to between 0 and 1 required for ResNet50 like models
], name = "data_augmentation")
base_model = tf.keras.applications.EfficientNetB0(include_top=False)
base_model.trainable = False
# Setup model architecture with trainable top layers
inputs = layers.Input(shape = (224,224,3), name = "input_layer")
x = data_augmentation(inputs), # augment images (only happens during training phase)
x = base_model(x, training = False) # Put the base model in inference mode so weights don't get updated
x = layers.GlobalAveragePooling2D(name = "global_avg_pool_layer")(x)
outputs = layers.Dense(len(train_data_all_10_percent.class_names), activation = "softmax", name = "output_layer")(x)
model = tf.keras.Model(inputs,outputs)
model.summary()
model.compile(loss = "categorical_crossentropy",
optimizer = tf.keras.optimizers.Adam(),
metrics = ["accuracy"])
# Fit
history_all_classes_10_percent = model.fit(train_data_all_10_percent,
epochs =5,
validation_data =test_data,
validation_steps = int(0.25*len(test_data)),
callbacks = [checkpoint_callback])
feature_extraction_results= model.evaluate(test_data)
feature_extraction_results
import matplotlib.pyplot as plt
plt.style.use('dark_background')
plot_loss_curves(history_all_classes_10_percent)
Ideally, the two curves should be very close to each other, but if they are not close to each other it means, our model maybe overfitting(Performing too well on the training data and not generalizing to unseen data)
base_model.trainable = True
# Refreeze every layer except the last 5
for layer in base_model.layers[:-5]:
layer.trainable = False
model.compile(loss = "categorical_crossentropy",
optimizer = tf.keras.optimizers.Adam(lr = 0.0001),
metrics = ["accuracy"])
# Check the trainable layers
for layer in model.layers:
print(layer.name, layer.trainable)
# Check which layers are trainable in our base model
for layer_number , layer in enumerate(model.layers[2].layers):
print(layer_number, layer.name, layer.trainable)
fine_tune_epochs = 10 # model has already done 5 epochs(feature extraction)
# This is the total no of epochs 5 for feature extraction and 5 for fine-tuning
# Fine-tune model
history_all_classes_10_percentp_fine_tune = model.fit(train_data_all_10_percent,
epochs = fine_tune_epochs,
validation_data = test_data,
validation_steps =int(0.25*len(test_data)),
initial_epoch = history_all_classes_10_percent.epoch[-1])
all_classes_10_percent_fine_tune_results = model.evaluate(test_data)
all_classes_10_percent_fine_tune_results
compare_historys(original_history = history_all_classes_10_percent,
new_history = history_all_classes_10_percentp_fine_tune,
initial_epochs = 5)
model.save("/content/drive/MyDrive/tensorflowcourseudemy/101_food_classes_10_percent_fine_tuned_model")
# Load and evaluate the saved model
loaded_model = tf.keras.models.load_model("/content/drive/MyDrive/tensorflowcourseudemy/101_food_classes_10_percent_fine_tuned_model")
loaded_model_results = loaded_model.evaluate(test_data)
loaded_model_results
all_classes_10_percent_fine_tune_results
Our loaded model gives the exact same results as the model we performed on this Notebook. That means we get the same predicts if we use the saved model in any application.
import tensorflow as tf
# Download a pre-trained model
!wget https://storage.googleapis.com/ztm_tf_course/food_vision/06_101_food_class_10_percent_saved_big_dog_model.zip
unzip_data("/content/06_101_food_class_10_percent_saved_big_dog_model.zip")
# Load in saved model
model = tf.keras.models.load_model("/content/06_101_food_class_10_percent_saved_big_dog_model")
results_downloaded_model = model.evaluate(test_data)
results_downloaded_model
preds_probs = model.predict(test_data, verbose = 1)
preds_probs
len(preds_probs)
preds_probs.shape
The predictions were done on 25250 images and the shape of our predictions is (25250,101)
preds_probs[:10]
preds_probs[0], len(preds_probs[0]), sum(preds_probs[0])
All of the prediction probabilites(array of with N number of variables, where N is the number of classes) for any prediction ideally should sum up to 1. But the value we got here is 1.0000000616546507 which is close to one, it's not because there is something wrong with the model, it is because of the way computers store numbers in memory. To understand better look into the precision of various datatypes.
print(f"Number of prediction probabilites for sample 0: {len(preds_probs)}")
print(f"What prediction probability sample 0 looks like: \n {preds_probs[0]}")
print(f"The class with the highest predicted probability by the model for sample 0: {preds_probs[0].argmax()}")
test_data.class_names[52]
pred_classes = preds_probs.argmax(axis =1)
pred_classes[:10]
len(pred_classes)
Now we've got a predictions array of all our model's predictions, to evaluate them, we need to compare them to the ground truth labels.
y_labels = []
for images, labels in test_data.unbatch():
y_labels.append(labels.numpy().argmax()) # currently test labels look like : [0,0,0,1,0...] we want the index value
y_labels[:10] # look at the first 10
len(y_labels)
results_downloaded_model
from sklearn.metrics import accuracy_score
sklearn_accuracy = accuracy_score(y_true = y_labels,
y_pred = pred_classes)
sklearn_accuracy
import numpy as np
np.isclose(results_downloaded_model[1], sklearn_accuracy)
from helper_functions import make_confusion_matrix
class_names = test_data.class_names
class_names[:10]
# plot_confusion_matrix function - https://scikit-learn.org/stable/modules/generated/sklearn.metrics.plot_confusion_matrix.html
import itertools
import matplotlib.pyplot as plt
import numpy as np
from sklearn.metrics import confusion_matrix
# Our function needs a different name to sklearn's plot_confusion_matrix
def make_confusion_matrix(y_true, y_pred, classes=None, figsize=(10, 10), text_size=15, norm=False, savefig=False):
"""Makes a labelled confusion matrix comparing predictions and ground truth labels.
If classes is passed, confusion matrix will be labelled, if not, integer class values
will be used.
Args:
y_true: Array of truth labels (must be same shape as y_pred).
y_pred: Array of predicted labels (must be same shape as y_true).
classes: Array of class labels (e.g. string form). If `None`, integer labels are used.
figsize: Size of output figure (default=(10, 10)).
text_size: Size of output figure text (default=15).
norm: normalize values or not (default=False).
savefig: save confusion matrix to file (default=False).
Returns:
A labelled confusion matrix plot comparing y_true and y_pred.
Example usage:
make_confusion_matrix(y_true=test_labels, # ground truth test labels
y_pred=y_preds, # predicted labels
classes=class_names, # array of class label names
figsize=(15, 15),
text_size=10)
"""
# Create the confustion matrix
cm = confusion_matrix(y_true, y_pred)
cm_norm = cm.astype("float") / cm.sum(axis=1)[:, np.newaxis] # normalize it
n_classes = cm.shape[0] # find the number of classes we're dealing with
# Plot the figure and make it pretty
fig, ax = plt.subplots(figsize=figsize)
cax = ax.matshow(cm, cmap=plt.cm.Blues) # colors will represent how 'correct' a class is, darker == better
fig.colorbar(cax)
# Are there a list of classes?
if classes:
labels = classes
else:
labels = np.arange(cm.shape[0])
# Label the axes
ax.set(title="Confusion Matrix",
xlabel="Predicted label",
ylabel="True label",
xticks=np.arange(n_classes), # create enough axis slots for each class
yticks=np.arange(n_classes),
xticklabels=labels, # axes will labeled with class names (if they exist) or ints
yticklabels=labels)
# Make x-axis labels appear on bottom
ax.xaxis.set_label_position("bottom")
ax.xaxis.tick_bottom()
### Added: Rotate xticks for readability & increase font size (required due to such a large confusion matrix)
plt.xticks(rotation=70, fontsize=text_size)
plt.yticks(fontsize=text_size)
# Set the threshold for different colors
threshold = (cm.max() + cm.min()) / 2.
# Plot the text on each cell
for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
if norm:
plt.text(j, i, f"{cm[i, j]} ({cm_norm[i, j]*100:.1f}%)",
horizontalalignment="center",
color="white" if cm[i, j] > threshold else "black",
size=text_size)
else:
plt.text(j, i, f"{cm[i, j]}",
horizontalalignment="center",
color="white" if cm[i, j] > threshold else "black",
size=text_size)
# Save the figure to the current working directory
if savefig:
fig.savefig("confusion_matrix.png")
make_confusion_matrix(y_true = y_labels,
y_pred = pred_classes,
classes = class_names,
figsize= (100,100),
text_size = 20,
savefig = True)
# Make a classification report
from sklearn.metrics import classification_report
print(classification_report(y_true = y_labels,
y_pred = pred_classes))
The numbers above give a great class-by-class evaluation of our model's predictions but with so many classes, they're quite hard to understand.
# Get a dictionary of the classification report
classification_report_dict = classification_report(y_labels, pred_classes, output_dict = True)
classification_report_dict
Let's plot all our classes F1-Score
classification_report_dict["99"]["f1-score"]
# Create empty dictionary
class_f1_scores = {}
# Loop through classification report dictionary items
for k, v in classification_report_dict.items():
if k == "accuracy": # Stop once we get the accuracy key
break
else:
#Add class names and f1-scores to new dictionary
class_f1_scores[class_names[int(k)]] = v["f1-score"]
class_f1_scores
# Turn f1-scores into dataframe for visualization
import pandas as pd
f1_scores = pd.DataFrame({"class_names": list(class_f1_scores.keys()),
"f1-score": list(class_f1_scores.values())}).sort_values("f1-score", ascending = False)
f1_scores
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(12, 25))
scores = ax.barh(range(len(f1_scores)), f1_scores["f1-score"].values)
ax.set_yticks(range(len(f1_scores)))
ax.set_yticklabels(list(f1_scores["class_names"]))
ax.set_xlabel("f1-score")
ax.set_title("F1-Scores for 10 Different Classes")
ax.invert_yaxis(); # reverse the order
def autolabel(rects): # Modified version of: https://matplotlib.org/examples/api/barchart_demo.html
"""
Attach a text label above each bar displaying its height (it's value).
"""
for rect in rects:
width = rect.get_width()
ax.text(1.03*width, rect.get_y() + rect.get_height()/1.5,
f"{width:.2f}",
ha='center', va='bottom')
autolabel(scores)
Visualizing predictions on custom images
How does our model go on food images not even in our test dataset
To visualize our model's predictions on our images we'll need a function to load and preprocess images, specifically it will need to:
- Read in a target image file path suing
tf.io.read_file() - Turn the image into a Tensor using
tf.io.decode_image() - Resize the image tensor to be the same size as the images our model has trained on using
tf.image.resize() - Scale the image to get all of the pixel values between 0 & 1 (if necessary)
def load_and_prep_image(filename, img_shape = 224, scale=True):
"""
Reads in an image from filename, turns it into a tensor and reshapes intoo
specified shape (img_shape, color_channels = 3)
Args:
filename (str) : path to target image
image_shape (int) : height/width dimension of target image size
scale(bool) : scale pixel values from 0-255 to 0-1 or not
"""
# Read in the image
img = tf.io.read_file(filename)
# Decode the image into tensor
img = tf.io.decode_image(img, channels = 3)
# Resize the image
img = tf.image.resize(img,[img_shape, img_shape])
# Scale(yes/no)
if scale:
# rescale the image (get all values between 0 and 1)
return img/255.
else:
return img # Don't need to rescale images for EfficientNetB0
Now, we got function to load and prepare target images, let's now write some code to visualize images, their target lanel and our model's predictions.
Specifically, we'll write some code to:
- Load a few random images from the test dataset
- Make predictions on the loaded images
- Plot the original image(s) along with the model's predictions, prediction probabilty and ground truth labels
import os
import random
plt.figure(figsize=(17, 10))
for i in range(3):
# Choose a random image from a random class
class_name = random.choice(class_names)
filename = random.choice(os.listdir(test_dir + "/" + class_name))
filepath = test_dir + class_name + "/" + filename
# Load the image and make predictions
img = load_and_prep_image(filepath, scale=False) # don't scale images for EfficientNet predictions
pred_prob = model.predict(tf.expand_dims(img, axis=0)) # model accepts tensors of shape [None, 224, 224, 3]
pred_class = class_names[pred_prob.argmax()] # find the predicted class
# Plot the image(s)
plt.subplot(1, 3, i+1)
plt.imshow(img/255.)
if class_name == pred_class: # Change the color of text based on whether prediction is right or wrong
title_color = "g"
else:
title_color = "r"
plt.title(f"actual: {class_name}, pred: {pred_class}, prob: {pred_prob.max():.2f}", c=title_color)
plt.axis(False);
Finding the most wrong predictions
- A good way to inspect your model's performance is to view the wrong predictions with the highest prediction probability (or highest loss)
- Can reveal insights such as:
- Data issues (wrong labels)
- Confusing classes(get better/more diverse data)
To find out where our model is most wrong, do the following:
- Get all of the image file paths in the test datasets using
list_files()method - Create a pandas DataFrame of the image filepaths, ground truth labels, predicted classes(from our model), max prediction probabilities, prediction_classnames and ground truth labels.
- Use our DataFrame based on wrong predictions (where the ground truth label doesn't match the prediction)
- Sort the DataFrame based on wrong predictions (have the highest prediction probability at the top)
- Visualize the images with the highest prediction probabilites but have the wrong prediction.
filepaths = []
for filepath in test_data.list_files("101_food_classes_10_percent/test/*/*.jpg",
shuffle=False):
filepaths.append(filepath.numpy())
filepaths[:10]
import pandas as pd
pred_df = pd.DataFrame({"img_path": filepaths,
"y_true": y_labels,
"y_pred": pred_classes,
"pred_conf": preds_probs.max(axis=1),
"y_true_classname": [class_names[i] for i in y_labels],
"y_pred_classname": [class_names[i] for i in pred_classes]}) # get the maximum prediction prob value
pred_df
pred_df["pred_correct"] = pred_df["y_true"] == pred_df["y_pred"]
pred_df.head()
top_100_wrong = pred_df[pred_df["pred_correct"] == False].sort_values("pred_conf", ascending = False)[:100]
top_100_wrong
images_to_view = 9
start_index = 0 # change the index to view more of the wrong predictions
plt.figure(figsize=(15,10))
for i, row in enumerate(top_100_wrong[start_index:start_index+9].itertuples()):
plt.subplot(3,3,i+1)
img = load_and_prep_image(row[1], scale = False)
_, _, _, _, pred_prob, y_true_classname, y_pred_classname, _ = row # only interested in a few parameters of each row
plt.imshow(img/255.)
plt.title(f"actual: {y_true_classname}, pred: {y_pred_classname},\n prob: {pred_prob}")
plt.axis(False)
These are the images where our model is predicting the image with high probability even when the prediction is not correct according to ground truth labels
from helper_functions import create_tensorboard_callback, plot_loss_curves, unzip_data, compare_historys
!wget https://storage.googleapis.com/ztm_tf_course/food_vision/custom_food_images.zip
unzip_data("custom_food_images.zip")
import os
custom_food_images = ["custom_food_images/" + img_path for img_path in os.listdir("custom_food_images")]
custom_food_images
for img in custom_food_images:
img = load_and_prep_image(img, scale=False) # load in target image and turn it into tensor
pred_prob = model.predict(tf.expand_dims(img, axis=0)) # make prediction on image with shape [None, 224, 224, 3]
pred_class = class_names[pred_prob.argmax()] # find the predicted class label
# Plot the image with appropriate annotations
plt.figure()
plt.imshow(img/255.) # imshow() requires float inputs to be normalized
plt.title(f"pred: {pred_class}, prob: {pred_prob.max():.2f}")
plt.axis(False)