Tuesday, February 7, 2023
HomeArtificial IntelligencePicture Augmentation for Deep Studying with Keras

Picture Augmentation for Deep Studying with Keras

Final Up to date on August 6, 2022

Information preparation is required when working with neural networks and deep studying fashions. More and more, information augmentation can be required on extra advanced object recognition duties.

On this publish, you’ll uncover learn how to use information preparation and information augmentation together with your picture datasets when creating and evaluating deep studying fashions in Python with Keras.

After studying this publish, you’ll know:

  • In regards to the picture augmentation API supplied by Keras and learn how to use it together with your fashions
  • Easy methods to carry out characteristic standardization
  • Easy methods to carry out ZCA whitening of your pictures
  • Easy methods to increase information with random rotations, shifts, and flips
  • Easy methods to save augmented picture information to disk

Kick-start your mission with my new e-book Deep Studying With Python, together with step-by-step tutorials and the Python supply code information for all examples.

Let’s get began.

  • Jun/2016: First printed
  • Replace Aug/2016: The examples on this publish had been up to date for the most recent Keras API. The datagen.subsequent() perform was eliminated
  • Replace Oct/2016: Up to date for Keras 1.1.0, TensorFlow 0.10.0 and scikit-learn v0.18
  • Replace Jan/2017: Up to date for Keras 1.2.0 and TensorFlow 0.12.1
  • Replace Mar/2017: Up to date for Keras 2.0.2, TensorFlow 1.0.1 and Theano 0.9.0
  • Replace Sep/2019: Up to date for Keras 2.2.5 API
  • Replace Jul/2022: Up to date for TensorFlow 2.x API with a workaround on the characteristic standardization concern

For an prolonged tutorial on the ImageDataGenerator for picture information augmentation, see:

Keras Picture Augmentation API

Like the remainder of Keras, the picture augmentation API is easy and highly effective.

Keras offers the ImageDataGenerator class that defines the configuration for picture information preparation and augmentation. This contains capabilities comparable to:

  • Pattern-wise standardization
  • Characteristic-wise standardization
  • ZCA whitening
  • Random rotation, shifts, shear, and flips
  • Dimension reordering
  • Save augmented pictures to disk

An augmented picture generator could be created as follows:

Slightly than performing the operations in your total picture dataset in reminiscence, the API is designed to be iterated by the deep studying mannequin becoming course of, creating augmented picture information for you simply in time. This reduces your reminiscence overhead however provides some further time price throughout mannequin coaching.

After you may have created and configured your ImageDataGenerator, you could match it in your information. This may calculate any statistics required to really carry out the transforms to your picture information. You are able to do this by calling the match() perform on the information generator and passing it to your coaching dataset.

The information generator itself is, the truth is, an iterator, returning batches of picture samples when requested. You’ll be able to configure the batch dimension and put together the information generator and get batches of pictures by calling the stream() perform.

Lastly, you may make use of the information generator. As a substitute of calling the match() perform in your mannequin, you could name the fit_generator() perform and cross within the information generator and the specified size of an epoch in addition to the whole variety of epochs on which to coach.

You’ll be able to be taught extra concerning the Keras picture information generator API within the Keras documentation.

Need assistance with Deep Studying in Python?

Take my free 2-week electronic mail course and uncover MLPs, CNNs and LSTMs (with code).

Click on to sign-up now and in addition get a free PDF Book model of the course.

Level of Comparability for Picture Augmentation

Now that you know the way the picture augmentation API in Keras works, let’s take a look at some examples.

We are going to use the MNIST handwritten digit recognition process in these examples. To start with, let’s check out the primary 9 pictures within the coaching dataset.

Working this instance offers the next picture that you should use as a degree of comparability with the picture preparation and augmentation within the examples beneath.

Instance MNIST pictures

Characteristic Standardization

Additionally it is potential to standardize pixel values throughout your complete dataset. That is referred to as characteristic standardization and mirrors the kind of standardization typically carried out for every column in a tabular dataset.

You’ll be able to carry out characteristic standardization by setting the featurewise_center and featurewise_std_normalization arguments to True on the ImageDataGenerator class. These are set to False by default. Nonetheless, the latest model of Keras has a bug within the characteristic standardization in order that the imply and normal deviation is calculated throughout all pixels. When you use the match() perform from the ImageDataGenerator class, you will notice a picture much like the one above:

For instance, the minimal, imply, and most values from the batch printed above are:

And the picture displayed is as follows:

Picture from feature-wise standardization

The workaround is to compute the characteristic standardization manually. Every pixel ought to have a separate imply and normal deviation, and it needs to be computed throughout completely different samples however unbiased from different pixels in the identical pattern. You simply want to switch the match() perform with your individual computation:

The minimal, imply, and most as printed now have a wider vary:

Working this instance, you possibly can see that the impact is completely different, seemingly darkening and lightening completely different digits.

Standardized characteristic MNIST pictures

ZCA Whitening

A whitening remodel of a picture is a linear algebraic operation that reduces the redundancy within the matrix of pixel pictures.

Much less redundancy within the picture is meant to higher spotlight the buildings and options within the picture to the training algorithm.

Sometimes, picture whitening is carried out utilizing the Principal Part Evaluation (PCA) method. Extra lately, an alternate referred to as ZCA (be taught extra in Appendix A of this tech report) reveals higher ends in reworked pictures that hold all the unique dimensions. And in contrast to PCA, the ensuing reworked pictures nonetheless seem like their originals. Exactly, whitening converts every picture right into a white noise vector, i.e., every component within the vector has zero imply and unit normal derivation and is statistically unbiased of one another.

You’ll be able to carry out a ZCA whitening remodel by setting the zca_whitening argument to True. However as a result of identical concern as characteristic standardization, you could first zero-center your enter information individually:

Working the instance, you possibly can see the identical common construction within the pictures and the way the define of every digit has been highlighted.

ZCA whitening MNIST pictures

Random Rotations

Typically pictures in your pattern information might have various and completely different rotations within the scene.

You’ll be able to practice your mannequin to higher deal with rotations of pictures by artificially and randomly rotating pictures out of your dataset throughout coaching.

The instance beneath creates random rotations of the MNIST digits as much as 90 levels by setting the rotation_range argument.

Working the instance, you possibly can see that pictures have been rotated left and proper as much as a restrict of 90 levels. This isn’t useful on this downside as a result of the MNIST digits have a normalized orientation, however this remodel may be of assist when studying from images the place the objects might have completely different orientations.

Random rotations of MNIST pictures

Random Shifts

Objects in your pictures will not be centered within the body. They might be off-center in quite a lot of alternative ways.

You’ll be able to practice your deep studying community to anticipate and at present deal with off-center objects by artificially creating shifted variations of your coaching information. Keras helps separate horizontal and vertical random shifting of coaching information by the width_shift_range and height_shift_range arguments.

Working this instance creates shifted variations of the digits. Once more, this isn’t required for MNIST because the handwritten digits are already centered, however you possibly can see how this may be helpful on extra advanced downside domains.

Random shifted MNIST pictures

Random Flips

One other augmentation to your picture information that may enhance efficiency on giant and complicated issues is to create random flips of pictures in your coaching information.

Keras helps random flipping alongside each the vertical and horizontal axes utilizing the vertical_flip and horizontal_flip arguments.

Working this instance, you possibly can see flipped digits. Flipping digits is just not helpful as they are going to all the time have the proper left and proper orientation, however this can be helpful for issues with images of objects in a scene that may have a diverse orientation.

Randomly flipped MNIST pictures

Saving Augmented Pictures to File

The information preparation and augmentation are carried out simply in time by Keras.

That is environment friendly by way of reminiscence, however you could require the precise pictures used throughout coaching. For instance, maybe you want to use them with a special software program package deal later or solely generate them as soon as and use them on a number of completely different deep studying fashions or configurations.

Keras lets you save the pictures generated throughout coaching. The listing, filename prefix, and picture file sort could be specified to the stream() perform earlier than coaching. Then, throughout coaching, the generated pictures can be written to the file.

The instance beneath demonstrates this and writes 9 pictures to a “pictures” subdirectory with the prefix “aug” and the file sort of PNG.

Working the instance, you possibly can see that pictures are solely written when they’re generated.

Augmented MNIST Images Saved To File

Augmented MNIST pictures saved to file

Ideas for Augmenting Picture Information with Keras

Picture information is exclusive in which you can assessment the information and reworked copies of the information and shortly get an thought of how the mannequin might understand it.

Under are some ideas for getting probably the most from picture information preparation and augmentation for deep studying.

  • Evaluate Dataset. Take a while to assessment your dataset in nice element. Take a look at the pictures. Pay attention to picture preparation and augmentations which may profit the coaching means of your mannequin, comparable to the necessity to deal with completely different shifts, rotations, or flips of objects within the scene.
  • Evaluate Augmentations. Evaluate pattern pictures after the augmentation has been carried out. It’s one factor to intellectually know what picture transforms you’re utilizing; it’s a very completely different factor to take a look at examples. Evaluate pictures each with particular person augmentations you’re utilizing in addition to the total set of augmentations you propose to make use of. You might even see methods to simplify or additional improve your mannequin coaching course of.
  • Consider a Suite of Transforms. Attempt multiple picture information preparation and augmentation scheme. Usually you could be shocked by the outcomes of a knowledge preparation scheme you didn’t suppose could be useful.


On this publish, you found picture information preparation and augmentation.

You found a variety of methods you should use simply in Python with Keras for deep studying fashions. You realized about:

  • The ImageDataGenerator API in Keras for producing reworked pictures simply in time
  • Pattern-wise and Characteristic-wise pixel standardization
  • The ZCA whitening remodel
  • Random rotations, shifts, and flips of pictures
  • Easy methods to save reworked pictures to file for later reuse

Do you may have any questions on picture information augmentation or this publish? Ask your questions within the feedback, and I’ll do my greatest to reply.

Develop Deep Studying Initiatives with Python!

Deep Learning with Python

 What If You Might Develop A Community in Minutes

…with just some traces of Python

Uncover how in my new Book:

Deep Studying With Python

It covers end-to-end initiatives on matters like:

Multilayer PerceptronsConvolutional Nets and Recurrent Neural Nets, and extra…

Lastly Carry Deep Studying To

Your Personal Initiatives

Skip the Teachers. Simply Outcomes.

See What’s Inside



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments