Using this library with Keras

[deepakanandece]

Can anyone give an example for using this library with image datagenerator flow from directory method of keras?

[marcelsan]

I do not have a example yet, but you should use the fit_generator from keras.

The keras documentation have a example showing how to use it.
https://keras.io/models/sequential/#fit_generator

[slothkong]

I will try to make this work and post if I succeed. Or did you try it, @deepakanandece? How did it go?

[jessestauffer]

Any updates on this? I would like to use this with Keras datagenerator as well.

[slothkong]

I decided to go with Keras ImageDataGenerator instead. It has all the basic augmentations covered.

[jessestauffer]

Was able to get this to play well with Keras.

`
def generator(features, labels, batch_size):

return_features = features.copy()
return_labels = labels.copy()

# create empty arrays to contains batch of features and labels
batch_features = np.zeros((batch_size, IMG_WIDTH, IMG_HEIGHT, NUM_CHANNELS))
batch_labels = np.zeros((batch_size, 2 * NUM_KEYPOINTS))	# X and Y coordinates

while True:

	for i in range(batch_size):

		# choose random index in features
		index = randint(0, len(return_features)-1)
		
		random_augmented_image, random_augmented_labels = augment(np.array([features[index]]), np.array([labels[index]]))
		batch_features[i] = random_augmented_image[0]
		batch_labels[i] = random_augmented_labels[0]

	yield batch_features, batch_labels`

then just call fit_generator on your model like this:

hist = model.fit_generator(generator(X_train, y_train, batch_size), steps_per_epoch = X_train.shape[0] / batch_size, epochs=epochs, validation_data=(X_test, y_test))

The augment() function is what is actually doing the imgaug work here. It just takes in X and y and calls seq_det.augment_images() and seq_det.augment_keypoints() to generate a new X and y if necessary. Haven't used the flow_from_directory method but am sure it would be similar to this. Hope this helps!

[jessestauffer]

NUM_KEYPOINTS is simply the number of keypoints you are performing the augmentation on. So if you are doing simple eye tracking with a network and are using 2 keypoints (one for each eye) then NUM_KEYPOINTS would be 2. It’s multiplied by 2 in the code to account for both the X and Y coordinates of each point.

…

On Apr 13, 2018, at 3:51 AM, Johan Bender ***@***.***> wrote: @jessestauffer Thanks for sharing! Could you help explain what the variable NUM_KEYPOINTS is referring to? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[titanbender]

Thanks for explaining @jessestauffer . In my case, I'm trying to predict between 5 different category classes, so I've made a change in order to save the one-hot encoded vectors for the y labels.

However, when I run the code, even though training works properly, I get much worse results with the fit_generator both with and without the augmenter. Actually, the network only performs at random accuracy (20%), while without the fit_generato the X_train and Y_train performs above 60% accuracy. Do you have any idea of what I'm doing wrong?

def generator(features, labels, batch_size):

    return_features = features.copy()
    return_labels = labels.copy()

    # create empty arrays to contains batch of features and labels
    batch_features = np.zeros((batch_size, 224, 224, 3))
    batch_labels = np.zeros(batch_size) # X and Y coordinates

    while True:
        batch_labels = []
        for i in range(batch_size):

            # choose random index in features
            index = np.random.randint(0, len(return_features)-1)
            random_augmented_image, random_augmented_labels = features[index], labels[index] #before it said Train_aug
            batch_features[i] = random_augmented_image
            batch_labels.append(random_augmented_labels)
            #batch_labels[i] = random_augmented_labels
        
        batch_labels = np.asarray(batch_labels)
        yield batch_features, batch_labels

model.fit_generator(generator(X_train, Y_train, batch_size), steps_per_epoch = X_train.shape[0] / batch_size, epochs=10, validation_data=(X_test, Y_test))

I've tried both with the augmenter turned on and off and the results seem to be the same for me :-( ...

[jessestauffer]

@titanbender Nothing stands out to me in the generator code. Could you share the code you were using to fit the model both with and without the generator?

[jessestauffer]

@titanbender Made a quick example of getting this to work with categories for you. https://github.com/jessestauffer/MNIST-CNN-Keras/blob/master/mnist.py

[titanbender]

Yes, here's the code with - and without the generator. I apologize for asking for your advice. It seems to me that I've switched out X_test, Y_test with X_valid and Y_valid. That being said, it shouldn't greatly decrease the accuracy of the training data which also is just about 20% with the generator.

With Generator

 model.fit_generator(generator(X_train, Y_train, batch_size), 
 steps_per_epoch = X_train.shape[0] / batch_size, epochs=10, 
 validation_data=(X_test, Y_test))

Without Generator

model.fit(X_train, Y_train,
               batch_size=batch_size,
               nb_epoch=nb_epoch,
               shuffle=True,
               verbose=1,
	           class_weight = class_weight,
              validation_data=(X_valid, Y_valid),
              )

Below is the print out from my 10 epochs.

Epoch 1/10
100/100 [==============================] - 263s 3s/step - loss: 1.8198 - acc: 0.1910 - val_loss: 2.6878 - val_acc: 0.2927
Epoch 2/10
100/100 [==============================] - 226s 2s/step - loss: 1.7798 - acc: 0.2020 - val_loss: 2.3771 - val_acc: 0.1438
Epoch 3/10
100/100 [==============================] - 226s 2s/step - loss: 1.7354 - acc: 0.2050 - val_loss: 1.7476 - val_acc: 0.1885
Epoch 4/10
100/100 [==============================] - 226s 2s/step - loss: 1.7376 - acc: 0.2005 - val_loss: 1.7086 - val_acc: 0.1954
Epoch 5/10
100/100 [==============================] - 226s 2s/step - loss: 1.6861 - acc: 0.2050 - val_loss: 8.1866 - val_acc: 0.0942
Epoch 6/10
100/100 [==============================] - 226s 2s/step - loss: 1.6665 - acc: 0.1950 - val_loss: 1.6018 - val_acc: 0.2837
Epoch 7/10
100/100 [==============================] - 226s 2s/step - loss: 1.6584 - acc: 0.1805 - val_loss: 1.6876 - val_acc: 0.1597
Epoch 8/10
100/100 [==============================] - 226s 2s/step - loss: 1.6462 - acc: 0.2015 - val_loss: 1.6303 - val_acc: 0.3075
Epoch 9/10
100/100 [==============================] - 226s 2s/step - loss: 1.6533 - acc: 0.1950 - val_loss: 1.8327 - val_acc: 0.1845
Epoch 10/10
100/100 [==============================] - 226s 2s/step - loss: 1.6364 - acc: 0.2160 - val_loss: 1.6592 - val_acc: 0.2659

[titanbender]

@jessestauffer thanks for putting together a sample for me. I really appreciate it! Will try to see if it helps in terms the medical images I work with. Have a great weekend!

[jessestauffer]

@titanbender no problem! Your fit and fit_generator code looks correct. Go through the sample and see if anything jumps out at you and let me know if you have any other questions.

[titanbender]

Thanks. Cheers!

[tavildar]

Hi -- thanks for the code! With respect to the original question, I was able to get this to play well with keras DataGenerator by using the preprocessing_function option. It has some limitations with respect to which Augmentations can be used, but it worked fo the cases I was trying.

from imgaug import augmenters as iaa

aug1 = iaa.GaussianBlur(sigma=(0, 2.0))
aug2 = iaa.AdditiveGaussianNoise(scale=0.01 * 255)

def additional_augmenation(image):
    image = aug1.augment_image(image)
    image = aug2.augment_image(image)
    return image

This additional function can be passed to keras data generator as a pre-processing function.

datagen = ImageDataGenerator(rotation_range=20, zoom_range=0.2,
                                    preprocessing_function=additional_augmenation)

One caveat to note is the following from Keras documentation regarding preprocessing_function:

function that will be implied on each input. The function will run after the image is resized and augmented. The function should take one argument: one image (Numpy tensor with rank 3), and should output a Numpy tensor with the same shape.

So, this approach may not work for all the augmentations.

[azmathmoosa]

@tavildar I tried your code and get this error in python 3

2018-05-15 22:51:57.949483: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:02:00.0, compute capability: 6.1)
Found 188609 images belonging to 128 classes.
Found 6200 images belonging to 128 classes.
Epoch 5/60
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py", line 401, in get_index
    return _SHARED_SEQUENCES[uid][i]
  File "/usr/local/lib/python3.6/dist-packages/keras/preprocessing/image.py", line 1034, in __getitem__
    return self._get_batches_of_transformed_samples(index_array)
  File "/usr/local/lib/python3.6/dist-packages/keras/preprocessing/image.py", line 1441, in _get_batches_of_transformed_samples
    x = self.image_data_generator.standardize(x)
  File "/usr/local/lib/python3.6/dist-packages/keras/preprocessing/image.py", line 789, in standardize
    x *= self.rescale
TypeError: Cannot cast ufunc multiply output from dtype('float64') to dtype('uint8') with casting rule 'same_kind'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py", line 578, in get
    inputs = self.queue.get(block=True).get()
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
TypeError: Cannot cast ufunc multiply output from dtype('float64') to dtype('uint8') with casting rule 'same_kind'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "train.py", line 275, in <module>
    main()
  File "train.py", line 272, in main
    callbacks = [checkpoint, early, tensorboard])
  File "/usr/local/lib/python3.6/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 2194, in fit_generator
    generator_output = next(output_generator)
  File "/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py", line 584, in get
    six.raise_from(StopIteration(e), e)
  File "<string>", line 3, in raise_from
StopIteration: Cannot cast ufunc multiply output from dtype('float64') to dtype('uint8') with casting rule 'same_kind'

[tavildar]

@azmathmoosa that seems like issue relating to your image being float -- I think imgaug requires data being uint8. See for example the following comment in the Readme

# All images must have numpy's dtype uint8. Values are expected to be in range 0-255.

[nilavghosh]

@titanbender . Hi, in your generator function you are really not augmenting your images but just randomizing the position of the images. To actually augment you should cal the augment code as specified by @jessestauffer
random_augmented_image, random_augmented_labels = augment(np.array([features[index]]), np.array([labels[index]]))

[nieszkodzi]

I have a nice example with keras, ImageDataGenerator and flow_from_dataframe, below fragment of code:

seq = iaa.Sequential([
iaa.Crop(px=(0, 16)), # crop images from each side by 0 to 16px (randomly chosen)
iaa.Fliplr(0.5), # horizontally flip 50% of the images
iaa.GaussianBlur(sigma=(0, 3.0)) # blur images with a sigma of 0 to 3.0
])

def augment(img):
seq_det = seq.to_deterministic()
aug_image = seq_det.augment_image(img)

return applications.inception_resnet_v2.preprocess_input(aug_image)

train_generator = image.ImageDataGenerator(preprocessing_function=augment)

train_flow = train_generator.flow_from_dataframe(
dataframe=train_df,
directory=train_data_dir,
x_col="path",
y_col=columns,
batch_size=batch_size,
class_mode="other",
target_size=(img_height ,img_width),
shuffle=True
)

[EXJUSTICE]

Thanks to @nieszkodzi and @tavildar for the excellent solutions. I've been trying to implement imgaug to work with Keras's ImageDataGenerator (and have found the thread where this is discussed), but am running into an Assertion Error with no explanation during the fit command

history = model.fit_generator(train_generator, steps_per_epoch = steps_per_epoch, epochs=epochs, workers=4, validation_data=validation_generator, validation_steps=validation_steps)

This yields the error:

AssertionError                            Traceback (most recent call last)
<ipython-input-39-e7ff542a51eb> in <module>()
     10                               workers=4,
     11                               validation_data=validation_generator,
---> 12                               validation_steps=validation_steps)

18 frames
/usr/local/lib/python3.6/dist-packages/imgaug/augmenters/blend.py in blend_alpha(image_fg, image_bg, alpha, eps)
     94     """
     95     assert image_fg.shape == image_bg.shape
---> 96     assert image_fg.dtype.kind == image_bg.dtype.kind
     97     # TODO switch to gate_dtypes()
     98     assert image_fg.dtype.name not in ["float128"]

`

I think this has to do with my images not being in the correct format I believe, but I'm not fully sure about this?
Any advice is appreciated!

[achillesrasquinha]

I found a much neater way to do this, I wrote a custom Augmenter for this at deeply

import deeply.img.augmenters as dia
import imgaug.augmenters as iaa
import imgaug as ia

image = ia.data.quokka()

Then,

augmentor = iaa.Sequential([
    dia.Combination([
        iaa.Fliplr(1.0),
        iaa.Flipud(1.0),
        iaa.Affine(scale = 1.3),
        iaa.Affine(scale = 0.7),
        iaa.Rotate(rotate = (-45, 45)),
        iaa.ShearX((-20, 20)),
        iaa.ShearY((-20, 20)),
        iaa.Translate(percent = (-0.1, 0.1)),
        iaa.Translate(percent = (-0.1, 0.1))
    ]),
    iaa.Resize({ "width": 250, "height": 250 })
])

images = augmenter(images = [image])

dia.Combination should perform a combination of augmenters listed (for e.g., Fliplr + Affine, Fliplr + ShearX, Affine + ShearY, Fliplr + Affine + ShearY, and so on...) and then each resultant image should be resized. This way, you're able to generate a large number of images just using imgaug. The caveat is to also consider a patch of this PR here.