Category: Misc

I’m using keras-tuner in order to do an hyperparameter optimization of a neural network.

I’m using an Hyperband optimization, and I call the search method as:

tuner.search(training_gen(), epochs=50, validation_data=valid_gen(), callbacks=[stop_early], steps_per_epoch=np.round(int(num_samples / batch_size), decimals=0), validation_freq=1, validation_steps=100) 

where the EarlyStopping callback is defined as:

stop_early = tf.keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0.1, mode='min', patience=15) 

The Keras Hyperband algorithm seems to work well: it simulates the various models, and when it reaches the Early Stopping condition on a model, the training of that model stops and the training of a new model starts.
What I noticed is that with this EarlyStopping callback implementation, when EarlyStopping stops the training of a model, before starting the training of the next model, the following two consecutive Python errors occur (anyway, the simulation doesn’t exit or generate exceptions, but it goes on, by simulating the successive model):

W tensorflow/core/framework/op_kernel.cc:1755] Invalid argument: ValueError: callback pyfunc_2 is not found Traceback (most recent call last): File "/home/username/anaconda3/envs/myenv/lib/python3.8/site-packages/tensorflow/python/ops/script_ops.py", line 233, in __call__ raise ValueError("callback %s is not found" % token) ValueError: callback pyfunc_2 is not found W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Invalid argument: ValueError: callback pyfunc_2 is not found Traceback (most recent call last): File "/home/username/anaconda3/envs/myenv/lib/python3.8/site-packages/tensorflow/python/ops/script_ops.py", line 233, in __call__ raise ValueError("callback %s is not found" % token) ValueError: callback pyfunc_2 is not found [[{{node PyFunc}}]] 

While if I don’t use the “min_delta” argument, these two errors don’t appear.
I also noted, looking for examples on the Internet, that, the “min_delta” argument of the EarlyStopping callback is never set – so it is always left at its default value – when tuning.
Do you know why?

PS:
I have another question:I noticed that if I set the Hyperband “max_epochs”, for example, equal to 100, the training of the model is performed by steps:
firstly, from epoch 1 to epoch 3; secondly, from 4 to 7; then, from 8 to 13; then, from 14 to 25; then, from 26 to 50; and finally, from 51 to 100.
If I set “patience=15”, I noticed that the EarlyStopping callback stops the training right after epoch 66 (thus, the first epoch at which EarlyStopping is able to operate, because 51+15=66); could it be a coincidence, or maybe should it be the normal behavior when tuning with Keras Hyperband, or what?

I’m using keras-tuner in order to do an hyperparameter optimization of a neural network.

I’m using an Hyperband optimization, implemented by this Python function:

import keras_tuner as kt def tuning_function(self): objective = Objective('val_loss', direction="min") tuner = kt.Hyperband(ann_model, objective=objective, max_epochs=100, factor=2, directory=/path/to/folder, project_name="name", seed=0) stop_early = tf.keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0.1, mode='min', patience=15) tensorboard = TensorBoard(log_dir=log_dir) tuner.search(training_gen(), epochs=50, validation_data=valid_gen(), callbacks=[stop_early, tensorboard], steps_per_epoch=np.round(int(total_num_samples / batch_size), decimals=0), validation_freq=1, validation_steps=100) 

where ann_model is the compiled model of the Artificial Neural Network under test, while both training_gen() and valid_gen() are Python generators.

As it can be seen, Early Stopping and Tensorbord are the callbacks passed to tuner.search().

The Keras Hyperband algorithm works well: it simulates the various models, and when it reaches the Early Stopping condition on a model, the training of that model stops and the training of a new model starts; the point is that I noticed that between these two events (stop and start), the following two consecutive Python errors occur (anyway, the simulation doesn’t exit or generate an Exception, but it goes on, by simulating the successive model):

W tensorflow/core/framework/op_kernel.cc:1755] Invalid argument: ValueError: callback pyfunc_2 is not found Traceback (most recent call last): File "/home/username/anaconda3/envs/myenv/lib/python3.8/site-packages/tensorflow/python/ops/script_ops.py", line 233, in __call__ raise ValueError("callback %s is not found" % token) ValueError: callback pyfunc_2 is not found W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Invalid argument: ValueError: callback pyfunc_2 is not found Traceback (most recent call last): File "/home/username/anaconda3/envs/myenv/lib/python3.8/site-packages/tensorflow/python/ops/script_ops.py", line 233, in __call__ raise ValueError("callback %s is not found" % token) ValueError: callback pyfunc_2 is not found [[{{node PyFunc}}]] 

The first error seems to refer to a “callback”, while the second one refers to a “callback” and a “generator”.

Which could be the cause of these errors? What could be this callback “not found”?

When I load a trained model that I have previously saved, the model topology is being loaded, but none of the weights are loaded (so I have to train the model from scratch). I am very confused by this, and can find noone else who has had the same problem (here, stackoverflow, …).

I would really apreciate some help, if anyone has any idea what is going on.

​

I am saving my model as follows:

model.save('file://' + path); 

To load the model, I am using:

model = tf.loadLayersModel('file://' + path + '/model.json'); 

I am using the Object detection API, i did everything in the EXACT way the procedure is described on this site: tensorflow-object-detection-api

I am using this model: SSD ResNet50 V1 FPN 640×640 (RetinaNet50) from the Model Zoo

I am running my training on a 1070 Ti with about 8Gb of VRAM and 6,5 are available. Now i am getting this error, when i use a batch size greater than 2

2022-01-24 23:28:40.444781: E tensorflow/stream_executor/cuda/cuda_driver.cc:802] failed to alloc 4294967296 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory

For me this looks like it is trying to allocate only 4294967296 byte and i have 8589900000 byte available. So im only trying to allocate about 50%. nvidia-smi shows im using 7488MiB/8192MiB of VRAM during training(batchsize = 1). And 14,6 /16GB of RAM.

Obviously training with a batch size of 1 is useless, 8 is just doable it seems, but i dont understand why? Most people say a batch size of 64 should be possible with my hardware, please correct me.

I’m working with tf.keras on a Machine Learning project, and I’d like to implement an Attention pooling layer.

The equation which describes it is in Table I of this paper (it’s indicated at the last row of the table, at the column “Pooling function”).

The paper also says:

in the attention pooling function, the weights for each frame w_i are learned with a dedicated layer in the network.

I tried to implement the Attention pooling layer, in tf.keras, by (after reading this Keras documentation page) subclassing the Keras Layer class, such as:

from tensorflow.keras import backend as K from tensorflow.python.keras.engine.base_layer import Layer from tensorflow.python.keras.engine.input_spec import InputSpec class AttentionPooling1D(Layer): def __init__(self, axis=0, **kwargs): super(AttentionPooling1D, self).__init__(**kwargs) self.axis = axis def build(self, input_shape): input_dim = input_shape[-1] self.w= self.add_weight(shape=(1, input_dim), name='w') def get_config(self): config = {'axis': self.axis} base_config = super(AttentionPooling1D, self).get_config() return dict(list(base_config.items()) + list(config.items())) def call(self, x, mask=None): product = x * self.w numerator = K.sum(product, axis=self.axis, keepdims=True) denominator = K.sum(x, axis=self.axis, keepdims=True) attention_output = numerator / denominator return attention_output 

I don’t if it is correct or not, so I post it here in order to have feedbacks, especially if there are any errors and/or I’m missing something.