Categories
Misc

Strange behavior of "min_delta" in EarlyStopping callback while tuning

I’m using keras-tuner in order to do an hyperparameter optimization of a neural network.

I’m using an Hyperband optimization, and I call the search method as:

tuner.search(training_gen(), epochs=50, validation_data=valid_gen(), callbacks=[stop_early], steps_per_epoch=np.round(int(num_samples / batch_size), decimals=0), validation_freq=1, validation_steps=100) 

where the EarlyStopping callback is defined as:

stop_early = tf.keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0.1, mode='min', patience=15) 

The Keras Hyperband algorithm seems to work well: it simulates the various models, and when it reaches the Early Stopping condition on a model, the training of that model stops and the training of a new model starts.
What I noticed is that with this EarlyStopping callback implementation, when EarlyStopping stops the training of a model, before starting the training of the next model, the following two consecutive Python errors occur (anyway, the simulation doesn’t exit or generate exceptions, but it goes on, by simulating the successive model):

W tensorflow/core/framework/op_kernel.cc:1755] Invalid argument: ValueError: callback pyfunc_2 is not found Traceback (most recent call last): File "/home/username/anaconda3/envs/myenv/lib/python3.8/site-packages/tensorflow/python/ops/script_ops.py", line 233, in __call__ raise ValueError("callback %s is not found" % token) ValueError: callback pyfunc_2 is not found W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Invalid argument: ValueError: callback pyfunc_2 is not found Traceback (most recent call last): File "/home/username/anaconda3/envs/myenv/lib/python3.8/site-packages/tensorflow/python/ops/script_ops.py", line 233, in __call__ raise ValueError("callback %s is not found" % token) ValueError: callback pyfunc_2 is not found [[{{node PyFunc}}]] 

While if I don’t use the “min_delta” argument, these two errors don’t appear.
I also noted, looking for examples on the Internet, that, the “min_delta” argument of the EarlyStopping callback is never set – so it is always left at its default value – when tuning.
Do you know why?

PS:
I have another question:I noticed that if I set the Hyperband “max_epochs”, for example, equal to 100, the training of the model is performed by steps:
firstly, from epoch 1 to epoch 3; secondly, from 4 to 7; then, from 8 to 13; then, from 14 to 25; then, from 26 to 50; and finally, from 51 to 100.
If I set “patience=15”, I noticed that the EarlyStopping callback stops the training right after epoch 66 (thus, the first epoch at which EarlyStopping is able to operate, because 51+15=66); could it be a coincidence, or maybe should it be the normal behavior when tuning with Keras Hyperband, or what?

submitted by /u/RainbowRedditForum
[visit reddit] [comments]

Leave a Reply

Your email address will not be published. Required fields are marked *