Category: Misc

I have replicated an architecture from one of the research papers and running it on GPU gives Out Of Memory even on colab (The model is quite deep and huge). So naturally, I want to train it using a TPU.

The same code using GPU doesn’t cause any issue. However it throws this error if I train on TPU:

InvalidArgumentError: 9 root error(s) found. (0) INVALID_ARGUMENT: {{function_node __inference_train_function_56959}} Reshape’s input dynamic dimension is decomposed into multiple output dynamic dimensions, but the constraint is ambiguous and XLA can’t infer the output dimension %reshape.8395 = f32[3,3,2,86,86,32]{5,4,3,2,1,0} reshape(f32[<=18,86,86,32]{3,2,1,0} %convolution.8393), metadata={op_type=”BatchToSpaceND” op_name=”model/conv2d_3/Conv2D/BatchToSpaceND”}.

[[{{node TPUReplicate/_compile/_11021259981217135469/_4}}]]

The colab notebook can be found here:

https://colab.research.google.com/drive/1D-laydrWwLnqAehVREhSkTPoqNXyRGS8?usp=sharing

Hi everyone. I’m trying to make tensorflow use my NVIDIA GTX 1060 gpu in my laptop. I created a python environment and installed tensorflow, python, pip, etc. I am using Ubuntu on Windows (so wsl-ubuntu). On CMD, the nvidia-smi command is showing my GPU. But with tensorflow, I get the following error:

2022-01-26 21:45:36.677191: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected 2022-01-26 21:45:36.678074: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (DESKTOP-P8QAQC0): /proc/driver/nvidia/version does not exist Num GPUs Available: 0 

I have CUDA 11.5 and 11.6 installed, with cudNN 8.3.2.44 installed. I manually copied and pasted the files into the CUDA directory and ran the exe (exe didn’t seem to install files though). I am not sure what else to do. Help would be really appreciated!

Hey all!

I have trained a TensorFlow modal with faces of people I want to detect.

While it detects the people and gives the correct label to the faces, I trained the model to detect. If I point the webcam at a face that I did not train the model with, it gives a label of one of the people I trained the model with.

I’ve tried many things to stop this, but nothing has worked.

I can share all the code and faces I am trying to detect if needed, but is there any way to stop this?

Any advice is greatly appreciated! I’m still learning TensorFlow and while I’m a little better than my pervious posts I’m still learning!

Thanks!

I’m using keras-tuner to perform a hyperparameter optimization of a neural network.

I’m using a Hyperband optimization, and I call the search method as:

import keras_tuner as kt tuner = kt.Hyperband(ann_model, objective=Objective('val_loss', direction="min"), max_epochs=100, factor=2, directory=/path/to/folder, project_name="project_name", seed=0) tuner.search(training_gen(), epochs=50, validation_data=valid_gen(), callbacks=[stop_early], steps_per_epoch=1000, validation_freq=1, validation_steps=100) 

where the EarlyStopping callback is defined as:

stop_early = tf.keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0.1, mode='min', patience=15) 

Hyperband initially trains many models (each one with a different combination of the hyperparameters previously chosen) for only 2 epochs; then, it discards poor performing models and it only trains the most promising ones, step by step, with an increasing number of epochs at each step (the final goal is to discard all models except one, the best perfoming one).

So the training of a specific model is not performed in one shot, but it’s perfomed by steps, where in each of them Keras saves the state of the training.

By setting max_epochs=100, I noticed that the training of a model is performed by these steps (called “Runnning trials“):

1. firstly, from epoch 1 to epoch 3;
2. secondly, from 4 to 7;
3. then, from 8 to 13;
4. then, from 14 to 25;
5. then, from 26 to 50;
6. and finally, from 51 to 100.

So, at the end of each “Running trial”, Keras saves the state of the training, in order to continue, at the next “Running trial”, the training from that state.

By setting patience=15: during “Runnning trials” 1), 2), 3), 4) of the list above, EarlyStopping could not operate because the number of training epochs is less than patience; thus, EarlyStopping could operate only during “Running trials” 5) and 6) of the list above.

Initially I thought that the patience count started at epoch 1 and should never reset itself when a new “Running trial” begins, but I noticed that the EarlyStopping callback stops the training at epoch 41, thus during the “Running trial” 5), which goes from epoch 26 to 50 .
Thus it seems to me that, at the beginning of each “Running trial”, patience count is reset; indeed: EarlyStopping arrests the training at epoch 41, the first epoch at which EarlyStopping is able to operate, because: start_epoch + patience = 26 + 15 = 41..

Is it normal/expected behavior that patience is automatically reset at the beginning of each “Running trial” while using Keras Hyperband tuning?