Month: December 2021

I have a data of 10 Gb in ram and the shape of data is (10000,3000,61). I am training a lstm model. But when I start training with batch size of 10242 the ram usage reach upto 50 GB, but GPU memory is half consumed. If I further increase the batch size , the system get hanged because of 100% ram consumption. Simply I am wasting 50% of my gpu capacity, because with a batch size of 10242, only 5.8 gb gpu is used and I have 12 gb gpu.

hi everyone, i was looking for a book / site / someone to explain me how to train a model for tensorflow lite. In particular I can’t figure out how to create a custom model through TensorFlow Lite Model Maker. Can anyone give me some advice?

I want to set up distributed RL. Multiple worker and 1 learner. I have 1 GPU and 1 CPU with multiple cores.

So GPU:0 and CPU:0.

Now if i start the programm normal via python programm.py

It detects the GPU and CPU and lists it once i call tf.config.list_physical_device()

However if i start it as MPI application via:

mpiexec -np 4 python programm.py

Every process just lists CPU:0 and no gpu is detected. How can i make at least one process see the gpu?

I use tf2.7 and mpi4py

Here is the full code. This currently works just fine on my M1 MacBook running Monterey and Tensorflow-Metal. However, when I export the dataset and code to my laptop with an RTX 3060 Laptop GPU with Pop_OS! that is when I start getting the [UNK] characters generated and “NaN” loss. I’m unsure of what steps to take to make this better. Any advice would be appreciated.

import os, sys, time import numpy as np import tensorflow as tf from tensorflow.keras.models import Sequential from tensorflow.keras.callbacks import ModelCheckpoint from tensorflow.keras.optimizers import Adam from tensorflow.keras.preprocessing.sequence import pad_sequences from tensorflow.keras.preprocessing.text import Tokenizer from tensorflow.keras.layers import StringLookup from tensorflow.keras.layers import Embedding from tensorflow.keras.layers import Bidirectional from tensorflow.keras.layers import SimpleRNN from tensorflow.keras.layers import Dense BATCH_SIZE = 128 BUFFER_SIZE = 10_000 EMBEDDING_DIMENSION = 128 RNN_UNITS = 1024 CHECKPOINT_DIR = './training_checkpoints' CHECKPOINT_PREFIX = os.path.join(CHECKPOINT_DIR, "ckpt_{epoch}") EPOCHS = 16 def text_from_ids(ids): return tf.strings.reduce_join(chars_from_ids(ids), axis=1) def split_input_target(sequence): input_text = sequence[:-1] target_text = sequence[1:] return input_text, target_text def generate_text(model, seed_text, next_words, max_sequence_len): for _ in range(next_words): token_list = Tokenizer().texts_to_sequences([seed_text])[0] token_list = pad_sequences([token_list], maxlen=max_sequence_len-1, padding='pre') predicted = model.predict(token_list, verbose=0) output_word = "" for word,index in Tokenizer().word_index.items(): if index == predicted: output_word = word break seed_text += " "+output_word return seed_text.title() def generate_char(inputs): input_ids = tf.convert_to_tensor(ids_from_chars(inputs)) predicted_logits = model(inputs=np.array([input_ids])) predicted_logits = predicted_logits[:, -1, :] # print(predicted_logits) predicted_logits = predicted_logits/1.0 # print(predicted_logits) predicted_ids = tf.random.categorical(predicted_logits, num_samples=1) predicted_ids = tf.squeeze(predicted_ids, axis=-1) return chars_from_ids(predicted_ids) text = open("./data.txt", "rb").read().decode(encoding="UTF-8") vocab = sorted(set(text)) vocab_size = len(vocab) print(f"Text Length: {len(text)}") print(f"Text Vocab: {vocab}") print(f"Text Vocab Size: {vocab_size}") ids_from_chars = StringLookup(vocabulary=list(vocab), mask_token=None, name='lookup') chars_from_ids = StringLookup(vocabulary=ids_from_chars.get_vocabulary(), invert=True, mask_token=None) all_ids = ids_from_chars(tf.strings.unicode_split(text, "UTF-8")) ids_dataset = tf.data.Dataset.from_tensor_slices(all_ids) sequence_length = 100 examples_per_epoch = len(text)//(sequence_length+1) sequences = ids_dataset.batch(sequence_length+1, drop_remainder=True) dataset = sequences.map(split_input_target) dataset = ( dataset.shuffle(BUFFER_SIZE) .batch(BATCH_SIZE, drop_remainder=True) .prefetch(tf.data.experimental.AUTOTUNE) ) model = Sequential() model.add(Embedding(vocab_size, EMBEDDING_DIMENSION, batch_input_shape=[BATCH_SIZE, None])) model.add(SimpleRNN(RNN_UNITS, return_sequences=True)) model.add(Dense(vocab_size,)) checkpoint_callback = ModelCheckpoint( filepath=CHECKPOINT_PREFIX, save_weights_only=False, save_best_only=False, verbose=1 ) loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True) model.compile(loss = loss, optimizer='adam', run_eagerly=True) model.summary() model.fit(dataset, batch_size=BATCH_SIZE, epochs=EPOCHS, callbacks=[checkpoint_callback]) model.save("./model/") model = tf.keras.models.load_model("./model/") next_char = tf.constant(["After "]) result = [] for n in range(256): next_char = generate_char(next_char) result.append(next_char) print(tf.strings.join(result)[0].numpy().decode("utf-8")) 

For the record, I’m absolutely expecting to have made some obvious error here, but I can’t seem to find it myself.

For context, I have implemented a very simple version of a game similar to doodle jump, and among other things I have attempted to train a FFNN to predict the correct / best inputs based on the state of the game. I have generated ~15k examples through manual play, each frame having a 25% of being recorded for both state and inputs, and for a slight bit of dataset balancing, half of all frames without any input are discarded.

I’m using a sequential model with 3 dense layers, using 15, 5 and 3 units respectively. I’ve specified a sigmoid activation for the input and hidden layers, and the input shape for the input layer.

For each example, the input consists of 15 scalar values (7 pairs of values representing the distance of a platform to the player sprite in horizontal and vertical direction respectively, plus 1 value for the currently remaining timer. (I’m using a 20 second timer to make comparisons reasonable.)), while the labels consist of 3 integers, each either 1 or 0, representing whether the key associated with that position has been pressed on that frame or not. (Left, Right, Up in that order.) The model compilation specifies the use of the Adam optimzer and MeanSquaredError loss.

What I’m specifically hoping to predict is a set of 3 values, which I can check against a set threshold to determine wheter the associated key should be pressed on that frame.

When training the model, however, I’m seeing no relevant drop in loss over 100 epochs, (most recently it went from 0.1923 to 0.1901), and indeed the trained models behaviour will consistently see it pressin right and jump, with the value for the left key often being negative, which on the one hand seems to indicate extreme underfitting (Since it’s predicting negative values when all example labels were positive), but on the other the sheer regularity with which this occurs might point to an error in my methodology.

I realise that any answer to this will be speculative at best, and that’s absolutely fine. I’ve tried everything I can think of (Varying number and size of the layers, varying the optimizer and/or loss function, varying the threshold, deleting and rebuilding the dataset…), so any ideas wouldbe very welcome.

As the title is self-descriptive, I’m getting a dramatic accuracy difference for the same model when I deploy it on MATLAB compared to TensorFlow & Keras. My model is actually a basic one, which uses transfer technique to fine-tune a pre-trained model, namely, ResNet50, by excluding the top and adding a Dense layer to utilize it for another classification task. I’ve added the sample code that shows the architecture of the model below. The model was trained under the same hyper-parameters on both platforms. The accuracy values I get on TensorFlow and MATLAB are 0.7455, and 0.424, respectively, on the CIFAR-10. What could be the reason behind this great accuracy difference? Could you please help me?

​

Model Architecture:

base_model = ResNet50(include_top=False, weights=’imagenet’, input_shape=(75, 75, 3),pooling=’max’, classes=10)

base_model.trainable = False

model = Sequential()

model.add(base_model)

model.add(Dense(10, activation=’softmax’))