I have a dataset (<class ‘tensorflow.python.data.ops.dataset_ops.MapDataset’>) which is a result of some neural network output. In order to get my final prediction, I am currently iterating over it as follows:
for row in dataset: ap_distance, an_distance = row y_pred.append(int(ap_distance.numpy() > an_distance.numpy()))
The dataset has two columns, each holding a scalar wrapped in a tensor.
The loop body is very simple, it takes < 1e-5 seconds to compute. Sadly, one iteration takes ca. 0.3 seconds, so fetching the next row from the dataset seems to take almost 0.3s! This behavior is very weird given my hardware (running on a rented GPU server with 16 AMD EPYC cores and 258GB RAM) and the fact that a colleague on his laptop can finish an iteration in 1-2 orders of magnitude less time than I can. The dataset has ca. 60k rows, hence it is unacceptable to wait for so long.
What I tried:
I tried mapping the above loop body onto every row of the dataset object, but sadly .numpy() is not available inside dataset.map()!
- Why does it take so long to get a new row from the dataset?
- How can I fix this performance degradation?