Categories
Misc

How can I train a model on a HUGE dataset?

So I have a huge dataset that devours my 32GB memory and then
crashes every time before I can even begin training. Is it possible
to break the dataset into chunks and train my model that way?

I’m fairly new to tensorflow so I’m not sure how to go about it.
Can anyone help?

Thank you.

EDIT: the data is time series data (from a csv) that I’m loading
into a pandas dataframe. From there, the data is being broken up
into samples with a 10 step window. I have about 90M samples with
the shape (90M, 10, 1) that should then be fed into the LSTM. The
problem is that the samples crash the RAM and I have to start all
over again each time.

submitted by /u/dsm88

[visit reddit]

[comments]

Leave a Reply

Your email address will not be published. Required fields are marked *