Hi Vardaan,

Great question and that’s actually the other benefit you have with HDF5. I will update my post with that.

You don’t need to load the entire HDF4 file to memory. You can load just a slice from your dataset at a time (similar to numpy slicing).

For example if you write,

train_dataset[1:1000,:,:,:]

it will load only the first 1000 rows of the HDF5 file.