How do i load part of the data set

[John6666]

I wonder if streaming would be a good option…

Stream

Best practices for a large dataset

Hi, I have a ~1TB large dataset stored at the HF hub. I can download this on my disk and read it successfully. Nevertheless, it’s large enough that I can’t fit it in my RAM. What is the best practice for training a model on such a dataset? I tried loading the dataset with load_dataset(..., streaming=True) and then having two buffers: one that is being loaded by training process onto GPU and one that, in a separate thread, streams from the dataset and fills up the other buffer. Then, when the i…

How to load a large hf dataset efficiently?

It’s generally a good idea indeed if you want to save disk space or to not have to wait to download the full dataset. For example you can stream the dataset using the datasets library, by passing streaming=True to load_dataset(). However even in streaming mode, you better have multiple shards in order to do parallel streaming. And ideally use file formats that work well with streaming like WebDataset or Parquet. @aaditya did try streaming mode on this dataset but the custom loading script of …