How do I save a Huggingface dataset?

Question

How do I write a HuggingFace dataset to disk?

I have made my own HuggingFace dataset using a JSONL file:

Dataset({ features: ['id', 'text'], num_rows: 18 })

I would like to persist the dataset to disk.

Is there a preferred way to do this? Or, is the only option to use a general purpose library like joblib or pickle?

Might help: https://discuss.huggingface.co/t/saving-dataset-in-the-current-state-without-cache/5892 — Bijay Regmi, Apr 27 '22 at 00:04

score 15 · Accepted Answer · edited Jul 13 '22 at 16:32

15

You can save a HuggingFace dataset to disk using the save_to_disk() method.

For example:

from datasets import load_dataset
  
test_dataset = load_dataset("json", data_files="test.json", split="train")

test_dataset.save_to_disk("test.hf")

edited Jul 13 '22 at 16:32

Timbus Calin

answered Apr 27 '22 at 00:09

2

Hey! This gets saved as an arrow file. Can I save it as a choice of my own? Can I view it as a normal text file? – DIVYA RATHOD Oct 08 '22 at 10:39

score 2 · Answer 2 · answered Nov 20 '22 at 12:35

You can save the dataset in any format you like using the to_ function. See the following snippet as an example:

from datasets import load_dataset
dataset = load_dataset("squad")
for split, dataset in dataset.items():
    dataset.to_json(f"squad-{split}.jsonl")

2 Answers2