1

My objective is to save tensorflow dataset objects to disk and then read them on a different machine.

Here's the source code to get dataset:

import tensorflow_datasets as tfds

datasets, info = tfds.load("imdb_reviews", as_supervised=True, with_info=True)

Now how do I save datasets and info to location such as './'? Also, this will be saved on Linux machine and read on Windows machine. So, I'd want the format to be platform independent.

I did try doing this myself using the link https://github.com/tensorflow/tensorflow/issues/38483:

def save(dataset, location='data/tf-records/'):
    dataset = dataset.map(tf.io.serialize_tensor)
    writer = tf.data.experimental.TFRecordWriter(location)
    writer.write(dataset)
    return location


def load(tf_record='data/tf-records/'):
    dataset = tf.data.TFRecordDataset(tf_record)
    dataset = dataset.map(lambda x: tf.io.parse_tensor(x, tf.int64))
    return dataset

However, when I run this code, I get the following error:

AttributeError: 'dict' object has no attribute 'map'

It has been a couple of days since I started working on tensorflow. So, I am not too sure how to fix these. I am a beginner.

Thanks for any help.

I have tensorflow 2.3 on Python 3.7.9

watchtower
  • 4,140
  • 14
  • 50
  • 92
  • This looks like a job for the pickle library, which can save an instance of virtually any object to a file, to be read later. – SimonR Dec 26 '20 at 23:42
  • @SimonR: Thank you! I tried it, here's what I got `import pickle;pickle.dump(datasets, open( "save.p", "wb" ) )`. I am getting this error `InternalError: Tensorflow type 21 not convertible to numpy dtype.` Any thoughts? – watchtower Dec 26 '20 at 23:53
  • I did a bit of googling on that error, came across this which explains why pickling doesnt work. https://stackoverflow.com/questions/56862492/how-to-save-tf-data-dataset-object It does suggest an alternative though. – SimonR Dec 26 '20 at 23:57
  • @SimonR: Thanks for sharing. I don't think that thread talks about saving dataset object to disk. They are talking about optimizing results. I am a beginner. That's what I could understand from that thread... – watchtower Dec 28 '20 at 18:24

0 Answers0