0

I have been looking at the high-level estimator interface in Tensorflow, walked through fairly well in the wide_n_deep tutorial. It doesn't seem to allow streaming input, which I think I require for a training set that doesn't fit in memory.

Does the high-level API support this? I was reading the source, and I can't quite tell. It looks like maybe I could write an input function that had generators instead of arrays, but maybe the code precludes that.

P.S. Sort of related to this question, but I want to stick to the high-level API if I could.

dfrankow
  • 20,191
  • 41
  • 152
  • 214

1 Answers1

1

You can certainly train data that does not fit into memory with TensorFlow using high-level APIs. Just use the Dataset API. You can search for: "The Dataset API supports a variety of file formats so that you can process large datasets that do not fit in memory" in that page. If you want to use Datasets with Estimators, search for "input_fn" on that page.

iga
  • 3,571
  • 1
  • 12
  • 22