Race Condition when reading from generator

Question

I am running a python program on Python2.7. Wherein I am trying to print the value of a generator using the following approach.

y = m.predict(input_fn=lambda:input_fn(df_predict), as_iterable=True)  
#output of this is a generator    
print(type(y))
print(list(y))

The following error is displayed:

Traceback (most recent call last):
  File "tensorflow/examples/wide_n_deep_model_predictions_v3.0.py", line 176, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 43, in run
    sys.exit(main(sys.argv[:1] + flags_passthrough))
  File "tensorflow/examples/wide_n_deep_model_predictions_v3.0.py", line 171, in main
    train_and_eval()
  File "tensorflow/examples/wide_n_deep_model_predictions_v3.0.py", line 167, in train_and_eval
    print (list(y))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/dnn_linear_combined.py", line 328, in _as_iterable
    for pred in preds:
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 909, in _infer_model_as_iterable
    restore_checkpoint_path=checkpoint_path):
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/graph_actions.py", line 867, in run_feeds_iter
    yield session.run(output_dict, f)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 766, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 964, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1014, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1034, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: WhereOp: Race condition between counting the number of true elements and writing them.  When counting, saw 2645 elements; but when writing their indices, saw 19 elements.
         [[Node: linear/linear/ip/ip_weights/Where = Where[_device="/job:localhost/replica:0/task:0/cpu:0"](linear/linear/ip/ip_weights/GreaterEqual/_409)]]

Caused by op u'linear/linear/ip/ip_weights/Where', defined at:
  File "tensorflow/examples/wide_n_deep_model_predictions_v3.0.py", line 176, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 43, in run
    sys.exit(main(sys.argv[:1] + flags_passthrough))
  File "tensorflow/examples/wide_n_deep_model_predictions_v3.0.py", line 171, in main
    train_and_eval()
  File "tensorflow/examples/wide_n_deep_model_predictions_v3.0.py", line 163, in train_and_eval
    y = m.predict(input_fn=lambda:input_fn(df_predict), as_iterable=True)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 245, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/dnn_linear_combined.py", line 747, in predict
    as_iterable=as_iterable)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 191, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 477, in predict
    as_iterable=as_iterable)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 847, in _infer_model
    infer_ops = self._get_predict_ops(features)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 1113, in _get_predict_ops
    return self._call_model_fn(features, labels, ModeKeys.INFER)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 1026, in _call_model_fn
    params=self.params)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/dnn_linear_combined.py", line 504, in _dnn_linear_combined_model_fn
    scope=scope)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/feature_column_ops.py", line 535, in weighted_sum_from_feature_columns
    weight_collections)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/feature_column_ops.py", line 332, in _create_embedding_lookup
    name=column.name + '_weights')
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/embedding_ops.py", line 123, in safe_embedding_lookup_sparse
    sparse_ids, sparse_weights = _prune_invalid_ids(sparse_ids, sparse_weights)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/embedding_ops.py", line 169, in _prune_invalid_ids
    sparse_ids = sparse_ops.sparse_retain(sparse_ids, is_id_valid)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/sparse_ops.py", line 880, in sparse_retain
    where_true = array_ops.reshape(array_ops.where(to_retain), [-1])
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.py", line 2501, in where
    return gen_array_ops.where(input=condition, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 3821, in where
    result = _op_def_lib.apply_op("Where", input=input, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2259, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1130, in __init__
    self._traceback = _extract_stack()
InvalidArgumentError (see above for traceback): WhereOp: Race condition between counting the number of true elements and writing them.  When counting, saw 2645 elements; but when writing their indices, saw 19 elements.
         [[Node: linear/linear/ip/ip_weights/Where = Where[_device="/job:localhost/replica:0/task:0/cpu:0"](linear/linear/ip/ip_weights/GreaterEqual/_409)]]

On running print ('Predictions: {}'.format(str(y))) the output is Predictions: <generator object _as_iterable at 0x7f1883235550>

Can someone guide me how can I print the generator?

have you tried `print(x)` ? Sounds like you might be on python 3 — Patrick Haugh, Nov 16 '16 at 18:23
I am running it on Python2.7. The result of print(x) breaks the program causing the following error `InvalidArgumentError (see above for traceback): WhereOp: Race condition between counting the number of true elements and writing them. When counting, saw 2037 elements; but when writing their indices, saw 19 elements. [[Node: linear/linear/device/device_weights/Where = Where[_device="/job:localhost/replica:0/task:0/cpu:0"](linear/linear/device/device_weights/GreaterEqual/_297)]]` — Vasanti, Nov 16 '16 at 18:43
On running the command `y = m.predict(input_fn=lambda:input_fn(df_predict), as_iterable=True)` and checking the type of `y`, it returns `generator`. So the object y is a generator object correct. I need to print the value that is stored in the generator object. — Vasanti, Nov 16 '16 at 19:04
On running `print ('Predictions: {}'.format(str(y)))` the output is `Predictions: ` — Vasanti, Nov 16 '16 at 19:05
Could you post the full traceback of the error you're getting as an edit to your question? — Patrick Haugh, Nov 16 '16 at 19:09
This is the only line of error that is printed and I have added it to the original post as well — Vasanti, Nov 16 '16 at 19:10
I am sorry, where is that mentioned? On running the command `print ('Predictions: {}'.format(str(y)))` there is no error, it just prints `Predictions: `. However the expected output must be [1 0 0 1 1 0 0] — Vasanti, Nov 16 '16 at 19:13
It still throws the same error `InvalidArgumentError (see above for traceback): WhereOp: Race condition between counting the number of true elements and writing them. When counting, saw 2645 elements; but when writing their indices, saw 19 elements. [[Node: linear/linear/ip/ip_weights/Where = Where[_device="/job:localhost/replica:0/task:0/cpu:0"](linear/linear/ip/ip_weights/GreaterEqual/_409)]]` — Vasanti, Nov 16 '16 at 19:20
I apologise for the misunderstanding. I have added the entire stack trace in the original question. — Vasanti, Nov 16 '16 at 19:21
You might have to post this as a new question. The problem you're getting isn't really anything to do with printing, but your generator is changing as it's being iterated over. I suggest you reask the question including that traceback and with a title like `Race Condition when when reading from generator.` — Patrick Haugh, Nov 16 '16 at 19:27
Sure, would do that. Thank you for your help, appreciate it. — Vasanti, Nov 16 '16 at 19:28
Since the question is not a duplicate could you please edit that? — Vasanti, Nov 16 '16 at 19:31
I cast a reopen vote, but at least 4 others (or a moderator) need to do the same to reopen your question. — Patrick Haugh, Nov 16 '16 at 19:33

score 0 · Accepted Answer · answered Dec 16 '16 at 09:45

I think you need to do something like this

# Print out predictions
y = regressor.predict(input_fn=lambda: input_fn(prediction_set))

# .predict() returns an iterator; convert to a list and print predictions
predictions = list(itertools.islice(y, 6))
print("Predictions: {}".format(str(predictions)))

I found it from the boston tutorial: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/input_fn/boston.py

Race Condition when reading from generator

1 Answers1