2

I am creating a tensorflow constant based on a ndarray list object. My understanding was that tensor itself wouldnt do a memory copy of the underlying data, but create a python object using the same underlying ndarray data. However, after running a little test, seems like it does copy the data 

def mem_test():
  printMemUsed("before r list")
  r = ['Supporter'] * 100000000
  printMemUsed("after r list")
  r_arr = np.array(r)
  printMemUsed("after nd_array")
  tf.convert_to_tensor(r_arr)
  printMemUsed("after tensor conversion")

def printMemUsed(discript):
    print("{}:\t{}".format(discript, psutil.virtual_memory().used))

Here's the ouput:

before r list:  727310336 -> 727 Mb
after r list:   1528782848 -> 1.5 GB
after nd_array: 2430574592 -> 2.4 GB
after tensor conversion:        8925667328 -> 8.9 GB

edit: r_arr had a dtype of 'S9' (null terminated string). After changing the input array element to type 'unicode' (U9), the virtual memory consumption bumped up to 5 GB after nd_array

cyrux
  • 233
  • 5
  • 15
  • What's the `dtype` of `r_arr` and of the result of the `convert`? `convert_to_tensor` doesn't say anything about copy or not; where did you read about it? – hpaulj Nov 28 '17 at 01:23
  • dtype of r_arr is S9 and dtype of tensorflow is constant. I read that tensorflow doesnt copy data but just creates a wrapper / data structure around the underlying data – cyrux Nov 28 '17 at 19:29

1 Answers1

1

tf.convert_to_tensor uses a _tensor_conversion_func_registry were conversions functions are registered for different input types. For Python list, tuple and numpy nd.arrays this happens in constant_op.py. And according to this answer tf.constant creates at least two copies.

Why is the value of a tf.constant() stored multiple times in memory?

Because data for a constant tensor is embedded into graph definition. This means this data is stored both in the client, which maintains the graph definition, and in the runtime, which allocates it's own memory for all tensors:

Edit:

According to this answer one option is to use tf.Variable and initialize it with a tf.placeholder.

Tillmann Radmer
  • 139
  • 2
  • 8