0

i have a problem with data from Numpy arrays and conversion to tf.data, can someone help me. i have a small example, instead of my data, i took random numbers here.

My model has two inputs, which become concatenate, and one result: Label,

import tensorflow as tf 
import numpy as np


input = []
label = []
for i in range(1):
    input1 = np.random.rand(9)
    input2 = np.random.rand(2,19)
    label1  = np.random.rand(4)
    input.append([input1 , input2])
    label.append(label1) 

dataset = tf.data.Dataset.from_tensor_slices((input, label))

The Result is: ValueError: Can't convert non-rectangular Python sequence to Tensor.

muellerelias
  • 77
  • 8

2 Answers2

0

This simply means that your input shapes don't match. Your three sources have different number of rows and columns. Just print them and look:

[array([0.13144276, 0.98472215, 0.99699807, 0.7377246 , 0.16636525,
        0.92194944, 0.1244091 , 0.6967843 , 0.58724495]),
 array([[0.61948976, 0.05685026, 0.87393599, 0.76267412, 0.20453778,
         0.55072301, 0.66798436, 0.14033839, 0.97168137, 0.10027466,
         0.48620577, 0.40916652, 0.1613872 , 0.21072357, 0.51055702,
         0.56713161, 0.21771354, 0.65349586, 0.99575239],
        [0.35485898, 0.8807598 , 0.67915537, 0.10144362, 0.80200975,
         0.69236555, 0.56384801, 0.20503875, 0.44132446, 0.09489097,
         0.95682325, 0.20710026, 0.67398689, 0.74433233, 0.105801  ,
         0.59680707, 0.95792024, 0.83041166, 0.58450471]])]

[array([0.11237042, 0.0468774 , 0.93132205, 0.67879769])]

It would work if you had a dataset with both the same number of columns for input, and the same number of rows between input and label:

input = []
label = []
for i in range(1):
    input1 = np.random.rand(1, 9)
    input2 = np.random.rand(1,9)
    label1  = np.random.rand(2, 4)
    input.append([input1 , input2])
    label.append(label1)

dataset = tf.data.Dataset.from_tensor_slices((input, label))
Nicolas Gervais
  • 33,817
  • 13
  • 115
  • 143
  • okay thanks, is there an other solution, how i can slice it in batches before i do them in model.fit, because all data is to much for my Computer. – muellerelias Jul 19 '20 at 17:12
  • i don't see how your second question is related. for the first, the solution is to have matching number of rows. that's not unique to tensorflow, every learning algorithm needs that. how did you end up with such a shape anyway? – Nicolas Gervais Jul 19 '20 at 17:35
  • I have a net with two inputs, one with matrix input and one as a vector, which are then joined together in the middle of the net. It works well with Numpy. Only if I give more than 10,000 data as input, it takes forever in the beginning until he starts to train. And I wanted to prepare the data in pieces. You can do that with tf.data. – muellerelias Jul 19 '20 at 18:21
0

With the help of the answer from above, I now knew how to google correctly. Here is a solution for multi input datasets. Reference #52582275

muellerelias
  • 77
  • 8