0

I'm using a LSTM (Long Short Term Memory) Network in TensorFlow with a linear layer as the final layer. I am currently using a concatenation of the input and the LSTM output as the input into the final layer. However I would like to add the states of the LSTM as predictors.
The difficulty is that tf.nn.dynamic_rnn() only produces the last state as an output, so I've resorted to using a for loop and running tf.nn.dynamic_rnn() one time step and then outputting and then running one more time step...

However I'm still getting errors with my code. Here it is. The input is some 30*238*3 tensor of [observations/batches,timesteps, predictors]: 

testinput = tf.placeholder(tf.float64, [None,238,3])
    weightin = tf.placeholder(tf.float64, [None])
    target = tf.placeholder(tf.float64, [1, None, 238])
    mask = tf.placeholder(tf.float64, [1,None,238])
    lin_weight = tf.Variable(numpy.concatenate((numpy.reshape(numpy.array([0.12504494, 0.326449906, -0.192413488]),
        (1,3,1)), numpy.random.normal(0,1/((3000000000.0)**(1/2.0)),[1,3*neurons,1])), axis = 1),dtype=tf.float64)#[0.12504494, 0.326449906, -0.192413488]
    bias = tf.Variable(1.76535047076, dtype=tf.float64, )#1.76535047076
    lstm = tf.contrib.rnn.BasicLSTMCell(neurons)
    state = lstm.zero_state(1,tf.float64)
    out1 =[0.0 for each2 in range(238)]
    a = [0.0 for each2 in range(238)]
    b = [0.0 for each2 in range(238)]
    for k in range(238):
        out1[k],state = tf.nn.dynamic_rnn(lstm, testinput, intial_state = state, sequence_length = [1], dtype=tf.float64)
        (a[k],b[k]) =  state
    print(out1)
    out1 = tf.reshape(numpy.array(out1),[-1,238,4])
    a = tf.reshape(numpy.array(a),[-1,238,4])
    b = tf.reshape(numpy.array(b),[-1,238,4])
    lineinput = tf.concat([testinput,out1,a,b], 2)
    output = tf.squeeze(tf.tensordot(lin_weight, lineinput, [[1],[2]]) + bias, [0])
    sqerror = tf.square(tf.subtract(output, target))

    masklayer  = tf.multiply(sqerror,mask)
    useweight = tf.tensordot(masklayer ,weightin,[[1],[0]])
    meansquared = tf.reduce_sum(useweight)
    #meansquared = tf.reduce_mean(tf.tensordot(tf.multiply(tf.square(tf.subtract(output, target)), mask),weightin,[[1],[0]]))
    optimizer = tf.train.AdamOptimizer()
    minimize1 = optimizer.minimize(meansquared)
    init_op = tf.global_variables_initializer()
    sess = tf.Session()
    sess.run(init_op)
    print(sess.run(a, {testinput: [xtesting[0,:,:]]}))
    batch_size = 5
    no_of_batches = int(30/batch_size)
    run = 0
    maxerror = 10000000000
    flag = True
    for each in range(10000):
        if flag:
            ptr = 0
            for j in range(no_of_batches):
                inp, out, win, maskin= xtesting[ptr:ptr+batch_size,:,:], ytesting[:,ptr:ptr+batch_size,:], weights2[ptr:ptr+batch_size], bmask[:,ptr:ptr+batch_size,:]
                ptr+=batch_size
                sess.run(minimize1, {testinput: inp, target: out, weightin: win, mask: maskin})
            validerror = sess.run(meansquared, {testinput: xtesting, target: ytesting, weightin: weights2, mask: bmaskvalidate})
            print(sess.run(meansquared, {testinput: xtesting, target: ytesting, weightin: weights2, mask: bmask}))
            print(validerror)
            print(each)
            if  validerror < maxerror:
                run = each
                maxerror = validerror
            if each > run + 25:
                flag = False
    print(sess.run(output, {testinput: [xtesting[0,:,:]]}))
    print(sess.run(meansquared, {testinput: [xtesting[0,:,:]], target: [ytesting[:,0,:]], weightin: [weights2[0]], mask: [bmask2[:,0,:]]})/24.0)

The error I get is: TypeError: Expected binary or unicode string, got < tf.Tensor 'rnn/transpose:0' shape=(?, 238, 4) dtype=float64 >. The error is caused from this line out1 = tf.reshape(numpy.array(out1),[-1,238,4]).

cfen
  • 95
  • 1
  • 10
  • What is the error you get? Also `tf.nn.dynamic_rnn()` should produce all the states in `outputs` and the last state in `state`. – pfm Jul 28 '17 at 13:41
  • I edited my text. In my problem though I don't want `state` to be just the last state but all the states. Thus I only use `tf.nn.dynamic_rnn()` for one time step at a time. It's a hack way to do this and I'm wondering if there is a better way. – cfen Jul 28 '17 at 19:28
  • My point is that all the states should be available in the `output` tensor. – pfm Jul 29 '17 at 06:50
  • See for example https://stackoverflow.com/a/44704934/4282745 – pfm Jul 29 '17 at 06:52
  • So the main way to do this is to go into source and make it output the c state? – cfen Jul 29 '17 at 19:13

0 Answers0