-1

Be kind, I'm new to TensorFlow. I have found a project that trains a Policy Gradient agent to trade the stock market, trained on only the daily Close prices. And I'm interested in making it train on the Open, High, Low, and Volume features as well, so I'm attempting to add them in to the existing code. I've stripped away most of the project to leave only what's necessary for diagnosis.

You can find my Colab notebook here. And I have done my best to comment each section to make it easier to browse, and where I THINK the issues are, but I need someone to show me where and why.

I'm currently getting the error:

ValueError: Cannot feed value of shape (30, 5) for Tensor 'Placeholder:0', which has shape '(?, 30)'

...which makes sense because the self.X = tf.compat.v1.placeholder(tf.float32, (None, self.state_size)) is designed for the only feature (Close), but I'm trying to add in the other features into the state as well. The state_size is the window_size of 30 (to look back 30 rows during training). And I'm trying to change the state to include the added features. So I try to change the placeholder to self.state_size,5, but then I get the error:

ValueError: Cannot feed value of shape (35760, 5) for Tensor 'Placeholder:0', which has shape '(30, 5)'

...which I'm a little unclear about, but I don't think that's the issue. I know I'm trying to feed the tensor data in a shape that it's not expecting, but I don't know how to adapt this on my own. (I think) What I'm looking to do is add in those extra features into the get_state function so that each row is 1 window_size, and each column represents the features. Then the training should take place over the iterations.

I've found similar questions/answers at the below links to help someone who knows more about this than me out. Most of them talk about reshaping the data at the placeholder, which I thought I had tried, but now I've just outrun my knowledge. Thanks in advance.

Here

Here

Here

Here

UPDATE

Boy I'm sure struggling to figure this out on my own, but I appreciate the guidance thus far and I think I'm getting close, I just have limited knowledge of what to change. Given the answer below, I understand that my window_size/number of rows can change depending on what the window_size/lookback value is, so that would be the None part of my the placeholder, and in this particular case, I WOULD know the number of features (in this case 5) ahead of time so setting that as a static number (5) would suffice. So I'm trying to not have individual placeholders for each new feature I want to add.

So trying to use the new placeholder self.X = tf.compat.v1.placeholder(tf.float32, (None,5)) now the error I'm getting is:

    InvalidArgumentError: Incompatible shapes: [3270,3] vs. [98100,3]
     [[{{node sub}}]]

During handling of the above exception, another exception occurred:

InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-4-78f4afe280e1> in <module>()
     22          skip = skip)
     23 
---> 24 agent.train(iterations = 500, checkpoint = 10, initial_money = initial_money)

<ipython-input-3-1788840ff10e> in train(self, iterations, checkpoint, initial_money)
    135             cost, _ = self.sess.run([self.cost, self.optimizer], feed_dict={self.X:np.vstack(ep_history[:,0]),
    136                                                     self.REWARDS:ep_history[:,2],
--> 137                                                     self.ACTIONS:ep_history[:,1]})
    138 
    139

...so I get the printed output of the ep_history array to check its shape, which has a starting shape of :

array([[0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
               ...
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.]]) # which is the current state (5x30)

       0 10000 # which is the action and reward/starting_money

 array([[ 0.000000e+00,  0.000000e+00,  0.000000e+00,  0.000000e+00,
     0.000000e+00],
   [ 0.000000e+00,  0.000000e+00,  0.000000e+00,  0.000000e+00,
     0.000000e+00],
                      ....
   [ 0.000000e+00,  0.000000e+00,  0.000000e+00,  0.000000e+00,
     0.000000e+00],
   [ 6.000060e-01,  1.669999e+00,  4.899980e-01,  2.309998e+00,
    -2.135710e+07]])] # which is the NEXT state

...which acts as the state (5 columns of features and 30 rows of window_size), action (0), starting_money (10,000), and next_state (again 5 x 30 with the next/future row of the dataset).

So what seems to be the issue now? Is it the array? I apologize for the lengthy code, but I want to be thorough for those helping me and to show that I'm actually trying to understand the logic behind the fix so I can apply it later. Any further input? Would it be something to do with get_state function? (I've added a couple of extra comments to the colab book as I work through this). Thanks a lot.

wildcat89
  • 1,159
  • 16
  • 47
  • Is `(35760, 5)` the shape of `np.vstack(ep_history[:,0])`? This seems like a time series - would it make sense to try a recurrent model (e.g. LSTM) instead of a fully-connected network? – rvinas Sep 12 '19 at 07:43
  • @rvinas You're right, it is a time series problem, and I do have an LSTM too, but I have a great many different similar models to this one that I'd like to be able to figure out how to implement this change here first. I've updated my question above with some more information that might help?? – wildcat89 Sep 13 '19 at 02:32
  • @rvinas I guess to answer your question, the shape of np.vstack(ep_history[:,0]) SHOULD be 30 rows x 5 columns, 5 features and 30 row window_size, which should be shown above. Thanks! – wildcat89 Sep 13 '19 at 02:35
  • can you share your data. i am trying to do in a different setting and colab is not my favorite one – eugen Sep 18 '19 at 06:31
  • @eugen Just download this file and that'll be what my data looks like: https://finance.yahoo.com/quote/SPY/history?period1=1537293795&period2=1568829795&interval=1d&filter=history&frequency=1d – wildcat89 Sep 18 '19 at 18:02

1 Answers1

0

you have used "self.X" as input for 1st layer, for a model number of rows (data points) can vary but number of features should be same during training and predicting because that determines the number of neurons on that layer.

But you can reuse the code for data with different number of features to create different model but it has to remain same for a model.

self.X = tf.compat.v1.placeholder(tf.float32, (None,feature_numbers))

You must know feature size of your input when you start training and it can't be changed latter. If you want to track other things then you need to create other variables for that or you can pre-process your data to know feature numbers before creating tensorflow graph

Dev Khadka
  • 5,142
  • 4
  • 19
  • 33
  • Thank you for the help, but I'm now encountering a new error. I've updated my question above and would love you're input. I'm trying to show that I'm doing my best to work through this problem! Let me know what you think. I'd like to be able to use one placeholder rather than several for each feature if I can? – wildcat89 Sep 13 '19 at 02:33
  • I would suggest to look shape of tensor at every step and verify that it is as expected by you. like you can print "rewards.shape" to inspect its shape. The error message suggests that some operation is not possible to do between tensor of the shapes – Dev Khadka Sep 13 '19 at 07:32
  • Well, I think I know what the problem is, but not sure where to pinpoint it. So the shape of the `ep_history` array is (3270, 4), where [0] of that array is my vstacked 5 features, [1] is the actions, and [2] the rewards. The error points to `self.ACTIONS` in the traceback when mentioning the `Incompatible shapes: [3270,3] vs. [98100,3]` error. My actions being either 0,1 or 2, but the 98100 shape is (3270 * 30), so for some reason, window_size is * the number of rows of actions somewhere???? I feel like I'm very close here! – wildcat89 Sep 13 '19 at 08:20