TensorFlow Dense Layers: 1 dimentional weights?

Question

I have my network set up in the following fashion:

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(10, activation='softmax')
])

I would expect this configuration to be like this:

[784 neurons]
(784,128 weights)
[128 neurons]
(128,10 weights)
[10 neurons]

But, when I print the network's weights with model.get_weights(), it produces the following output:

for w in model.get_weights():
    print(w.shape,"\n")

(784, 128)

(128,)

(128, 10)

(10,)

Why do (128,) and (10,) exist in this model?

`(784, 128)` and `(128, 10)` are the last two layers of **weights**. `(128,)` and `(10,)` are the last two layers of **biases**. — giser_yugang, Apr 19 '19 at 06:43
@giser_yugang Oh, I forgot about biases! Does this mean every node in a dense layer has a bias? — Kento Nishi, Apr 19 '19 at 06:46
I just read https://stackoverflow.com/questions/35000215/neural-net-bias-per-layer-or-per-node-non-input-node, and it helped me understand. Thanks for the helpful comment! — Kento Nishi, Apr 19 '19 at 06:53
Someone downvoted and requested this question to be closed. How can I improve the phrasing of the question to make it better? — Kento Nishi, Apr 19 '19 at 18:19
@desertnaut Thank you for your advice. I added it as an answer. — giser_yugang, Apr 20 '19 at 01:53

score 1 · Accepted Answer · answered Apr 20 '19 at 01:52

(784, 128) and (128, 10) are the last two layers weights. (128,) and (10,) are the last two layers biases. If you don't need biases, you can use use_bias parameter to set it. For example:

import keras

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, use_bias=False,activation='relu'),
    keras.layers.Dense(10, use_bias=False,activation='softmax')
])

for w in model.get_weights():
    print(w.shape,"\n")

# print
(784, 128) 

(128, 10)

TensorFlow Dense Layers: 1 dimentional weights?

1 Answers1