2

From the docs it seems to me that it is using a embedding matrix to transform a one-hot encoding like sparse input vector to a dense vector. But how is this different from just using a fully connected layer?

Edmonds Karp
  • 155
  • 9
  • 1
    The difference is that each one-hot encoding vector is mapped to (possibly) a different dense vector. Essentially it's like a look-up table. For example it has applications when you want to represent each word (i.e. encoded as a one hot vector) with a dense embedded vector. Read [this](https://stats.stackexchange.com/questions/182775/what-is-an-embedding-layer-in-a-neural-network) on Cross Validated and [this](https://stackoverflow.com/questions/47868265/what-is-the-difference-between-an-embedding-layer-and-a-dense-layer) on Stackoverflow for more info. – today Mar 12 '18 at 07:58
  • 1
    And I think their difference with Dense layers can be summarized in efficiency: they are faster (because of look-up rather than matrix multiplications) and more data efficient (if fed with index vectors rather than one-hot vectors). – today Mar 12 '18 at 08:09

2 Answers2

2

Summarizing the answer from comments to here. The main difference is efficiency. Instead of having to encode data points in these very long one hot vectors and do matrix multiplication, using embedding_column allows you to use index vectors and do a matrix lookup.

Edmonds Karp
  • 155
  • 9
0

To represent categories.

Both one-hot encoding and embedding column are options to represent categorical features.

One of the problem with one-hot encoding is that it doesn't encode any relationships between the categories. They are completely independent from each other, so the neural network has no way of knowing which ones are similar to each other.

This problem can be solved by representing a categorical feature with an embedding column. The idea is that each category has a smaller vector. The values are weights, similar to the weights that are used for basic features in a neural network.

For more:

https://developers.googleblog.com/2017/11/introducing-tensorflow-feature-columns.html
Razan Paul
  • 13,618
  • 3
  • 69
  • 61