0

I have the following code :

self.wi = nn.Embedding(num_embeddings, embedding_dim)
self.wj = nn.Embedding(num_embeddings1, embedding_dim)
self.bi = nn.Embedding(num_embeddings, 1)
self.bj = nn.Embedding(num_embeddings1, 1)

self.wi.weight.data.uniform_(-1, 1)
self.wj.weight.data.uniform_(-1, 1)
self.bi.weight.data.zero_()
self.bj.weight.data.zero_()

I want to initialize the weights with numpy array, and I want to create a constant tensor, which is also a numpy array. I am new to PyTorch, and I appreciate any help.

Abrar
  • 621
  • 1
  • 9
  • 17
  • If I understand you correctly, you might want to split up your question into separate parts (initializing with the `numpy.array`, and then also a different question about the constant tensor). Otherwise, you can generally look for answers on how to initialize embeddings with other values, see [here](https://stackoverflow.com/questions/49710537/pytorch-gensim-how-to-load-pre-trained-word-embeddings) – dennlinger Dec 12 '19 at 10:14

2 Answers2

1

You can initialize embedding layers with the function nn.Embedding.from_pretrained().

In your specific case, you would still have to firstly convert the numpy.array to a torch.Tensor, but otherwise it is very straightforward:

import torch as t
import torch.nn as nn
import numpy as np

# This can be whatever initialization you want to have
init_array = np.zeros([num_embeddings, embedding_dims])

# As @Daniel Marchand mentioned in his answer, 
# you do have to cast it explicitly as a tensor, otherwise it won't work.
wi = nn.Embedding.from_pretrained(t.tensor(init_array), freeze=False)

The parameter freeze=False is important if you still want to train your network afterwards, as otherwise you would keep the embeddings at the same constant values. Generally, .from_pretrained is used to "transfer" learned embeddings, but it does work for your case, too.

dennlinger
  • 9,890
  • 1
  • 42
  • 63
0

You may wish to look at converting a numpy array into a torch tensor How to convert Pytorch autograd.Variable to Numpy?

Daniel Marchand
  • 584
  • 8
  • 26