ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds

Question

Trying to convert a question-generation t5 model to torchscript model, while doing that Running into this error

ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds

here's the code that I ran on colab.

!pip install -U transformers==3.0.0
!python -m nltk.downloader punkt

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch

model = AutoModelForSeq2SeqLM.from_pretrained('valhalla/t5-base-qg-hl')

t_input =  'Python is a programming language. It is developed by <hl> Guido Van Rossum <hl>. </s>'

tokenizer = AutoTokenizer.from_pretrained('valhalla/t5-base-qg-hl', return_tensors = 'pt')

def _tokenize(
    inputs,
    padding=True,
    truncation=True,
    add_special_tokens=True,
    max_length=64
):
    inputs = tokenizer.batch_encode_plus(
        inputs, 
        max_length=max_length,
        add_special_tokens=add_special_tokens,
        truncation=truncation,
        padding="max_length" if padding else False,
        pad_to_max_length=padding,
        return_tensors="pt"
    )
    return inputs

token = _tokenize(t_input, padding=True, truncation=True)


traced_model = torch.jit.trace(model, [token['input_ids'], token['attention_mask']] )
torch.jit.save(traced_model, "traced_t5.pt")

got this error

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-f9b449524ef1> in <module>()
     32 
     33 
---> 34 traced_model = torch.jit.trace(model, [token['input_ids'], token['attention_mask']] )
     35 torch.jit.save(traced_model, "traced_t5.pt")

7 frames
/usr/local/lib/python3.6/dist-packages/transformers/modeling_t5.py in forward(self, input_ids, attention_mask, encoder_hidden_states, encoder_attention_mask, inputs_embeds, head_mask, past_key_value_states, use_cache, output_attentions, output_hidden_states)
    682         else:
    683             if self.is_decoder:
--> 684                 raise ValueError("You have to specify either decoder_input_ids or decoder_inputs_embeds")
    685             else:
    686                 raise ValueError("You have to specify either input_ids or inputs_embeds")

ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds

how to resolve this issue? or is there a better way for converting the t5 model to torchscript.

thank you.

I assume that `torch.jit.trace` calls the forward method of the model (please correct me if I am wrong). The forward method requires either `decoder_input_ids` or `decoder_inputs_embeds` just as the error says. [Link](https://huggingface.co/transformers/model_doc/t5.html#transformers.T5ForConditionalGeneration.forward) link to the documentation. — cronoik, Dec 04 '20 at 20:21
yes, I need to pass three args (`input_ids`, `attention_mask`, `decoder_input_ids` ). but I'm not sure what to pass for `decoder_input_ids` in the above context. — kiranr, Dec 05 '20 at 14:52

kiranr · Accepted Answer · 2021-03-31T15:46:25.870

3

Update: refer to this answer and if you are exporting t5 to onnx, it can be done easily using the fastT5 library.

I figured out what was causing the issue. Since the above model is sequential, it has both an encoder and a decoder. We need to pass the features into the encoder and labels (targets) into the decoder.

traced_model = torch.jit.trace(model, 
                               (input_ids, attention_mask, decoder_input_ids, decoder_attention_mask)
                               )
torch.jit.save(traced_model, "qg_model.pt")

the decoder_input_ids is tokenized ids of the question (here the question is a label).

Even though the torchscript model is created, it does not have the generate() method as the huggingface' t5 do.

edited Mar 31 '21 at 15:46

answered Dec 06 '20 at 20:38

kiranr

2,087
14
32

What do you mean by "it doe not have the "generate()" method"? If it is true, can't we do inference with the Torchscripted T5 model? – Ryan Jun 09 '23 at 07:48
@Ryan for inference we need to use the [generate](https://huggingface.co/docs/transformers/v4.30.0/en/main_classes/text_generation#transformers.GenerationMixin.generate) method, which can only be obtained from subclassing the `T5ForConditionalGeneration` as done [here](https://github.com/Ki6an/fastT5/blob/20441b33394e71f7612f39f228ecbe1925cd10ae/fastT5/onnx_models.py#L111) in fastT5. please refer to this library for further clarification. – kiranr Jun 14 '23 at 13:54

ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds

1 Answers1

Linked