-1

I understand reformer is able to handle a large number of tokens. However it does not appear to support the summarization task:

>>> from transformers import ReformerTokenizer, ReformerModel
>>> from transformers import pipeline
>>> summarizer = pipeline("summarization", model="reformer")
404 Client Error: Not Found for url: https://huggingface.co/reformer/resolve/main/config.json
...

How would you construct the pipeline "manually" to use reformer for summarization?

Mittenchops
  • 18,633
  • 33
  • 128
  • 246

1 Answers1

-1

Try this:

summarizer = pipeline("summarization", model="google/reformer-enwik8")

via here.

However, this produces...

/lib/python3.7/site-packages/sentencepiece.py", line 177, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
TypeError: not a string
Mittenchops
  • 18,633
  • 33
  • 128
  • 246
  • I think this should either be a separate question, or, better yet, a github issue on the transformers repository, if it doesn't work with the most recent version. – dennlinger Jan 11 '22 at 10:13
  • Update: it seems that this specific model does not provide a tokenizer altogether, which is why you're seeing the error message. You could use an alternative model instead, which provides a tokenizer, such as [`google/reformer-crime-and-punishment`](https://huggingface.co/google/reformer-crime-and-punishment) – dennlinger Jan 11 '22 at 10:22