OpenAI GPT-2 model use with TensorFlow JS

Question

Is that possible to generate texts from OpenAI GPT-2 using TensorFlowJS?

If not what is the limitation, like model format or ...?

I'm having a hard time finding input and output nodes for the solution provided by @frederik-bode. Instead, I'm using "Pytorch serve" to expose the model through Rest API. Also GPT2 model is too large to serve in JS and I'm not seeing any advantage converting to TFJS for my use case. — jay, Jul 09 '20 at 12:30
aaah interesting! Have you done any time profiling to this solution? — Mohamed Taher Alrefaie, Jul 09 '20 at 14:57
This seems useful: https://github.com/tensorflow/tfjs/issues/3582 — Heath Mitchell, Feb 04 '21 at 14:16

Frederik Bode · Answer 1 · 2021-07-29T09:46:12.533

8

I don't see any reason as to why not, other than maybe some operation that is in gpt-2 that is not supported by tensorflowjs.

I don't know how to do it, but here's a nice starting point:

install.sh

python3 -m pip install -q git+https://github.com/huggingface/transformers.git
python3 -m pip install tensorflow

save.py

from transformers import TFGPT2LMHeadModel, GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
# add the EOS token as PAD token to avoid warnings
model = TFGPT2LMHeadModel.from_pretrained("gpt2", pad_token_id=tokenizer.eos_token_id)
model.save("./test_gpt2")

that will give you a SavedModel file. Now you can try figure out the input and output nodes, and use tensorflowjs_converter to try and convert it. Pointer: https://www.tensorflow.org/js/tutorials/conversion/import_saved_model.

edited Jul 29 '21 at 09:46

answered Jul 01 '20 at 14:33

Frederik Bode

2,632
1
10
17

2

This is in Python, I believe the OP is looking for JS answer. – denislexic Dec 07 '20 at 02:06
5

Yes, this is how to save the existing model from Python so it can be imported into JS – Heath Mitchell Feb 04 '21 at 14:15
1

any reason why `tensorflowjs` is imported? – Jules G.M. Jul 28 '21 at 19:07
removed it because I didn't see one – Jules G.M. Jul 28 '21 at 19:08

score 2 · Answer 2 · answered Jul 28 '23 at 17:34

It's possible. Maybe someone finds this useful in 2023:

One way to achieve this is to convert a TF model with tensorflowjs-converter as Frederik described (possible problem with this approach is missing custom layers)
Use gpt-tfjs - implementation of GPT model in TensorFlow.js. It's possible to load weights directly from HF (example). I developed it to experiment with model training in the browser.

If you just want to generate text without training, you have more options:

Use transformers.js or ONNX in general. The lib is great and follows Python's transformers library API. Unfortunately - inference only.
Use ggml + WASM. It's a C/C++ model implementation compiled to WebAssembly (example, talk)

OpenAI GPT-2 model use with TensorFlow JS

2 Answers2

Linked