I'm trying to use the bigscience/bloom model on macOS for text completion. Here's the python script I'm using:
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained('bigscience/bloom')
inputs = tokenizer("My dog is ", return_tensors="pt")
model = AutoModelForCausalLM.from_pretrained("bigscience/bloom", device_map="auto",
offload_folder="offload", torch_dtype=torch.bfloat16)
output = model.generate(inputs["input_ids"].to(0), min_length=30, max_length=30,
do_sample=True)
print(tokenizer.decode(output[0].tolist()))
I've installed python3 3.9.12, and used pip3 to install accelerate, torch, and the HuggingFace transformers library.
When I run this, the model.generate fails with:
Traceback (most recent call last):
File "/Users/stevex/bloom/download.py", line 8, in <module>
model = AutoModelForCausalLM.from_pretrained("bigscience/bloom", device_map="auto", offload_folder="offload", torch_dtype=torch.bfloat16)
File "/opt/homebrew/lib/python3.9/site-packages/transformers-4.21.0.dev0-py3.9.egg/transformers/models/auto/auto_factory.py", line 446, in from_pretrained
return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
File "/opt/homebrew/lib/python3.9/site-packages/transformers-4.21.0.dev0-py3.9.egg/transformers/modeling_utils.py", line 2119, in from_pretrained
model, missing_keys, unexpected_keys, mismatched_keys, error_msgs = cls._load_pretrained_model(
File "/opt/homebrew/lib/python3.9/site-packages/transformers-4.21.0.dev0-py3.9.egg/transformers/modeling_utils.py", line 2332, in _load_pretrained_model
new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
File "/opt/homebrew/lib/python3.9/site-packages/transformers-4.21.0.dev0-py3.9.egg/transformers/modeling_utils.py", line 553, in _load_state_dict_into_meta_model
offload_index = offload_weight(param, param_name, offload_folder, offload_index)
File "/opt/homebrew/lib/python3.9/site-packages/accelerate/utils/offload.py", line 25, in offload_weight
array = weight.numpy()
TypeError: Got unsupported ScalarType BFloat16
There's a bfloat16 python library but when I attempt to install it:
pip3 install bfloat16
This fails with errors compiling bfloat16.cc; many errors but here's a sample:
bfloat16.cc:462:26: error: unexpected type name 'bfloat16': expected expression
/*alignment=*/alignof(bfloat16),
^
bfloat16.cc:757:43: error: a space is required between consecutive right angle brackets (use '> >')
struct TypeDescriptor<std::complex<float>>
^~
> >
bfloat16.cc:764:44: error: a space is required between consecutive right angle brackets (use '> >')
struct TypeDescriptor<std::complex<double>>
^~
> >
I suspect it's not supported on macOS.
I'm not that familiar with Python so not sure where to go from here. Thanks.
