Load XGBoost from an s3 bucket

Question

I have an XGBoost model sitting in an AWS s3 bucket which I want to load. currently, I'm attempting to use s3fs to load the data, but I keep getting type errors:

from s3fs.core import S3FileSystem
import xgboost as xgb

fs = S3FileSystem()
bst = xgb.XGBClassifier()

with fs.open(f'{bucket}/{modelPath}', mode='rb') as f:
    bst.load_model(f.read())

I would include output, this I'm pretty sure this approach is completely wrong. How do I get this working?

(also, I think I would rather use boto)

score 3 · Answer 1 · answered Nov 10 '21 at 22:01

3

Converting the content of the buffer to bytearray should do the trick. My inspiration come from the source code found here. It worked for me, without downloading locally the json file

with fs.open(f'{bucket}/{modelPath}', mode='rb') as f:
    bst.load_model(bytearray(f.read()))

answered Nov 10 '21 at 22:01

Guillaume Vergnolle

61
6

1

This should be the selected answer. It's more straightforward. – saujosai Jul 05 '23 at 17:44

score 0 · Accepted Answer · answered Jan 26 '21 at 00:40

Provided I don't receive a better answer, one good approach would be to download the file onto the lambda functions storage, then load it with bst.load_model('path_to_model'). based on this, each lambda function is allocated 500 MB of non persistent storage, which actually sits around for subsequent invocations of the same lambda function (until the lambda function goes cold).

Load XGBoost from an s3 bucket

2 Answers2