1

My function looks like the following:

@celery.task(name='large_task')
def large_task(self, user_info, data):
    # 1. long time to load model based on user information
    custom =  custom_Class(user_info)
    # 2. apply custom_function
    results = custom.apply(data)
    return results

The custom class loads a large neural network (> 500 mb) which dimensions are based on the input of the user. It is also fine-tuned based on the information provided by the user (10 sec). Each neural network is therefore unique. It is therefore not possible to just pre-load it at the start of the celery worker, furthermore it is too big to serialize it and store it on redis every time. The goal would be to initialize and fine-tune the neural network and apply it to a later point in time to input data which is also dependent on the choices of the user. The celery task is activated via a post request coming through flask.

Thimm Zwiener
  • 13
  • 6
  • "dimensions are based on the input of the user" but still, there's finite and few amount of possible models, right? – Slam Feb 11 '19 at 17:05
  • The neural network gets initiated and fine-tuned based on the information provided by the user. Therefore, a unique network is served to each user. This network is then used at a later point to provide results – Thimm Zwiener Feb 11 '19 at 17:29

0 Answers0