Python gpt-2-simple, load multiple models at once

Question

I'm working on a discord bot and one of the functions I want to implement responds with text generated by the gpt-2-simple library. I want to have more then one model loaded to have multiple models available to respond to messages from my users.

However I get the following error when i run the load_gpt2() function in the second model

File "main.py", line 22, in <module>
    main()
  File "main.py", line 16, in main
    text_events.register_Message(client)
  File "U:\discord_bot\text_events\__init__.py", line 19, in register_Message
    event.init()
  File "U:\discord_bot\text_events\model2.py", line 20, in init
    gpt2.load_gpt2(sess, run_name='model2', checkpoint_dir="characters")
  File "C:\Program Files\Python36\lib\site-packages\gpt_2_simple\gpt_2.py", line 389, in load_gpt2
    output = model.model(hparams=hparams, X=context, gpus=gpus)
  File "C:\Program Files\Python36\lib\site-packages\gpt_2_simple\src\model.py", line 183, in model
    initializer=tf.compat.v1.random_normal_initializer(stddev=0.01))
  File "C:\Program Files\Python36\lib\site-packages\tensorflow_core\python\ops\variable_scope.py", line 1500, in get_variable
    aggregation=aggregation)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow_core\python\ops\variable_scope.py", line 1243, in get_variable
    aggregation=aggregation)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow_core\python\ops\variable_scope.py", line 567, in get_variable
    aggregation=aggregation)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow_core\python\ops\variable_scope.py", line 519, in _true_getter
    aggregation=aggregation)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow_core\python\ops\variable_scope.py", line 868, in _get_single_variable
    (err_msg, "".join(traceback.format_list(tb))))
ValueError: Variable model/wpe already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:

  File "C:\Program Files\Python36\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()
  File "C:\Program Files\Python36\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)

I've tried to find a way to keep the gpt2 instances seperate between modules but i can't find anything that achieves this sandboxing effect, or any other advice for seperating the models or their instances. Does anyone have any ideas?

score 2 · Answer 1 · answered Jun 17 '21 at 00:36

Per @Kedar suggestion, you could use separate Python Processes to isolate execution and load the model in each process separately. Alternatively, you could ensure that only one instance of the model is loaded at a time using a Singleton pattern, or more simply, add the lru_cache decorator (https://docs.python.org/3/library/functools.html) to a wrapper around the load_gpt2 function. Example:

from functools import lru_cache

@lru_cache
def load_gpt2_wrapper(*args, **kwargs):
    # do some logic and return the model

In this way, the same model is returned for each call of load_gpt2_wrapper, assuming the same arguments are provided.

Alternatively, in that wrapper function, call tf.reset_default_graph() each time, similar to the suggestions made in https://github.com/minimaxir/gpt-2-simple/issues/80.

Overall, I think the best solution would be to provide the option to provide a reuse when creating the Tensorflow session in the GPT-2 repo, which I have done here: https://github.com/minimaxir/gpt-2-simple/pull/272. In your case, since it looks like you are creating the session separately and supplying it to load_gpt2, you can provide the reuse option explicitly:

sess = tf.compat.v1.Session(reuse=reuse, ...)
model = load_gpt2(sess, ...)

That should mitigate the issue, assuming you can keep one session running for your application.

score 0 · Answer 2 · answered Jun 16 '21 at 08:15

0

if you just want to run multiple instances of the same model in isolation then you could do that with docker.

answered Jun 16 '21 at 08:15

Snawar

61
3

score 0 · Answer 3 · answered Jun 16 '21 at 08:19

Have you tried ProcessPoolExecutor to isolate execution?

from concurrent import futures

def do_stuff(arg)
   # load gpt module here
   return "response for ({})".format(arg)


with futures.ProcessPoolExecutor(max_workers=1) as exec:
   future = exec.submit(do_stuff, 1)
   for f in futures.as_completed([future]):
       return_val = f.result()

Python gpt-2-simple, load multiple models at once

3 Answers3