3

I'm trying to train StyleGAN2, but everytime I'm starting to train I'm getting this error. It's weird that it can't allocate 0 bytes.

Log:

Traceback (most recent call last):
  File "train.py", line 561, in <module>
    main()
  File "train.py", line 553, in main
    run_training(**vars(args))
  File "train.py", line 416, in run_training
    run_desc, training_options = setup_training_options(**hyperparam_options)
  File "train.py", line 105, in setup_training_options
    dataset_obj = dataset.load_dataset(**args.train_dataset_args) # try to load the data and see what comes out
  File "C:\Temporary Software\stylegan2-ada\training\dataset.py", line 231, in load_dataset
    mirror_augment=mirror_augment, repeat=repeat, shuffle=shuffle)
  File "C:\Temporary Software\stylegan2-ada\training\dataset.py", line 114, in __init__
    self._tf_labels_var = tflib.create_var_with_large_initial_value(self._np_labels, name='labels_var')
  File "C:\Temporary Software\stylegan2-ada\dnnlib\tflib\tfutil.py", line 234, in create_var_with_large_initial_value
    zeros = tf.zeros(initial_value.shape, initial_value.dtype)
  File "C:\Users\Andrew\anaconda3\envs\stylegan\lib\site-packages\tensorflow\python\ops\array_ops.py", line 1871, in zeros
    output = _constant_if_small(zero, shape, dtype, name)
  File "C:\Users\Andrew\anaconda3\envs\stylegan\lib\site-packages\tensorflow\python\ops\array_ops.py", line 1829, in _constant_if_small
    return constant(value, shape=shape, dtype=dtype, name=name)
  File "C:\Users\Andrew\anaconda3\envs\stylegan\lib\site-packages\tensorflow\python\framework\constant_op.py", line 246, in constant
    allow_broadcast=True)
  File "C:\Users\Andrew\anaconda3\envs\stylegan\lib\site-packages\tensorflow\python\framework\constant_op.py", line 284, in _constant_impl
    allow_broadcast=allow_broadcast))
  File "C:\Users\Andrew\anaconda3\envs\stylegan\lib\site-packages\tensorflow\python\framework\tensor_util.py", line 464, in make_tensor_proto
    nparray = np.empty(shape, dtype=np_dt)
numpy.core._exceptions.MemoryError: Unable to allocate 0 bytes for an array with shape (1073741824, 0) and data type float32

Is it a numpy bug or I'm doing something wrong? I'd be very much appreciated if you could help me

Denking
  • 31
  • 1
  • 4
  • It is a memory error, your computer simply doesn't have enough ram to perform the operation – Mayeul sgc Oct 04 '21 at 01:52
  • @Mayeulsgc: It's literally 0 bytes. – user2357112 Oct 04 '21 at 02:00
  • Memory is allocated in one page (4kb) increments. So, if you're out of ram, it is possible that `malloc` will return an error for a zero bytes allocation. However, normally it is handled by offloading least used pages to virtual RAM (swap file), so it is still weird – Marat Oct 04 '21 at 02:17
  • I've never seen a memory error with a (N,0) shape (for large N). But then creating arrays with a 0 dimension is usually a mistake. Even when used as the starting point for iterative `concatenate` it's a bad idea. – hpaulj Oct 04 '21 at 06:00
  • same here, with two GTX 1080ti GPUs and 32GB RAM system memory – noam gaash Jan 15 '22 at 10:06

1 Answers1

0

This was also reported in https://github.com/numpy/numpy/issues/16410, and fixed in https://github.com/numpy/numpy/pull/21477, which is probably included in numpy version 1.23.

Ling
  • 535
  • 4
  • 9