0

I'm trying to run my CNN python code with use_multiprocessing=True in fit_generator function but i get error, and its work just fine with single process but the CPU load: 20% and GPU: 8%.

I'm running on MSI laptop with windows 10 core i7-7820HK CPU and NVIDIA GTX 1080 , using tensorflow backend

this is my code:

# Part 1 - Building the CNN

# Importing the Keras libraries and packages
from keras.models import Sequential
from keras.layers import Conv2D
from keras.layers import MaxPooling2D
from keras.layers import Flatten
from keras.layers import Dense
from keras.preprocessing.image import ImageDataGenerator


# Initialising the CNN
classifier = Sequential()

# Step 1 - Convolution
classifier.add(Conv2D(32, (3, 3), input_shape=(64, 64, 3), activation='relu'))

# Step 2 - Pooling
classifier.add(MaxPooling2D(pool_size=(2, 2)))

# Adding a second convolutional layer
classifier.add(Conv2D(32, (3, 3), activation='relu'))
classifier.add(MaxPooling2D(pool_size=(2, 2)))

# Step 3 - Flattening
classifier.add(Flatten())

# Step 4 - Full connection
classifier.add(Dense(units=128, activation='relu'))
classifier.add(Dense(units=1, activation='sigmoid'))

# Compiling the CNN
classifier.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Part 2 - Fitting the CNN to the images
train_datagen = ImageDataGenerator(rescale=1./255,
                                   shear_range=0.2,
                                   zoom_range=0.2,
                                   horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale = 1./255)

training_set = train_datagen.flow_from_directory('dataset\\training_set',
                                                 target_size = (64, 64),
                                                 batch_size = 32,
                                                 class_mode = 'binary')

test_set = test_datagen.flow_from_directory('dataset\\test_set',
                                            target_size = (64, 64),
                                            batch_size = 32,
                                            class_mode = 'binary')

if __name__ == '__main__':
    classifier.fit_generator(training_set,
                             workers=8,
                             max_queue_size=100,
                             use_multiprocessing=True,
                             steps_per_epoch=(8000 / 32),
                             epochs=25,
                             validation_data=test_set,
                             validation_steps=(2000 / 32))

and i get this error:

Using TensorFlow backend. Found 8000 images belonging to 2 classes. Found 2000 images belonging to 2 classes. Epoch 1/25 Exception in thread Thread-24: Traceback (most recent call last):   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\threading.py", line 916, in
_bootstrap_inner
    self.run()   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\site-packages\keras\utils\data_utils.py", line 548, in _run
    with closing(self.executor_fn(_SHARED_SEQUENCES)) as executor:   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\site-packages\keras\utils\data_utils.py", line 522, in <lambda>
    initargs=(seqs,))   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\multiprocessing\context.py", line 119, in Pool
    context=self.get_context())   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\multiprocessing\pool.py", line 174, in __init__
    self._repopulate_pool()   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\multiprocessing\pool.py", line 239, in _repopulate_pool
    w.start()   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj) TypeError: can't pickle _thread.lock objects

Exception in thread Thread-23: Traceback (most recent call last):   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\threading.py", line 916, in _bootstrap_inner
    self.run()   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\site-packages\keras\utils\data_utils.py", line 548, in _run
    with closing(self.executor_fn(_SHARED_SEQUENCES)) as executor:   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\site-packages\keras\utils\data_utils.py", line 522, in <lambda>
    initargs=(seqs,))   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\multiprocessing\context.py", line 119, in Pool
    context=self.get_context())   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\multiprocessing\pool.py", line 174, in __init__
    self._repopulate_pool()   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\multiprocessing\pool.py", line 239, in _repopulate_pool
    w.start()   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)   File "C:\Users\MSI-GT75\Anaconda3\envs\cnn\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj) TypeError: can't pickle _thread.lock objects

after updating all packages this error shows instead of the above one:

ValueError: Using a generator with `use_multiprocessing=True` is not supported on Windows (no marshalling of generators across process boundaries). Instead, use single thread/process or multithreading.
abdu saad
  • 3
  • 4
  • did you install TensorFlow using`pip install` or `conda install` ? – 7kemZmani Aug 25 '18 at 05:09
  • I used pip to install it – abdu saad Aug 25 '18 at 06:10
  • okay, now this error message is much clearer; and this seems to be an ongoing issue for [windows users](https://github.com/matterport/Mask_RCNN/issues/13#issuecomment-348775427). The suggested fix is to change `use_multiprocessing` from True to False. – 7kemZmani Aug 25 '18 at 08:42
  • I know that i can do that, but whats the point from it if not using the full resources? – abdu saad Aug 25 '18 at 09:48
  • 1
    as the error message said clearly `use_multiprocessing=True` is **not** supported on Windows. No one in SO can solve it for you. If you really need it to work you can use any Linux distro like Fedora or Ubuntu on a virtual machine inside windows or have it installed alongside windows; then your code will work in Linux. – 7kemZmani Aug 25 '18 at 09:56
  • Does this answer your question? [Is the class generator (inheriting Sequence) thread safe in Keras/Tensorflow?](https://stackoverflow.com/questions/52932406/is-the-class-generator-inheriting-sequence-thread-safe-in-keras-tensorflow) – adam.hendry Oct 12 '22 at 23:45
  • @abdusaad Please see my answer [here](https://stackoverflow.com/questions/52932406/is-the-class-generator-inheriting-sequence-thread-safe-in-keras-tensorflow/63641535#63641535) – adam.hendry Oct 12 '22 at 23:47

0 Answers0