MultiprocessIterator throws error when changing batch_size

Question

I want to train a Faster R-CNN with ChainerCV. As a first test I mostly copied the provided example, I only changed the lines corresponding the dataset to use my custom dataset. I checked if my dataset is fully functional with all operations discribed in this tutorial.

If I run the script without changes everything works perfect, but if I change the batch_size I get an error. I tried increasing the shared_mem from 100 MB to 1000 MB, but the error didn’t disappear.

Error when setting the batch_size=2:

Exception in main training loop: all the input array dimensions except for the concatenation axis must match exactly
Traceback (most recent call last):
  File "/home/cv/anaconda3/envs/chainer/lib/python3.6/site-packages/chainer/training/trainer.py", line 315, in run
    update()
  File "/home/cv/anaconda3/envs/chainer/lib/python3.6/site-packages/chainer/training/updaters/standard_updater.py", line 165, in update
    self.update_core()
  File "/home/cv/anaconda3/envs/chainer/lib/python3.6/site-packages/chainer/training/updaters/standard_updater.py", line 171, in update_core
    in_arrays = self.converter(batch, self.device)
  File "/home/cv/anaconda3/envs/chainer/lib/python3.6/site-packages/chainer/dataset/convert.py", line 134, in concat_examples
    [example[i] for example in batch], padding[i])))
  File "/home/cv/anaconda3/envs/chainer/lib/python3.6/site-packages/chainer/dataset/convert.py", line 164, in _concat_arrays
    return xp.concatenate([array[None] for array in arrays])
Will finalize trainer extensions and updater before reraising the exception.
Traceback (most recent call last):
  File "/home/cv/ChainerCV/faster_rcnn/train.py", line 131, in <module>
    main()
  File "/home/cv/ChainerCV/faster_rcnn/train.py", line 126, in main
    trainer.run()
  File "/home/cv/anaconda3/envs/chainer/lib/python3.6/site-packages/chainer/training/trainer.py", line 329, in run
    six.reraise(*sys.exc_info())
  File "/home/cv/anaconda3/envs/chainer/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/home/cv/anaconda3/envs/chainer/lib/python3.6/site-packages/chainer/training/trainer.py", line 315, in run
    update()
  File "/home/cv/anaconda3/envs/chainer/lib/python3.6/site-packages/chainer/training/updaters/standard_updater.py", line 165, in update
    self.update_core()
  File "/home/cv/anaconda3/envs/chainer/lib/python3.6/site-packages/chainer/training/updaters/standard_updater.py", line 171, in update_core
    in_arrays = self.converter(batch, self.device)
  File "/home/cv/anaconda3/envs/chainer/lib/python3.6/site-packages/chainer/dataset/convert.py", line 134, in concat_examples
    [example[i] for example in batch], padding[i])))
  File "/home/cv/anaconda3/envs/chainer/lib/python3.6/site-packages/chainer/dataset/convert.py", line 164, in _concat_arrays
    return xp.concatenate([array[None] for array in arrays])
ValueError: all the input array dimensions except for the concatenation axis must match exactly

System info:

__Hardware Information__
Machine                                       : x86_64
CPU Name                                      : skylake
Number of accessible CPU cores                : 8

__OS Information__
Platform                                      : Linux-4.15.0-45-generic-x86_64-with-debian-stretch-sid
Release                                       : 4.15.0-45-generic
System Name                                   : Linux
Version                                       : #48~16.04.1-Ubuntu SMP Tue Jan 29 18:03:48 UTC 2019
OS specific info                              : debianstretch/sid
glibc info                                    : glibc 2.10

__CUDA Information__
Found 1 CUDA devices
id 0     b'GeForce GTX 1080'                              [SUPPORTED]
                      compute capability: 6.1
                           pci device id: 0
                              pci bus id: 1
Summary:
    1/1 devices are supported
CUDA driver version                           : 10000

__Conda Information__
conda_build_version                           : 3.17.6
conda_env_version                             : 4.6.3
platform                                      : linux-64
python_version                                : 3.7.1.final.0

EDIT: When running the example with batch_size=2 the error also occurs.

Answer 1

While trying to fix the error I got another error.

ValueError: Currently only batch size 1 is supported.

Waiting seems to be the solution.

Answer 2

Current Faster-RCNN implementation does not support multi batch training, but you can rewrite it to support it like code below. https://github.com/knorth55/chainer-light-head-rcnn/blob/master/light_head_rcnn/links/model/light_head_rcnn_train_chain.py

Another option is using Faster-RCNN with FPN in ChainerCV. The latest version of ChainerCV has Faster-RCNN with FPN which supports multi batch training. https://github.com/chainer/chainercv/blob/master/examples/fpn/train_multi.py

Answer 3

self.converter assumes that the first argument of batch is composed of inputs that have the same shape. For example, if you use image dataset, all images are supposed to have the shape of (C, H, W).

So, could you check your dataset returns images of the same shape? And if your dataset has various shapes of images, how about using TransformDataset like https://github.com/chainer/chainercv/blob/df63b74ef20f9d8c830e266881e577dd05c17442/examples/faster_rcnn/train.py#L86?