2

In an effort to combine medical expertise with deep learning I would like to combine in one tensor multiple images which represent different maps of the same CT image.

So say we begin with the following CT image:

enter image description here

We can hand craft different mappings of this particular image (i.e. transformations) that will embody relevant medical diagnostic knowledge.

So we may end up with say 10 2D grayscale images starting from 1 which we can then pack in a 3D tensor.

Is it possible to use transfer learning leveraging trained DL CNN models in the public domain (e.g. Keras) using as inputs these 3D images with 10 Channels (rather than 3 -- R-G-B)?

If yes how? If not, what are the alternatives?

halfer
  • 19,824
  • 17
  • 99
  • 186
user8270077
  • 4,621
  • 17
  • 75
  • 140
  • 1
    Giving multiple channel input to a network using transfer learning is doable. A quick search for a resource got me this- https://forums.fast.ai/t/classifier-with-multiple-images-as-input-and-multiple-labels-as-output/29704/2 Hope it helps. – mrtpk Oct 03 '19 at 12:45

1 Answers1

1

I think it is possible, but it probably will not give you good results, see the post script for the why.

You can do it like this but you have to add a conv2D layer that have 3 filters. As alternative you can remove first two layers (input layer and first conv2D layer) and replace with a different conv2d (because the pre-trained Conv2d layer have weights that have 3 channels as depths).

PS: Remember that, with a pre-trained you have to normalize as same, but considering that you don't have a 3 channel image you can't

halfer
  • 19,824
  • 17
  • 99
  • 186
Kubaba
  • 136
  • 1
  • 8