Superimposing 2D heat map on 3D Image with transparency

Question

I have a 3D python Image and its 2D feature vector . I displayed each with imshow() as shown in the code down here and I can see it clearly. What I wanted now is to overlay the 2D feature vector as a heat map on top of its 3D image. I tried adding them but dimension issue was raised , I expanded the 2D feature vector by adding third dimension but superimposed image is messed up. The dimension are are here :- feature >> (32,96) , image >> (32,96,3)

img_feature = np.uint8(feature22 + img_raw_np_resize)

ax0.imshow(img,extent =extent)     
ax1.imshow(feature,alpha = 0.75, interpolation = 'gaussian', cmap = plt.cm.jet,extent =extent)  
ax2.imshow(img_feature,alpha = 0.75, interpolation = 'gaussian', cmap = plt.cm.jet,extent =extent)

extent refers to make all of them display in same size and its is defined already. here is what I go from my attempt by the above code

The display I am looking for something like below ...

and here is the pesudo-code that does this ( I am trying to produce minimal code to share but it is taking time - so I share this in case it helps to figure out the issue : -

def save_feature_to_img_2(self):

    feature = (
        get_feature()
    )  #  get feature of size -- > torch.Size([1, 256, 32,96])

    feature2 = features[
        :, 0, :, :
    ]  #  I pick one from the 256 feature maps - [1,1,32,96]

    feature2 = (
        feature2.data.numpy()
    )  # convert it to numpy array for later fusion with image array - [1,1,32,96]

    features2 = features2.view(
        features2.shape[1], features2.shape[2]
    )  # reshape into 2D feature map  [32,96]

    img_raw1 = Image.open(self.img_path)  # read raw image size [128,64,3]

    img_raw_np = np.array(img_raw1)  # imge to numpy array

    newsize = (h, w)  # shape to re-size image to same size a feature
    img_raw_resize = img_raw1.resize(newsize)
    img_raw_np_resize = np.array(img_raw_resize)  # size is now [32,96,3]

    # use sigmod to  normalize feature to  [0,1]
    feature2 = 1.0 / (1 + np.exp(-1 * feature2))

    # display setup
    dx, dy = 0.05, 0.05

    y = np.arange(-8, 8, dx)  # Y - axis range
    x = np.arange(-4.0, 4.0, dy)  # X -axis range
    X, Y = np.meshgrid(x, y)  # meshgrid to enclose my display into
    extent = (
        np.min(x),
        np.max(x),
        np.min(y),
        np.max(y),
    )  # extent - the X and Y range  - just to make unifrom display

    feature2 = (255.5 * feature2 / np.amax(feature2)).astype(
        np.uint8
    )  # put feature into [0-255] range for colour image

    # to display img, feature and img_feature
    fig = plt.figure()
    ax0 = fig.add_subplot(131, title="Image")
    ax1 = fig.add_subplot(132, title="Heatmap")
    ax2 = fig.add_subplot(133, title="overlayed")

    # hereunder I used this code share in the answer to fuse
    alpha = 0.5

    img_heatmap = (
        feature2[:, :, None].astype(np.float64) * alpha
        + img_raw_np_resize * (1 - alpha)
    ).astype(np.uint8)

    ax0.imshow(img_raw1, alpha=1.0, interpolation="gaussian", cmap=plt.cm.jet)
    ax1.imshow(feature2, alpha=1.0, interpolation="gaussian", cmap=plt.cm.jet)
    ax2.imshow(
        img_heatmap, alpha=0.7, interpolation="gaussian", cmap=plt.cm.jet
    )

    cv2.imwrite("./img_heatmap.jpg", img_heatmap)

This is the new display I got

I use the following to fuse image and feature ...

alpha = 0.5         
img_feature = ((plt.cm.jet(feature2)[:, :, :3] * 255) * alpha + (1-alpha)*img_raw_np_resize).astype(np.uint8)  and displaying it with ax2.imshow(img_feature,alpha = 0.7, interpolation = 'gaussian', cmap = plt.cm.jet)

What are you trying to achieve? If you want to show only heated part then you have to multiply, not add heatmap. Do `img_feature = (img * (feature[:, :, None].astype(np.float64) / np.amax(feature)).astype(np.uint8)` — Arty, Dec 30 '20 at 18:00
Thank for your the answer. I am trying to overlay the feature over the image. But, since the feature is extracted from the image, i want to overlay it on the original image and show which part of the image got focused during feature extraction. I want to overlay the feature on the image without changing the original image but having control over the brightness of the overplayed feature , making a bit transparent to see the image through it. I tried your tip but I didnt get that effect.In stead , it change both the original image and the feature , and when fused it even got messed. — Hatfim, Jan 01 '21 at 17:55
Based on your update to your question, where your provided example of correctly overlayed image, I updated [my answer](https://stackoverflow.com/a/65511738/941531) with **Variant 3** (at end of answer) which does same-looking picture as you provided [here is example](https://i.stack.imgur.com/wwVR3.png) of rendered picture. — Arty, Jan 03 '21 at 11:48
The reason that your image is not seen after my **Variant 3** overlaying is because I expect in my code for image to have values in range 0-255 (as regular RGB image), probably you have normalized as floating point between 0-1, so in my formula instead of sub-expression `img * (1 - alpha)` you have to use `img * 255 * (1 - alpha)` in your case. Please try and tell me. Anyway sooner or later we will find right solution, be patient! — Arty, Jan 05 '21 at 12:43

Arty · Accepted Answer · 2021-01-13T09:43:15.780

1

Variant 1

If you want to show only heated region of image you have to multiply by heatmap instead of adding.

Formula for you will be img_feature = (img * (feature[:, :, None].astype(np.float64) / np.amax(feature))).astype(np.uint8).

Full example code (with my own image and auto-generated example heatmap):

Try it online!

import requests, PIL.Image, io, numpy as np, matplotlib.pyplot as plt
# load some image
img = np.array(PIL.Image.open(io.BytesIO(requests.get('https://i.stack.imgur.com/vPlCG.jpg').content)))
# load or compute some features
h, w, _ = img.shape
mg = np.mgrid[:h, :w]
feature = mg[0].astype(np.float64) * mg[1].astype(np.float64)
feature = (255.5 * feature / np.amax(feature)).astype(np.uint8)
# compute heated image
img_feature = (img * (feature[:, :, None].astype(np.float64) / np.amax(feature))).astype(np.uint8)
# show images
fig, (ax0, ax1, ax2) = plt.subplots(1, 3)
ax0.imshow(img)
ax1.imshow(feature, alpha = 1., interpolation = 'gaussian', cmap = plt.cm.jet)
ax2.imshow(img_feature, alpha = 1., interpolation = 'gaussian', cmap = plt.cm.jet)
plt.show()

Output:

Variant 2

If you want just brighten (whiten) regions that are heated and darken (blacken) regions that are un-heated you just do alpha blending by formula alpha = 0.5; img_feature = (feature[:, :, None].astype(np.float64) * alpha + img * (1 - alpha)).astype(np.uint8).

Try it online!

import requests, PIL.Image, io, numpy as np, matplotlib.pyplot as plt
# load some image
img = np.array(PIL.Image.open(io.BytesIO(requests.get('https://i.stack.imgur.com/vPlCG.jpg').content)))
# load or compute some features
h, w, _ = img.shape
mg = np.mgrid[:h, :w]
feature = mg[0].astype(np.float64) * mg[1].astype(np.float64)
feature = (255.5 * feature / np.amax(feature)).astype(np.uint8)
# compute heated image
alpha = 0.5; img_feature = (feature[:, :, None].astype(np.float64) * alpha + img * (1 - alpha)).astype(np.uint8)
# show images
fig, (ax0, ax1, ax2) = plt.subplots(1, 3)
ax0.imshow(img)
ax1.imshow(feature, alpha = 1., interpolation = 'gaussian', cmap = plt.cm.jet)
ax2.imshow(img_feature, alpha = 1., interpolation = 'gaussian', cmap = plt.cm.jet)
plt.show()

Output:

Variant 3

Same as Variant-2 (with alpha-blending) but instead of black-and-white feature array is used RGB feature array based on matplotlib.pyplot.cm.jet coloring scheme.

As you can see from code you can use any coloring scheme in expression plt.cm.jet(feature) instead of plt.cm.jet colors.

Try it online!

import requests, PIL.Image, io, numpy as np, matplotlib.pyplot as plt
# load some image
img = np.array(PIL.Image.open(io.BytesIO(requests.get('https://i.stack.imgur.com/vPlCG.jpg').content)))
# load or compute some features
h, w, _ = img.shape
mg = np.mgrid[:h, :w]
feature = mg[0].astype(np.float64) * mg[1].astype(np.float64)
feature = (255.5 * feature / np.amax(feature)).astype(np.uint8)
# compute heated image
alpha = 0.5; img_feature = ((plt.cm.jet(feature)[:, :, :3] * 255) * alpha + img * (1 - alpha)).astype(np.uint8)
# show images
fig, axs = plt.subplots(2, 2)
axs[0, 0].imshow(img)
axs[0, 1].imshow(feature, alpha = 1., interpolation = 'gaussian', cmap = plt.cm.jet)
axs[1, 0].imshow(img_feature, alpha = 1., interpolation = 'gaussian', cmap = plt.cm.jet)
axs[1, 1].remove()
plt.show()

Output:

PS. I just noticed that Matplotlib does normalization (feature - min) / (max - min) when drawing heatmap hence I decided to do same thing in overlaying formula, final formula becomes such:

alpha = 0.5; img_feature = ((
    plt.cm.jet(
        (feature - np.amin(feature)).astype(np.float32)
        / (np.amax(feature) - np.amin(feature)).astype(np.float32)
    )[:, :, :3] * 255
) * alpha + img * (1 - alpha)).astype(np.uint8)

Example code of using formula above is here (or here, and for your images). Resulting image of using formula above is here (and for your images). Final your image overlayed:

edited Jan 13 '21 at 09:43

answered Dec 30 '20 at 18:21

Arty

14,883
6
36
69

@user3315422 Maybe you can share your whole code if it is not secret so that we can debug the problem. Also would be good to have your image. Also not really whole code is necessary but only the one that overlays your heatmap on image and shows them. – Arty Jan 01 '21 at 18:03
@user3315422 BTW, you can share your code through site [https://godbolt.org/](https://godbolt.org/), copy-paste your code there and click Share. And image can be shared through site [https://imgur.com/](https://imgur.com/). – Arty Jan 01 '21 at 18:05
@user3315422 Can you tell me what are you trying to achieve? How do you want heated area to be shown/overlayed over original image? Do you want this part of image to be more REDish or how? – Arty Jan 01 '21 at 18:18
Thank you so much for the detailed answer, incidentally wish you happy new year. I want the feature to be semi-transparent , while be able see its pattern, and at the same time allowing to notice which part of the image it is focusing on. I tried you code segment and as I mentioned in the comment above, both the image and feature change their appearance a lot in the resulting merged display – Hatfim Jan 01 '21 at 18:47
@user3315422 Can you share your code that mixes heatmap and image and also shows them? Maybe it contains some bugs, as it draws damaged image. Because as you can see in my answer/post above all overlayed images look good. – Arty Jan 01 '21 at 19:01
@user3315422 How about [variant on this image](https://i.stack.imgur.com/JCiI6.png) ? I mad heatmap RED and final overlayed image (bottom-left) is more reddish where in the more heated area (bottom-right area of image). Is it what you want? Maybe you can explain what do you expect final image to look like? – Arty Jan 01 '21 at 19:05
@Hatfim Just updated my answer, added **Variant 3** which draws overlayed images same like one that you provided in your question-post. Put a look if this is what you need. – Arty Jan 03 '21 at 12:29
I added sample display and also include pesudo code, I will try to make a minimal code to generate that .. actually this was part of CNN model where I feature is extracted from intermediate layers and overlayed on the original image to highlight responsive regions of the image. ( see the new sample i added and the code fragment under it) – Hatfim Jan 03 '21 at 12:33
@Hatfim Yes, I saw your new updates and images, that's why I'm saying that I updated my answer-post with **Variant 3**, just 1 hour ago, please put a look, in my answer I created [image like this](https://i.stack.imgur.com/wwVR3.png) which looks almost similar to what you wanted. – Arty Jan 03 '21 at 12:55
sorry for my late follow up. I update and put some sample in my question once again, that is the output I got following the suggestion in your answer in variant 3 .. I dont know why i cant get the feature as transparent as yours .. the image is so much blurred , I guess may be the alapha value usage or normalization seems to cause this effect ..still cant figure it out – Hatfim Jan 05 '21 at 10:51
@Hatfim The reason that your image is not seen after overlaying is because I expect in my code for image to have values in range 0-255 (as regular RGB image), probably you have normalized as floating point between 0-1, so in my formula instead of sub-expression `img * (1 - alpha)` you have to use `img * 255 * (1 - alpha)` in your case. Please try and tell me. Anyway sooner or later we will find right solution, be patient! – Arty Jan 05 '21 at 10:54
Thank you Arty.I did not normalize the image , but the feature was extracted from a normalized image. img_raw1 = Image.open(self.img_path) this is the image which I convert to array and resize it to match feature dimenion. with alpha = 0.5 , feature superimposed on img as :- img_feature = ((plt.cm.jet(feature2)[:, :, :3] * 255) * alpha + (1-alpha)*img_raw_normalized).astype(np.uint8) , note that my feature dim is [HXW] and img dim [HXWX3] , what are you doing with plt.cm.jet(feature2)[:, :, :3] * 255 in your code ? May be color mapping? Are u adding 3rd dim to feature too? – Hatfim Jan 06 '21 at 04:55
@Hatfim `plt.cm.jet(feature2)[:, :, :3] * 255` does several things, 1) it takes `feature2` as 2D (HxW) array of floating point numbers ranging from 0 to 1. 2) Applies coloring function to it `plt.cm.jet(feature2)`, this function converts each single floating point number to array of 4 numbers `[red, green, blue, alpha]`, each component is from 0 to 1. 3) Hence after step 2) your 2D array becomes 3D array of shape (HxWx4). 4) Then applying `[:, :, :3]` removes alpha channel thus your array becomes (HxWx3). 5) Then `* 255` makes all values to be from 0 to 255 (before they were 0 to 1). – Arty Jan 06 '21 at 10:52
@Hatfim As I see on your last overlayed image your heatmap is very well seen, while picture of boy is not seen at all, to fix this just try to replace `img * (1 - alpha)` inside my formula to `img * (1 - alpha) * 255.0`, it should fix the problem. Also notice that I used `alpha`, it is basically proportion in which either picture or heatmap is used in overlaying. Try changing alpha (currently it is 0.5) to other values between 0 and 1, when you change it you'll see more of heatmap or more of picture. But before changing alpha first try to replace with `img * (1 - alpha) * 255.0` in my formula – Arty Jan 06 '21 at 11:07
@Hatfim Basically your formula should become `img_feature = ((plt.cm.jet(feature2)[:, :, :3] * 255.0) * alpha + img_raw_np_resize * (1-alpha) * 255.0).astype(np.uint8)` – Arty Jan 06 '21 at 11:07
@Hatfim Look to my picture at bottom left [here](https://i.stack.imgur.com/wwVR3.png), as you can see it has very good clarity, you can see both picture of landscape and heatmap. It means that formula is correct and produces desired result. We just need to figure out what is the difference between your picture and my picture, and also between my heatmap and your heatmap. Try printing to console in your code hundred of values from `img_raw_np_resize` and from `feature2`, to see what they contain. Maybe they contain some bad stuff that's why my formula doesn't work. – Arty Jan 06 '21 at 11:10
@Hatfim Also don't forget to read all of 4-5 my last comments above, I wrote them just now. – Arty Jan 06 '21 at 11:21
Hi Arty, I go through your comments one by one and carefully. I try displaying the values in img and feature2. I didnt see strange stuff as you alluded in your comment. But, dont we need to aply plt.cm.jet function to img too ? like (plt.cm.jet(img)[:,:,:3])*alpha ... something like this ? or the PIL loaded image has go the alpha channel already ? I still couldn't figure out what went wrong. img pixel are in range 0 to 255 and features are 0 to 1 which later on scalled by *255 to same range as image. – Hatfim Jan 06 '21 at 19:49
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/226946/discussion-between-arty-and-hatfim). – Arty Jan 06 '21 at 19:59
@Hatfim Please come to [this chat link](https://chat.stackoverflow.com/rooms/226946/discussion-between-arty-and-hatfim), we will continue solving task there! – Arty Jan 06 '21 at 20:00
My Image data was not normalzied to 0 to 1 , i dont think there is a need to multiply 255.0. I actually tried it while I was not aware and It didnt work. I drop message a day ago on the chat box with the data you asked me ..look forward chatting you there – Hatfim Jan 09 '21 at 12:37
@Hatfim Answered you [in chat](https://chat.stackoverflow.com/rooms/226946/discussion-between-arty-and-hatfim). But you didn't respond. Maybe you don't see notifications about me writing in chat? If so please come there from time to time, if you want to solve the task. – Arty Jan 10 '21 at 06:32
I make change on the 'paste' setting and put the new link – Hatfim Jan 11 '21 at 02:04
@Hatfim I made a description of what to do for Debugging this problem, [in the chat](https://chat.stackoverflow.com/rooms/226946/discussion-between-arty-and-hatfim). Also don't forget to come to chat from time to time, because I'm writing messages there often. Also always in chat when you write a message put `@Arty` in the message. Because if you don't put this `@Arty` reference then I will be not notified about new chat messages, hence I will come to chat too but much later, not immediately. – Arty Jan 11 '21 at 08:17
Thank you so much again, I put the two png image's for feature and img and put link in the chat box , I appreciate your time to check it our . – Hatfim Jan 12 '21 at 03:53
@Hatfim Updated our [char discussion](https://chat.stackoverflow.com/rooms/226946/discussion-between-arty-and-hatfim) with details regarding next steps to improve quality of overlayed image. – Arty Jan 12 '21 at 07:56
Thank you for the detailed answer @Arty , finally that worked out for me. – Hatfim Jan 13 '21 at 08:39

Superimposing 2D heat map on 3D Image with transparency

1 Answers1