Thermal Image Processing

Question

I'm trying to stream FLIR Lepton 3.5 with Python OPENCV using "VideoCapture" & "cv2.imshow" script. Then, I'll do some detection and control. Here's the problem I have, I was only able to get a very faint black/gray video stream with what seems to be a couple of rows of dead pixels at the bottom of the stream. This is expected since the output is supposed to be 16-Bit RAW image data. So,

I'm trying to convert to RGB888 image data so the stream has "colors".
Why is the stream video in static mode, it doesn't stream video like normal embedded notebook webcam?

I've tried the codes/scripts that were shared by others and even the example code from FLIR application notes, but didn't work. Your help is appreciated.

Environment: Windows 10, Python 3.7.6, PyCharm, OpenCV (latest), FLIR Lepton 3.5 camera/PureThermal2

Code:

import cv2
import numpy as np


image_counter = 0
video = cv2.VideoCapture(0,cv2.CAP_DSHOW)
video.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter.fourcc('Y','1','6',' '))
video.set(cv2.CAP_PROP_CONVERT_RGB, 0)

if video.isOpened(): # try to get the first frame
    rval, frame = video.read()
else:
    rval = False

while rval:
    normed = cv2.normalize(frame, None, 0, 65535, cv2.NORM_MINMAX)

    nor=cv2.cvtColor(np.uint8(normed),cv2.COLOR_GRAY2BGR)
    cv2.imshow("preview", cv2.resize(nor, dsize= (640, 480), interpolation = cv2.INTER_LINEAR))

    key = cv2.waitKey(1)
    if key == 27: # exit on ESC
        break

The image looks about right. Remember that the sensor is only 160x120, and IR images never have the kind of sharp-edge definition you've come to expect from color images. — Tim Roberts, Apr 01 '21 at 20:28

Rotem · Accepted Answer · 2021-04-07T12:26:20.303

It's challenging to give an answer without having the camera.
Please note that I could not verify my solution.

I found the following issues with your code:

rval, frame = video.read() must be inside the while loop.
The code grabs the next frame.
If you want to grab more than one frame, you should execute it in a loop.
normed = cv2.normalize(frame, None, 0, 65535, cv2.NORM_MINMAX)
Returns uint16 values in range [0, 65535].
You are getting an overflow when converting to uint8 by np.uint8(normed).
I recommend normalizing to range [0, 255].
You may also select the type of the result to be uint8:
```
 normed = cv2.normalize(frame, None, 0, 255, cv2.NORM_MINMAX, cv2.CV_8U)
```

Here is the complete updated code (not tested):

import cv2
import numpy as np

image_counter = 0
video = cv2.VideoCapture(0,cv2.CAP_DSHOW)
video.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter.fourcc('Y','1','6',' '))
video.set(cv2.CAP_PROP_CONVERT_RGB, 0)

if video.isOpened(): # try to get the first frame
    rval, frame = video.read()
else:
    rval = False

# Create an object for executing CLAHE.
clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8))

while rval:
    # Get a Region of Interest slice - ignore the last 3 rows.
    frame_roi = frame[:-3, :]

    # Normalizing frame to range [0, 255], and get the result as type uint8.
    normed = cv2.normalize(frame_roi, None, 0, 255, cv2.NORM_MINMAX, cv2.CV_8U)

    # Apply CLAHE - contrast enhancement.
    # Note: apply the CLAHE on the uint8 image after normalize.
    # CLAHE supposed to work with uint16 - you may try using it without using cv2.normalize
    cl1 = clahe.apply(normed)

    nor = cv2.cvtColor(cl1, cv2.COLOR_GRAY2BGR)  # Convert gray-scale to BGR (no really needed).

    cv2.imshow("preview", cv2.resize(nor, dsize=(640, 480), interpolation=cv2.INTER_LINEAR))
    key = cv2.waitKey(1)
    if key == 27: # exit on ESC
        break

    # Grab the next frame from the camera.
    rval, frame = video.read()

Colorizing:

https://groups.google.com/g/flir-lepton/c/Cm8lGQyspmk

Result:

Here is the code sample with colorizing (using "Iron Black" color map):

import cv2
import numpy as np

# https://groups.google.com/g/flir-lepton/c/Cm8lGQyspmk
def generateColourMap():
    """
    Conversion of the colour map from GetThermal to a numpy LUT:
        https://github.com/groupgets/GetThermal/blob/bb467924750a686cc3930f7e3a253818b755a2c0/src/dataformatter.cpp#L6
    """

    lut = np.zeros((256, 1, 3), dtype=np.uint8)

    colormapIronBlack = [
        255, 255, 255, 253, 253, 253, 251, 251, 251, 249, 249, 249, 247, 247,
        247, 245, 245, 245, 243, 243, 243, 241, 241, 241, 239, 239, 239, 237,
        237, 237, 235, 235, 235, 233, 233, 233, 231, 231, 231, 229, 229, 229,
        227, 227, 227, 225, 225, 225, 223, 223, 223, 221, 221, 221, 219, 219,
        219, 217, 217, 217, 215, 215, 215, 213, 213, 213, 211, 211, 211, 209,
        209, 209, 207, 207, 207, 205, 205, 205, 203, 203, 203, 201, 201, 201,
        199, 199, 199, 197, 197, 197, 195, 195, 195, 193, 193, 193, 191, 191,
        191, 189, 189, 189, 187, 187, 187, 185, 185, 185, 183, 183, 183, 181,
        181, 181, 179, 179, 179, 177, 177, 177, 175, 175, 175, 173, 173, 173,
        171, 171, 171, 169, 169, 169, 167, 167, 167, 165, 165, 165, 163, 163,
        163, 161, 161, 161, 159, 159, 159, 157, 157, 157, 155, 155, 155, 153,
        153, 153, 151, 151, 151, 149, 149, 149, 147, 147, 147, 145, 145, 145,
        143, 143, 143, 141, 141, 141, 139, 139, 139, 137, 137, 137, 135, 135,
        135, 133, 133, 133, 131, 131, 131, 129, 129, 129, 126, 126, 126, 124,
        124, 124, 122, 122, 122, 120, 120, 120, 118, 118, 118, 116, 116, 116,
        114, 114, 114, 112, 112, 112, 110, 110, 110, 108, 108, 108, 106, 106,
        106, 104, 104, 104, 102, 102, 102, 100, 100, 100, 98, 98, 98, 96, 96,
        96, 94, 94, 94, 92, 92, 92, 90, 90, 90, 88, 88, 88, 86, 86, 86, 84, 84,
        84, 82, 82, 82, 80, 80, 80, 78, 78, 78, 76, 76, 76, 74, 74, 74, 72, 72,
        72, 70, 70, 70, 68, 68, 68, 66, 66, 66, 64, 64, 64, 62, 62, 62, 60, 60,
        60, 58, 58, 58, 56, 56, 56, 54, 54, 54, 52, 52, 52, 50, 50, 50, 48, 48,
        48, 46, 46, 46, 44, 44, 44, 42, 42, 42, 40, 40, 40, 38, 38, 38, 36, 36,
        36, 34, 34, 34, 32, 32, 32, 30, 30, 30, 28, 28, 28, 26, 26, 26, 24, 24,
        24, 22, 22, 22, 20, 20, 20, 18, 18, 18, 16, 16, 16, 14, 14, 14, 12, 12,
        12, 10, 10, 10, 8, 8, 8, 6, 6, 6, 4, 4, 4, 2, 2, 2, 0, 0, 0, 0, 0, 9,
        2, 0, 16, 4, 0, 24, 6, 0, 31, 8, 0, 38, 10, 0, 45, 12, 0, 53, 14, 0,
        60, 17, 0, 67, 19, 0, 74, 21, 0, 82, 23, 0, 89, 25, 0, 96, 27, 0, 103,
        29, 0, 111, 31, 0, 118, 36, 0, 120, 41, 0, 121, 46, 0, 122, 51, 0, 123,
        56, 0, 124, 61, 0, 125, 66, 0, 126, 71, 0, 127, 76, 1, 128, 81, 1, 129,
        86, 1, 130, 91, 1, 131, 96, 1, 132, 101, 1, 133, 106, 1, 134, 111, 1,
        135, 116, 1, 136, 121, 1, 136, 125, 2, 137, 130, 2, 137, 135, 3, 137,
        139, 3, 138, 144, 3, 138, 149, 4, 138, 153, 4, 139, 158, 5, 139, 163,
        5, 139, 167, 5, 140, 172, 6, 140, 177, 6, 140, 181, 7, 141, 186, 7,
        141, 189, 10, 137, 191, 13, 132, 194, 16, 127, 196, 19, 121, 198, 22,
        116, 200, 25, 111, 203, 28, 106, 205, 31, 101, 207, 34, 95, 209, 37,
        90, 212, 40, 85, 214, 43, 80, 216, 46, 75, 218, 49, 69, 221, 52, 64,
        223, 55, 59, 224, 57, 49, 225, 60, 47, 226, 64, 44, 227, 67, 42, 228,
        71, 39, 229, 74, 37, 230, 78, 34, 231, 81, 32, 231, 85, 29, 232, 88,
        27, 233, 92, 24, 234, 95, 22, 235, 99, 19, 236, 102, 17, 237, 106, 14,
        238, 109, 12, 239, 112, 12, 240, 116, 12, 240, 119, 12, 241, 123, 12,
        241, 127, 12, 242, 130, 12, 242, 134, 12, 243, 138, 12, 243, 141, 13,
        244, 145, 13, 244, 149, 13, 245, 152, 13, 245, 156, 13, 246, 160, 13,
        246, 163, 13, 247, 167, 13, 247, 171, 13, 248, 175, 14, 248, 178, 15,
        249, 182, 16, 249, 185, 18, 250, 189, 19, 250, 192, 20, 251, 196, 21,
        251, 199, 22, 252, 203, 23, 252, 206, 24, 253, 210, 25, 253, 213, 27,
        254, 217, 28, 254, 220, 29, 255, 224, 30, 255, 227, 39, 255, 229, 53,
        255, 231, 67, 255, 233, 81, 255, 234, 95, 255, 236, 109, 255, 238, 123,
        255, 240, 137, 255, 242, 151, 255, 244, 165, 255, 246, 179, 255, 248,
        193, 255, 249, 207, 255, 251, 221, 255, 253, 235, 255, 255, 24]

    def colormapChunk(ulist, step):
        return map(lambda i: ulist[i: i + step], range(0, len(ulist), step))

    chunks = colormapChunk(colormapIronBlack, 3)

    red = []
    green = []
    blue = []

    for chunk in chunks:
        red.append(chunk[0])
        green.append(chunk[1])
        blue.append(chunk[2])

    lut[:, 0, 0] = blue
    lut[:, 0, 1] = green
    lut[:, 0, 2] = red

    return lut

# Generate color map - used for colorizing the video frame.
colorMap = generateColourMap()


image_counter = 0
video = cv2.VideoCapture(0,cv2.CAP_DSHOW)
video.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter.fourcc('Y','1','6',' '))
video.set(cv2.CAP_PROP_CONVERT_RGB, 0)

if video.isOpened(): # try to get the first frame
    rval, frame = video.read()
else:
    rval = False

# Create an object for executing CLAHE.
clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8))

while rval:
    # Get a Region of Interest slice - ignore the last 3 rows.
    frame_roi = frame[:-3, :]

    # Normalizing frame to range [0, 255], and get the result as type uint8.
    normed = cv2.normalize(frame_roi, None, 0, 255, cv2.NORM_MINMAX, cv2.CV_8U)

    # Apply CLAHE - contrast enhancement.
    # Note: apply the CLAHE on the uint8 image after normalize.
    # CLAHE supposed to work with uint16 - you may try using it without using cv2.normalize
    cl1 = clahe.apply(normed)

    nor = cv2.cvtColor(cl1, cv2.COLOR_GRAY2BGR)  # Convert gray-scale to BGR (no really needed).
        
    colorized_img = cv2.LUT(nor, colorMap)  # Colorize the gray image with "false colors".

    cv2.imshow("preview", cv2.resize(colorized_img, dsize=(640, 480), interpolation=cv2.INTER_LINEAR))
    key = cv2.waitKey(1)
    if key == 27: # exit on ESC
        break

    # Grab the next frame from the camera.
    rval, frame = video.read()

Note:
The IR sensor is not a colored sensor.
Colorizing the frames uses "false colors" - colorizing may used for eustatic purposes.
The "false colors" has no physical meaning.
There are many ways to colorize an IR image, and there is no "standard colorization" method.

Thank you Rotem, Yes, it is impossible to provide an answer without the hardware unfortunately. But I appreciate your guidance, I'll test and keep you posted! Thank you again! — Darkgenius007, Apr 01 '21 at 21:36
Hi Rotem, so I tested with your code. A couple notes here: 1. video.read() inside while loop did fix the issue. I can stream the video as expected now 2. normed = cv2.normalize(frame, None, 0, 255, cv2.NORM_MINMAX, cv2.CV_8U) is showing fainted/grey screen — Darkgenius007, Apr 06 '21 at 16:16
You may try to apply [CLAHE](https://docs.opencv.org/master/d5/daf/tutorial_py_histogram_equalization.html), for improving the contrast. I updated the post for showing you how to use it. — Rotem, Apr 06 '21 at 19:40
Now that you have a `uint8` image, can you save an image after `cv2.normalize` and before `cv2.resize` (and before `clahe.apply`)? Can you take a photo of your hand, and add it to your post (please use PNG format). I think the resolution is going to be 160x120 — Rotem, Apr 06 '21 at 19:44
Hi Rotem, screenshot added. Not sure why was the color not converted from Grayscale to color (RGB) format.... — Darkgenius007, Apr 06 '21 at 20:09
I asked for an image not for a screenshot. You can add a temporary code line for saving an image after `normed = cv2.normalize`. Add the command: `cv2.imsave('normed.png', normed)` — Rotem, Apr 06 '21 at 20:23
The last 3 rows of the frame are "messing" the dynamic range (last 3 rows are rows of data, and not true pixels). Use: `frame_roi = frame[:-3, :]`. Please save and post `frame_roi`. (I updated my post to cut the last 3 rows). — Rotem, Apr 06 '21 at 21:12
Hi Rotem - now the last 3 rows of the frame are "missing" issue seemed to be addressed. But is it's still in greyscale, I'm hoping to convert to RGB888 so mapped to colors — Darkgenius007, Apr 06 '21 at 22:13
I found the solution for colorizing [here](https://groups.google.com/g/flir-lepton/c/Cm8lGQyspmk) — Rotem, Apr 06 '21 at 23:04
Hi Rotem, so if I understand correctly I would need to use both cv2.cvtcolor & cv2.LUT function to colorizing? Thank you again for all your help! — Darkgenius007, Apr 06 '21 at 23:24
I would appreciate if you could show me how the colorizing solution works along with the code you shared above please. Thank you! — Darkgenius007, Apr 07 '21 at 00:42
Thank you so much for all your help and guidance Rotem! Understood, since the RAW data is coming from the IR sensor, so we are doing pseudocoloring , in others words, applying fake colors I guess. Thank you sir! — Darkgenius007, Apr 07 '21 at 12:41
Rotem, one question though, what if I simply just use opencv's cv2.applyColorMap(nor, cv2.COLORMAP_JET) method? I actually did, but the colorization didn't seem to be correct — Darkgenius007, Apr 07 '21 at 12:45
There is no such thing as "correct colorization". I don't know what is the recommended color map (recommended by FLIR Systems). In my opinion, grayscale is more "correct". — Rotem, Apr 07 '21 at 13:36
Hi Rotem, one more question, I'm trying to print the pixel values so that I can convert to the centikelvin value. Will the below methods give me the right pixel readings? for i in range(frame.shape[0]): for j in range(frame.shape[1]): print (frame[i][j]) — Darkgenius007, Apr 07 '21 at 15:04
This code worked with FLIR Boson camera as well. If there is an error in `cv2.VideoCapture` or `video.set` Please consider `cv2.VideoCapture('/dev/video2')` or `video.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc('Y','1','6',' '))` — Cloud Cho, Apr 08 '21 at 22:07
@Cloud Cho, I used the above method to print the pixel value, I was wondering if the pixel value i'm getting is correct. Because when I used the formula as described in the engineering data I would be getting either a ridiculously high temperature or negative value, either one makes sense to me.... Please advise, thank you! — Darkgenius007, Apr 09 '21 at 16:00
@Darkgenius007 If you estimate temperature with the pixel value, do your mean your RGB value at the specific pixel (2nd snippet)? Would you share the equation? My guess at this moment, you may scale at the end. Have you tried calculate temperature before/without normalization nor Clahe (1st snippet)? By the way, I tried only gray scale one (1st snippet) for my camera. — Cloud Cho, Apr 09 '21 at 16:41
@CloudCho - Hi, i used the following method for i in range(frame.shape[0]): #for j in range(frame.shape[1]): print (frame.shape) basically I'm trying to read the pixel values of the 16bit RAW video frame, that's why it's frame.shape..... according to the SDK guide, "Each pixel represents the temperature at that point in centikelvin (cK)" that's why I'm trying to get the pixel values. And I don't know if the method of getting the pixel values is correct... thanks — Darkgenius007, Apr 10 '21 at 04:37
@Darkgenius007 Have you tried to read `frame` information at `rval, frame = video.read()`? When you check the matrix, it would be [width, height, channel] and the channel probably 3 (RGB). Did the SDK mention way to calculate these three R, G, B values? — Cloud Cho, Apr 10 '21 at 07:26
@Rotem - Hi Rotem, I tried to run print frame[i][j], but getting the following error: Traceback (most recent call last): File "C:/Users/T9003EH/.PyCharmCE2019.3/config/scratches/test2.py", line 28, in cl1 = clahe.apply(normed) cv2.error: OpenCV(4.5.1) C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-r2ue8w6k\opencv\modules\imgproc\src\clahe.cpp:353: error: (-215:Assertion failed) _src.type() == CV_8UC1 || _src.type() == CV_16UC1 in function '`anonymous-namespace'::CLAHE_Impl::apply' — Darkgenius007, Apr 12 '21 at 19:23
I apologize, but I am going to stop answering your comments from now on... — Rotem, Apr 12 '21 at 20:48
@Darkgenius007, please check `frame.shape`. I think you may getting empty `frame`. — Cloud Cho, Apr 13 '21 at 07:21
@CloudCho, since it's grayscale 16bit data should I use frame.size? - according to the OpenCV tutorial https://docs.opencv.org/master/d3/df2/tutorial_py_basic_ops.html I'll try both method and compare the results and share. Thank you CloudCho! — Darkgenius007, Apr 13 '21 at 14:07
@Darkgenius007, there should be some confusion. The frame size is automatically decided by `rval, frame = video.read()`. You could change the `frame` size after grep from the camera input. — Cloud Cho, Apr 13 '21 at 17:22
@CloudCho Hi, so I attached two screenshots. code to get the pixel values (vertical set of numbers) and I converted to celsius... please let me know your thoughts, thank you. — Darkgenius007, Apr 13 '21 at 18:05
@Darkgenius007 Thanks. I don't think you couldn't do `pixel/100` because pixel would be [R, G, B], which is a 1 by 3 matrix (verify by `print( pixel )` or `print( pixel.shape )`. — Cloud Cho, Apr 13 '21 at 21:04
@CloudCho, screenshot attached (print (pixel)) - remember frame = 16bit grayscale data. So R=G=B — Darkgenius007, Apr 14 '21 at 00:45
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/231090/discussion-between-darkgenius007-and-cloud-cho). — Darkgenius007, Apr 14 '21 at 02:38
@CloudCho Just curious, I'm trying to putTEXT (converted temperature) on the video using : cv2.putText(color, str(Temp_C), (20,20), cv2.FONT_HERSHEY_SIMPLEX, 0.01, (0,0,255),cv2.LINE_AA) But I'm i'm getting a red circle in the video.. any ideas? — Darkgenius007, Apr 14 '21 at 21:46

Luis M. Muñoz · Answer 2 · 2022-08-26T08:48:14.337

1

For colorizing you could try radiometric_image=cv2.applyColorMap(radiometric_image,cv2.COLORMAP_JET) Previously having a temperature data, modified to normalized data by radiometric_image=cam_source.raw_to_8bit(thermal_frame)

It works for me on Lepton 3.5 on Nano

edited Aug 26 '22 at 08:48

answered Aug 26 '22 at 08:46

Luis M. Muñoz

11
3

(please if someones see this upvote me, StackOverflow not allows me to publish answers anymore...) – Luis M. Muñoz Oct 18 '22 at 13:05

Thermal Image Processing

2 Answers2

Linked