0

I am displaying some very simple numpy array with openCV by using cv2.imshow() and cv2.waitKey(). Sometimes, I want a display to stay for a few seconds and then resume my program. I simply add a time.sleep() with the desired sleep value. Most of the time, everything works fine; however, sometimes the display doesn't change and remains on the previous one. It is very inconsistent. The same code might work 80% of the time, and might not work for the remaining 20%. Interestingly enough, it always seems to be the same image/display which doesn't work across multiple runs; although it does work as well sometimes. As I said, it's very inconsistent.

Here is a small portion of the code I used to create and show the displays:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import sys
import numpy as np
import cv2
import time

from matplotlib import colors

class Visual:
    def __init__(self, window, screen_size=None):
        self.window = window
        # screen size and message setting
        if screen_size is None:
            if sys.platform.startswith('win'):
                from win32api import GetSystemMetrics
                self.screen_width = GetSystemMetrics(0)
                self.screen_height = GetSystemMetrics(1)
            else:
                self.screen_width = 1024
                self.screen_height = 768
        else:
            self.screen_width, self.screen_height = screen_size
        self.screen_center = (self.screen_width//2, self.screen_height//2)
        
    def draw_background(self, color=(210, 210, 210)):
        if isinstance(color, str):
            r, g, b, _ = colors.to_rgba(color)
            color = [int(c*255) for c in (b, g, r)]
            
        self.img = np.full((self.screen_height, self.screen_width, 3), 
                           fill_value=color, dtype=np.uint8)
        
    def show(self, wait=1):
        cv2.imshow(self.window, self.img)
        cv2.waitKey(wait)
        
class CrossVisual(Visual):
    def __init__(self, window, screen_size=None):
        super().__init__(window, screen_size)
    
    def draw_cross(self, width_ratio=0.2, height_ratio=0.02, position='center', color='black'):
        if isinstance(color, str):
            r, g, b, _ = colors.to_rgba(color)
            color = [int(c*255) for c in (b, g, r)]
        
        rect_width = int(width_ratio*self.screen_width)
        rect_height = int(height_ratio*self.screen_height)
        
        if position == 'center':
            x = self.screen_center[0]
            y = self.screen_center[1]
        else:
            x, y = position
        
        # Rectangle 1
        xP1 = x - rect_width//2
        yP1 = y + rect_height//2
        xP2 = x + rect_width//2
        yP2 = y - rect_height//2
        cv2.rectangle(self.img, (xP1, yP1), (xP2, yP2), color, -1)
        
        # Rectangle 2
        xP1 = x - rect_height//2
        yP1 = y + rect_width//2
        xP2 = x + rect_height//2
        yP2 = y - rect_width//2
        cv2.rectangle(self.img, (xP1, yP1), (xP2, yP2), color, -1)
        
class TextVisual(Visual):
    def __init__(self, window, screen_size=None):
        super().__init__(window, screen_size)
        
    def putText(self, text, fontFace=cv2.FONT_HERSHEY_DUPLEX, 
                fontScale=2, color='white', thickness=2, vertical_offset=0):
        if isinstance(color, str):
            r, g, b, _ = colors.to_rgba(color)
            color = [int(c*255) for c in (b, g, r)]
        
        # Measuring text and choosing position on the screen
        textWidth, textHeight = cv2.getTextSize(text, fontFace, fontScale, thickness)[0]
        xtext = int(self.screen_center[0] - textWidth//2)
        ytext = int(self.screen_center[1] - textHeight//2 + vertical_offset)
        
        # print warnings
        if textWidth > self.screen_width or textHeight > (self.screen_height - ytext):
            print ('WARNING: The text and text settings lead to an output too large for the screen size.')
            
        cv2.putText(self.img, text=text, org=(xtext, ytext),
                    fontFace=fontFace, fontScale=fontScale, 
                    color=color, thickness=thickness, lineType=cv2.LINE_AA)

Sadly, I am unable to reproduce the problem with small piece of code. But I can demonstrate with the following function:

def display_3s_countdown(window, images=None):
    if images is None:
        images = list()
        
        Visual = TextVisual(window=WINDOW)
        Visual.draw_background(color='lightgrey')
        Visual.putText('3', color='black')
        images.append(Visual.img)
        
        Visual = TextVisual(window=WINDOW)
        Visual.draw_background(color='lightgrey')
        Visual.putText('2', color='black')
        images.append(Visual.img)
        
        Visual = TextVisual(window=WINDOW)
        Visual.draw_background(color='lightgrey')
        Visual.putText('1', color='black')
        images.append(Visual.img)
    
    for image in images:
        cv2.imshow(window, image)
        cv2.waitKey(1)
        time.sleep(1)

As you can see, this small function simply creates 3 images, each with a different number written, in order to create a small 3 seconds countdown. When I do something like:

v = TextVisual('test window')
v.draw_background(color='lightgrey')
v.putText('some random text', color='black')
time.sleep(3) # To let enough time to read the text

display_3s_countdown(window='test window', images=None)

Sometimes, the number 3 will not be displayed, and the display in the window test window will directly go from 'some random text' to the number 2.

Finally, usually, this behavior goes in pairs with a spinning wheel mouse when I have the mouse on the active cv2 window (only one window is opened at a time anyway). As you might have guessed from the header of my file, I am working on macOS. However, this behavior also appears on Windows.

I am not very familiar with openCV, thank you for any tip and information you could provide.

EDIT: Some of my displays are dynamic and are shown as soon as the image is created within a while loop. Thus, efficiency and speed in the display are important.

Mathieu
  • 5,410
  • 6
  • 28
  • 55
  • are you sure that your images dont share matrix element data? OpenCV at least in C++ is trying to not create new data for each matrix, but to reuse data. If you dont deep copy a matrix, the data is shared. In your code, if some matrices are sharing data, you would display the same same image multiple times and it will look like the last images that you prepared (among images with shared data). – Micka Dec 09 '20 at 11:56
  • can you try something like `images.append(Visual.img.deep-copy)`? – Micka Dec 09 '20 at 11:58
  • @Micka I am not sure I completely understand that. I guess you mean that when adding an element to an existing image, openCV won't recreate the matrix from scratch. Is that true also for the display of images, i.e. if 2 images share a common part, e.g. the background, then this part won't have to be reshown? In the 3s countdown example, the array (img attribute) is initialized by the draw_background() method which creates a background color with np.full(). I am creating 3 different objects, one for each second, thus the 3 matrix (which are the Visual.img) are different objects. – Mathieu Dec 09 '20 at 12:47
  • can you try to display all 3 images of your display_3s_countdown sample code in 3 different windows, just for testing? – Micka Dec 09 '20 at 13:06
  • @Micka Works as expected. 3 windows are created with the 3 digits, at a one-second interval. The weird thing is that the countdown function does work properly most of the time. Just as this function is not the only case where a display is skipped. e.g. I had a loop on 5 runs starting with instruction displayed at the beginning, then a fixation cross displayed, followed by about 20 seconds of sounds stimuli. For some reason, during one loop, the fixation cross did not display, and the display was stuck on the instructions of that loop until the next imshow call (instructions of next loop). – Mathieu Dec 09 '20 at 13:17
  • @Micka Just to complete my example with the 5 loops, the instruction were TextVisual objects (created each time inside the loop with the correct text); and the fixation cross was a CrossVisual object (only one created prior to the loop). – Mathieu Dec 09 '20 at 13:19
  • in the `display_3s_countdown` example, the countdown is correctly displayed (tried 10 times) on my machine in my environment (windows 10, anaconda). The 'some random text' is not displayed, since there is no imshow/waitKey in that sample code for that. – Micka Dec 09 '20 at 13:23
  • btw. if you are trying to use that for some psychological experiments and timing (like display/response times) is important, dont use opencv imshow. It might be faster than others, but still you dont have control about how fast imshow can render. In matlab there is a psych toolbox with specialized functions, afair. – Micka Dec 09 '20 at 13:25
  • @Micka Yes in my example I forgot to write down the v.show() for the random text. To me, it really feels like the PC was too busy and could not display the image (spinning wheel, busy mouse icon). But instead of delaying the display and catching up asap, the display is skipped and the program just continues. Thank you for the tip for the psych toolbox, I am aware of it. I do not need to control the timings of the display for my paradigms, thus openCV is good enough.. and all my code is in python anyway ;) – Mathieu Dec 09 '20 at 13:31
  • @Micka Is there a way to check if the display was updated? i.e. is there a way to acquire the matrix corresponding to the image displayed in an openCV window to then compare it to the expected matrix stored in the last Visual.img used? Then, if I notice that the display has not been updated, I could retry with imshow() until it is. That would be good enough for my use case. – Mathieu Dec 09 '20 at 13:34
  • can you try to remove the sleep and instead use waitKey(1000*sleep_time)? If it works that way, you can write some kind of loop to make sure that you waitKey long enough. I guess that waitKey(1) in combination with the sleep isnt working well together. I dont know for sure that there is some kind of cancelled rendering, if device is too busy, but if it is, then in your sleep it wont catch up, but in longer waitKeys it probably would. – Micka Dec 09 '20 at 13:46
  • @Micka It does look like this works; I had one script which was systematically skipping the first '3' display on the first of 2 countdowns. Removing the time.sleep() and using directly a wait time in the waitKey seems to work. I will have to use this fix a bit more to make sure it always work. However, doesn't it means that now if someone presses a key on the keyboard during this waiting time, the display will go to the next? It is not suppose to happen, but still.. Can I deactivate the key press action somehow? – Mathieu Dec 09 '20 at 13:53
  • you could use something like `start_time = now(); while(now() < start_time + duration): cv2.waitKey(1);` – Micka Dec 09 '20 at 14:52

2 Answers2

0

OpenCV's display of images has always been a real issue for me. Maybe I didn't start from the basics of OpenCV or what was the actual reason but instead I've found matplotlib really helpful here. The biggest tip I'd like to give you, and this is specifically in case of images: Use matplotlib's imshow instead.

My biggest reason for using matploblib for image display is because I don't want to open a new window each time to show the image(some might prefer that, like if you're working with scripts).

Instead of cv2.imshow() and in addition to that multiple other things that you'd have to write, much better is:

import matplotlib.pyplot as plt
plt.imshow(img)

Not just that, you can do so much other stuff here.

Like for removing axis, add:

plt.axis('off')

Or having multiple images as output, using subplot.

NOTE- Image output using matplotlib will be different in comparison to OpenCV's. One simple reason is that if image was read using OpenCV, the colour space of image would be BGR. While matplotlib reads images as RGB. Well for this you'd first need to convert the image to RGB before plotting using matplotlib. Or rather import image using matplotlib itself. Here's a great answer explaining the issue here.

Amit Amola
  • 2,301
  • 2
  • 22
  • 37
  • Matplotlib is way too slow compared to openCV. Some of my visuals (not shown in the piece of code above) are dynamic, i.e. they are displayed as soon as created inside a while loop. openCV is more efficient and allows a higher refresh rate which is needed in my usecase. – Mathieu Dec 09 '20 at 11:01
  • I see, my answer is completely irrelevant in that case. Apologies. – Amit Amola Dec 09 '20 at 11:02
  • You could not guess it based on my question. I would add it in edit. Thank you for the try :) Upvoted, this answer could help others with similar problems who do not need a high refresh rate. I am also usually a matplotlib user ;) – Mathieu Dec 09 '20 at 11:05
0

I would suggest that you not use time.sleep in GUI programs (OpenCV's imshow counts as a GUI).

work with the delay argument to waitKey(). pass 1000 for one second of maximum delay.

the GUI locks up if waitKey() isn't running enough. waitKey runs the event/message loop.

when you simply sleep for a second, the whole GUI doesn't get to do its vital housekeeping.

Christoph Rackwitz
  • 11,317
  • 4
  • 27
  • 36