36

What is the fastest way to display images to a Qt widget? I have decoded the video using libavformat and libavcodec, so I already have raw RGB or YCbCr 4:2:0 frames. I am currently using a QGraphicsView with a QGraphicsScene object containing a QGraphicsPixmapItem. I am currently getting the frame data into a QPixmap by using the QImage constructor from a memory buffer and converting it to QPixmap using QPixmap::fromImage().

I like the results of this and it seems relatively fast, but I can't help but think that there must be a more efficient way. I've also heard that the QImage to QPixmap conversion is expensive. I have implemented a solution that uses an SDL overlay on a widget, but I'd like to stay with just Qt since I am able to easily capture clicks and other user interaction with the video display using the QGraphicsView.

I am doing any required video scaling or colorspace conversions with libswscale so I would just like to know if anyone has a more efficient way to display the image data after all processing has been performed.

Thanks.

Jason B
  • 12,835
  • 2
  • 41
  • 43

3 Answers3

36

Thanks for the answers, but I finally revisited this problem and came up with a rather simple solution that gives good performance. It involves deriving from QGLWidget and overriding the paintEvent() function. Inside the paintEvent() function, you can call QPainter::drawImage(...) and it will perform the scaling to a specified rectangle for you using hardware if available. So it looks something like this:

class QGLCanvas : public QGLWidget
{
public:
    QGLCanvas(QWidget* parent = NULL);
    void setImage(const QImage& image);
protected:
    void paintEvent(QPaintEvent*);
private:
    QImage img;
};

QGLCanvas::QGLCanvas(QWidget* parent)
    : QGLWidget(parent)
{
}

void QGLCanvas::setImage(const QImage& image)
{
    img = image;
}

void QGLCanvas::paintEvent(QPaintEvent*)
{
    QPainter p(this);

    //Set the painter to use a smooth scaling algorithm.
    p.setRenderHint(QPainter::SmoothPixmapTransform, 1);

    p.drawImage(this->rect(), img);
}

With this, I still have to convert the YUV 420P to RGB32, but ffmpeg has a very fast implementation of that conversion in libswscale. The major gains come from two things:

  • No need for software scaling. Scaling is done on the video card (if available)
  • Conversion from QImage to QPixmap, which is happening in the QPainter::drawImage() function is performed at the original image resolution as opposed to the upscaled fullscreen resolution.

I was pegging my processor on just the display (decoding was being done in another thread) with my previous method. Now my display thread only uses about 8-9% of a core for fullscreen 1920x1200 30fps playback. I'm sure it could probably get even better if I could send the YUV data straight to the video card, but this is plenty good enough for now.

Jason B
  • 12,835
  • 2
  • 41
  • 43
  • I converted yuv packets to rgb with swscale in my decoder thread and used your method. running flawlessly. – baci Sep 10 '16 at 22:59
  • `p.SetRenderHint(QPainter::SmoothPixmapTransform, 1);` - setRenderHint should be with lowercase. – Otar Magaldadze Dec 20 '16 at 09:52
  • What is `QGLWidget` here Qt Designer says, no such file? – Bahramdun Adil Jun 08 '17 at 06:41
  • Nice example, saved me a lot of time!! – Bahramdun Adil Jun 08 '17 at 08:02
  • @BahramdunAdil, `QGLWidget` has been deprecated and was replaced by `QOpenGLWidget`. It is slightly different, but should work similarly for this simple example. – Jason B Jun 08 '17 at 11:39
  • @JasonB If I have an `AVFrame`, then the next step is to convert it to a `YUV 420P` and follow this procedure? Also, I've multiple streams to handle, would this method scale? 8-9% for a single stream is pretty high in say 16 simultaneous streams. – Sleeba Paul Apr 05 '19 at 06:39
  • could you please share the full working source, thank you – Lê Vũ Huy Jun 10 '21 at 16:15
  • @JasonB why GL widget? your code will work the same as just derived from QWidget. I think it is better to show the frame via OpenGL context and upload texture buffer from raw data. – iperov Aug 26 '21 at 08:47
3

I have the same problem with gtkmm (gtk+ C++ wrapping). The best solution besides using a SDL overlay was to update directly the image buffer of the widget then ask for a redraw. But I don't know if it is feasible with Qt ...

my 2 cents

neuro
  • 14,948
  • 3
  • 36
  • 59
3

Depending on your OpenGL/shading skills you could try to copy the videos frames to a texture, map the texture to a rectangle (or anything else..fun!) and display it in a OpenGL scene. Not the most straight approach, but fast, because you're writing directly into the graphics memory (like SDL). I would also recoomend to use YCbCR only since this format is compressed (color, Y=full Cb,Cr are 1/4 of the frame) so less memory + less copying is needed to display a frame. I'm not using Qts GL directly but indirectly using GL in Qt (vis OSG) and can display about 7-11 full HD (1440 x 1080) videos in realtime.

count0
  • 2,537
  • 2
  • 22
  • 30