Sending RGB image data from Python numpy to a browser HTML page

Question

I need to send realtime image RGB data (in Numpy format) to a HTML page in a browser (web-based GUI), through HTTP. The following code works with the well-known multipart/x-mixed-replace trick: run this and access http://127.0.0.1:5000/video_feed: you will see a video in the browser.

from flask import Flask, render_template, Response
import numpy as np, cv2
app = Flask('')
def gen_frames():  
    while True:
        img = np.random.randint(0, 255, size=(1000, 1000, 3))
        ret, buf = cv2.imencode('.jpg', img)
        frame = buf.tobytes()
        yield (b'--frame\r\nContent-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')
@app.route('/video_feed')
def video_feed():
    return Response(gen_frames(), mimetype='multipart/x-mixed-replace; boundary=frame')
app.run()

However, according to my benchmark, the real performance bottleneck is the cv2.imencode('.jpg', img).

In my real application, if I just generate the image, the CPU is ~ 1% for Python. When I imencode(...), the CPU jumps to 25%, and 15% for Chrome.

I also tried with PNG format but it's similar.

Question: how to efficiently send RGB image data from a numpy array (example: 1000 x 1000 pixels x 3 colors because of RGB) to a browser HTML page?

(Without compression/decompression it might be better, but how?)

Here is the benchmark

          FPS       CPU PYTHON      CPU CHROME
PNG       10.8      20 %            10 %
JPG       14        23 %            12 %
JPG       10.7      16 %            10 %           (with time.sleep to match PNG 10.8 fps)
BMP       19        17 %            23 %
BMP       10.8      8 %             12 %           (with time.sleep to match PNG 10.8 fps)

Have you tried with ".bmp"? https://answers.opencv.org/question/207286/why-imencode-taking-so-long/ — smcrowley, Jun 21 '22 at 16:03
@smcrowley Good idea, I tried with BMP too, I added the benchmark at the end of the question. I wonder if it's possible to do better, and send the raw bytes (24 bit per pixel) and let Javascript handle the decoding. Any idea? — Basj, Jun 22 '22 at 07:09
Kind of surprising that chrome went up when using bmp. If you want to send the bytes themselves, I would just avoid using opencv. Convert the numpy matrix directly to bytes using .toBytes() method. That feels wrong though, since the whole point of including opencv is likely to do image processing, which requires encoding the numpy matrix into an image. If the goal is to have some generated image streamed via flask, I'd take out opencv If the goal is to process some image (or process some generated image) opencv is still great at this, and the jump in processing has to be accepted. — smcrowley, Jun 23 '22 at 18:42
cont.. If the goal is to do basic image processing (high light areas that are always the same, include text on the image, ...) then I would just make method to modify the numpy matrix directly before converting to bytes. What are the two different bmp benchmarks you have? the second one looks like what would be expected — smcrowley, Jun 23 '22 at 18:44
See also: [Chrome + another process: interprocess communication faster than HTTP / XHR requests?](https://stackoverflow.com/questions/72731175/chrome-another-process-interprocess-communication-faster-than-http-xhr-requ) — Basj, Jun 25 '22 at 08:51
It looks to me as the requirement is not to send one image but many. Is streaming of images not what video streaming formats were designed for? So what about AVI or MP4? And BTW, the video link points to localhost. No video for me. — Queeg, Jul 01 '22 at 09:29
@HiranChaudhuri Yes but compressing in MP4 in realtime is surely CPU-heavy as well. (Because the input images are collected in realtime) — Basj, Jul 01 '22 at 09:30
Ok, then a simpler format that maybe just transfers differences in RGB data so you save on network bandwitdth without stressing too much CPU... — Queeg, Jul 01 '22 at 09:32

score 0 · Answer 1 · answered Jun 26 '22 at 14:59

Try using the PILLOW module instead and see if that improves performance. My benchmark shows that each iteration of the gen_frames() generator function based on PILLOW requires less than half the CPU of the CV2 version.

from flask import Flask, render_template, Response
from PIL import Image
import numpy as np
from io import BytesIO

app = Flask('')

def gen_frames():
    while True:
        img = np.random.randint(0, 255, size=(1000, 1000, 3), dtype=np.uint8)
        rgb_image = Image.fromarray(img, 'RGB')
        buf = BytesIO()
        rgb_image.save(buf, 'JPEG')
        frame = buf.getbuffer()
        yield (b'--frame\r\nContent-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')

@app.route('/video_feed')
def video_feed():
    return Response(gen_frames(), mimetype='multipart/x-mixed-replace; boundary=frame')

app.run()

score 0 · Answer 2 · answered Jun 28 '22 at 12:52

According to the docs you could check if the default optimization is turned on:

Default Optimization in OpenCV

Many of the OpenCV functions are optimized using SSE2, AVX, etc. It contains the unoptimized code also. So if our system support these features, we should exploit them (almost all modern day processors support them). It is enabled by default while compiling. So OpenCV runs the optimized code if it is enabled, otherwise it runs the unoptimized code. You can use cv.useOptimized() to check if it is enabled/disabled and cv.setUseOptimized() to enable/disable it.

So try this:

In [5]: cv.useOptimized()         # check for optimization
Out[5]: False
In [7]: cv.setUseOptimized(True)  # turn it on if not already turned on

score 0 · Answer 3 · answered Jul 01 '22 at 13:21

As it stands AVI or MP4 compression would be good quality even for movies, but the compression itself takes too much CPU time to perform it on live data.

If some arbitrary protocol/format were created, one would not just have to program the server but also the client to consume this protocol/format. Therefore still some standard solution should be preferred.

I believe you can find a compromise between compression and CPU load in video conferencing systems, where the live camera data needs to be compressed and streamed via the network. With that in mind I believe here are sources of information that can help pursuing the topic:

https://www.reachcambridge.com/wp-content/uploads/wp_videocompression_33085_en_0809_lo.pdf
https://cs.haifa.ac.il/~nimrod/Compression/Video/V3h261-2005.pdf
Open Source Video Conferencing software like Jitsi Meet or Apache OpenMeetings.

score 0 · Answer 4 · answered Jul 01 '22 at 16:29

0

Maybe you can try encoding it as base64 for compression of the video/image and then send it to the browser with the base64 mime type e.g.data:image/jpeg;base64.

answered Jul 01 '22 at 16:29

Carlon van spijker

11

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jul 03 '22 at 04:12

Sending RGB image data from Python numpy to a browser HTML page

4 Answers4

Linked