2

I am a newbie in the field and don't know if i am doing this right. So I need help with architectural and design issue, just to make sure if I am moving in right direction.

I am trying to process a real time video capturing from the client's camera. The captured video stream is converted into frames and send to the server for processing. After processing, server is sending it back to client's screen.

My backend code is written in Python and I am using SocketIo to send the frames from frontend to backend. You can have a look at this design to get a better idea about what's happening - image

  1. My server(app.py) will be running in backend and client will be accessing index.html
  2. SocketIo connection will get establish and video stream captured using webcam will be send to server frames by frames.
  3. These frames will be then processed at the backend and emit back to the client.
  4. Processed frames coming form the server can be shown in img tag.

So, my problem with this architecture is the lag. There is a noticeable delay even after limiting fps. From client to server then processing and back to client, each frame is take on an average 150 ms and is not giving a real time response. I have tried my image processing code separately with no socket programming on local and it is more near to real time (it is taking 80 ms for one frame).

But I fail to understand here that how can I minimize lagging with this design or improvise it? Is there any better way to do this task to get more real time results?

Thanks in advance!

akan
  • 421
  • 1
  • 4
  • 13
  • The code I am following is - https://stackoverflow.com/a/59998983/8030133 – akan Jan 31 '20 at 07:36
  • Does the server process the image in Python? If so, you should consider switching the server code to C++. Python code runs much slower than C++ for OpenCV tasks. – karlphillip Jan 31 '20 at 10:46

1 Answers1

1

Python is a great language for rapid prototyping Computer Vision applications. However, it is not recommended for production code because of its performance. Convert the code to C++ and you'll see significant improvement in runtime performance.

SocketIo its not the best option since it uses a TCP connection. You'll notice better performance from a library that transfers data through UDP. You might loose a few pixels here and there, but it's usually a good trade-off for streaming videos.

Here's a few more thoughts on SocketIo and TCP vs UDP..

karlphillip
  • 92,053
  • 36
  • 243
  • 426
  • Of course, C++ can give a boost over python. If no option, I will convert my code. But apart from the image processing in opencv, I was also concern about SocketIO. The time taken in sending and rendering result back is also taking pretty much noticeable time, which I think is also cause of lag. Can you please also help me in that direction. How can I debug into that? – akan Feb 03 '20 at 08:00
  • Updated answer. That's all I can offer at the moment. Good luck! – karlphillip Feb 03 '20 at 09:27
  • @akan There is no need to thank people since you can up-vote answers that are helpful. You might not have enough reputation on the site to do that right now, but remember to come back in the future. See you around, cheers! – karlphillip Feb 03 '20 at 11:50