1

I'm trying to stream video over a websocket and process it server-side with Node.JS. The client is reading from a video file (.mp4) and sending it over the web socket via a stream object. However, I'm having trouble extracting frames from the stream at the server so it can be processed by opencv.

Do I need to break the video up into frames and stream each individual frame? What format can opencv most easily process in real time?

The end goal here is to enable opencv to process each frame of a video (in real time) that is being received by the server. Think I'm having some trouble understanding the paradigm here.

Patrick Connors
  • 5,677
  • 6
  • 27
  • 54
  • Don't use a pseudo streaming format like mp4. Use a streaming format like transport streams, or flv. – szatmary Apr 25 '19 at 19:17
  • Easy enough - what are the advantages of this? @szatmary – Patrick Connors Apr 25 '19 at 19:20
  • The advantage is that the format is streamable, and does not require a full frame index be present in memory to locate each frame offset and size like mp4 does. – szatmary Apr 25 '19 at 19:22
  • So in other words the frames are all the same size? Are you familiar with any tools that can read individual frames into memory? – Patrick Connors Apr 25 '19 at 19:23
  • No, in a streamable format, each frame is tagged with its own timing and size information. Mp4 indexes all sizes and timestamps in one place at the end (or beginning if fast start is applied) of the file. – szatmary Apr 25 '19 at 19:25
  • `Read frames into memory` Yes, posix read(). But I assume you are asking about a parser? – szatmary Apr 25 '19 at 19:27
  • Exactly, I'm consuming the stream in Buffer objects that are being sent via websocket. I'm having trouble parsing them to reconstruct a part of the whole video (if I wait for the whole stream to finish I can reconstruct the whole file). – Patrick Connors Apr 25 '19 at 19:31
  • libavformat from ffmpeg is the most comprehensive collection of parsers. – szatmary Apr 25 '19 at 19:33
  • @PatrickConnors there is a new MP4 format fMP4 (fragmented MP4), which I think does exactly what you need: https://stackoverflow.com/questions/35177797/what-exactly-is-fragmented-mp4fmp4-how-is-it-different-from-normal-mp4. You could use FFmpeg to convert your MP4 to fMP4. Could you solve this? – Oliver Hausler Feb 22 '20 at 15:53

0 Answers0