9

I have a situation where I have multiple cameras (rtspsrc), and a singleton element, that does analytics on the incoming video stream. I call it a singleton element, because it has request source and sink pads. Only one of them should exist in the application, because it does it's work on the GPU, and can get better performance by doing things in batch. Think of the application I'm building as an API to add cameras, remove cameras, turn analytics on and off per camera, etc. Cameras will have analytics done on them, capturing the results, and sending them onwards. The complication being, I need to share a Gstreamer element (the analytics element).

So I have multiple cameras, feeding into this single element, then feeding out, into appsinks. This works reasonably well, but I want to be able to:

  • Pause a specific camera
  • Have each rtspsrc be completely isolated, so errors in one, don't affect the entire pipeline
  • Listen for events on a particular camera

If I have all the cameras in a pipeline together, I cannot figure out how to pause a specific camera. I cannot pause the entire pipeline, because that will stop all cameras. The best I've come up with is to remove and unlike the elements for a specific cameras, then when resuming, re-add and re-link. This works sort of. If a specific rtspsrc stops responding, then the entire pipeline stops. If a specific rtspsrc doesn't exist then the entire pipeline won't transition to PLAYING state

How should I architect my application? Do you think I should have a single big pipeline? Or should I have a pipeline containing the singleton analytics element, and a pipeline per camera, then connect them using appsink and appsrc? This approach might make it easier to handle things, as each pipeline is entirely separate?

Let me know if you need more info.

Dominic Bou-Samra
  • 14,799
  • 26
  • 100
  • 156
  • I would probably split this in 2 separate pipelines with a mix of session management logic in between. First pipeline to get the rtsp stream and play on appsink (session mgmt code). Then session mgmt will route the data through appsrc to next pipeline. – manishg May 13 '19 at 15:03
  • Are you developing plugins on your own or can you only use existing gstreamer plugins? And are we talking about typical video stuff like fullhd, <=30fps per input stream or is it more special? – Harry May 14 '19 at 09:28
  • @Harry We have a suite of 10+ custom plugins. We are talking 4k streams, but low FPS. Under 5 fps generally. – Dominic Bou-Samra May 14 '19 at 17:02

2 Answers2

3

Monolithic architectures should generally be avoided in programming, and your scenario is no exception to this. You have already experienced some complications of managing everything in one pipeline, and the found workarounds will likely cause more problems down the road, plus they do not provide convenient access for managing each camera.

I would therefore recommend to take the second approach to have a pipeline per camera, and additionally implement queues for buffering with an architecture similar to the one from this SO answer. You may also want to ensure your singleton is thread safe to avoid any race conditions between the pipelines when analytics are sent from the cameras.

Siavas
  • 4,992
  • 2
  • 23
  • 33
  • 1
    I'd even go one step further and introduce separate processes: one to decode from each source and one for the processing "singleton" exchanging their data using shared memory and using a socket control channel (or an embedded webserver). This way a crash or hang in one of the recording processes does not stop the entire processing chain. – Harry May 21 '19 at 07:06
  • Thanks all. This is the approach I have gone with. I have a proof of concept, and it works well. The complexity at the edges is much much less. – Dominic Bou-Samra May 21 '19 at 12:20
2

Given your requirements, I would build most of the API, camera management and GUI in c#, with an MVVM pattern and a DI container, so you decouple as much as possible your API parts, and make them as testable as possible. One other motivation is that it's very quick to produce a UI with this ecosystem (C#, Visual Studio); also for most projects, you know that maintenance will be the main cost Development cost versus maintenance cost, hence decoupling and testing against interfaces is excellent to keep those costs as low as possible.; MVVM will allow you to test your UI Writing a Testable Presentation Layer with MVVM. Decoupling your software components will also allow you to upgrade a certain implementation without touching the rest, and compose your software with its components in the composition root. Often such practices allow you to start with tests (TDD).

I would make sure to have 1 pipeline per camera, to simplify resource management, and if you use cudastreams (cudastreams to simplify concurrency), you can have multiple tasks of video analytics on one GPU, each stream doing the analytics of one camera video stream. You might want to use some proven code from opencv and make sure that it is transposable in a cudastream. If the amount of data, your performance requirements and your hardware won't need/allow such thing, you can just use one opencv processing.

On the native part, (gstreamer), it's relatively easy to interface your components with c# with interop; for instance:

extern "C" __declspec(dllexport) auto Query(myClass* p, const wchar_t* somePath)
    -> structResult*
{ return p->Query(somePath); }

and on the managed part:

[DllImport("myAssembly.dll", CallingConvention = CallingConvention.Cdecl)]
internal static extern IntPtr Query(IntPtr myClassPointer, IntPtr somePath);
Soleil
  • 6,404
  • 5
  • 41
  • 61