6

I have a custom pipeline that looks roughly like this in gstreamer shorthand:

gst-launch-1.0 rtspsrc location=rtsp://<url-for-stream> ! rtph264depay ! h264parse ! imxvpudec ! *any-sink*
  • any-sink doesn't matter, could be fakesink, imxipusink, or whatever (I'm on imx6 platform using freescale imx plugins). I can output to whichever sink I want and the issue is the same.

This type of pipeline works fine in gst-launch-1.0 because it doesn't need to clean itself up properly, but I need to use it inside my C++ application using direct GST API. This means I use myPipeline = gst_pipeline_new("custom-pipeline"), then allocate each plugin by name, link them, and run the pipeline. I later have a requirement to stop the pipeline and call gst_object_unref(myPipeline). When doing this, I observe file descriptors being left behind. I later need to start the pipeline all over again, and so the leak is compounding. This needs to happen often enough that the leaking descriptors give me an exception:

GLib-ERROR **: Creating pipes for GWakeup: Too many open files

I can profile the open files with lsof...

lsof +E -aUc myGstApplication        
lsof: netlink UNIX socket msg peer info error
COMMAND    PID USER   FD   TYPE     DEVICE SIZE/OFF     NODE NAME
myGstApplication 5943 root    3u  unix 0xabfb6c00      0t0 11200335 type=STREAM
myGstApplication 5943 root   11u  unix 0xd9d47180      0t0 11207020 type=STREAM

... many more, depending on how long it runs...

myGstApplication 5943 root   50u  unix 0xabe99080      0t0 11211987 type=STREAM

I appear to get two new 'type=STREAM' file descriptors each time i unref() and rebuild the pipeline.

This is all fine and dandy to see the descriptors in lsof, but I don't know how to track down where these files are coming from in the code. Does any of the lsof output actually lead me to better debug information, for instance? . How do I track down where these leaks are really coming from and stop them? There has be be a better way... right?

I suspect rtspsrc gstreamer pipeline element is having something to do with this, but rtspsrc is itself a morass of underlying gstreamer implementation (udpsrcs, demuxers, etc, etc.) I'm not convinced that it's a bug within rtspsrc, because so many other people appear to use this one without reproducing the same thing. Could I be doing something in my application code that can bring about this behavior in a non-obvious way?

Any help is much appreciated, thanks!

adowdy
  • 329
  • 2
  • 16
  • Have you tried using the `GST_DEBUG` option, that provides debug related info for each element in the pipeline. – gst Oct 09 '18 at 09:31
  • Yes, the only thing I see there is this: `0:00:57.383872340 6101 0x69b34fb0 WARN GST_PADS gstpad.c:3990:gst_pad_peer_query: could not send sticky events 0:00:57.389069340 6101 0x69b34830 WARN GST_PADS gstpad.c:3990:gst_pad_peer_query: could not send sticky events` – adowdy Oct 09 '18 at 23:09
  • 2
    Had exactly same case. Mine was caused by not removing bus watch when freeing pipeline - remeber to do `gst_bus_remove_watch` if you do `gst_bus_add_watch` before. – Raber Jan 23 '20 at 08:47

1 Answers1

4

Well researched & interesting question!

According to the lsof output, the leaking file descriptors seem to originate from socketpair syscalls. You confirm this with strace:

strace -fe socketpair myGstApplication

After this, you could leave out filtering for the socketpair sycall, and look through the full strace output, trying to understand what these FDs are used for. I tried this with gst-launch-1.0, with inconclusive results. These FDs seem to be set readonly on both ends and nothing is ever transferred... so they must be used for control/coordination between several threads/subprocesses of the same application only.

Next try would be gdb:

gdb -ex 'break socketpair' -ex run myGstApplication

When it halts at the breakpoint, look at the stacktrace with the bt command. Probably installing the debug packages of gstreamer is a good idea to get more readable stacktraces.

HTH :)

micha137
  • 1,195
  • 8
  • 22
  • Thanks micha -- I was already starting to play with strace, but you've given me more context to play around with. Will follow up again! – adowdy Oct 15 '18 at 23:54
  • I've identified a gst func that calls socketpair which is leaking. the call `gst_poll_new_timer()` seems to be the one creating the socketpair as shown in GDB, but i haven't been successful at getting a stack trace: Thread 1 hit Breakpoint 1, 0x6b07b0bc in gst_poll_new_timer () from target:/usr/lib/libgstreamer-1.0.so.0 (gdb) bt #0 0x6b07b0bc in gst_poll_new_timer () from target:/usr/lib/libgstreamer-1.0.so.0 #1 0x6b096960 in ?? () from target:/usr/lib/libgstreamer-1.0.so.0 Backtrace stopped: previous frame identical to this frame (corrupt stack?) ...with debug packages AFAIK – adowdy Oct 20 '18 at 01:09
  • scratch my previous comment - I need to reconfigure my gdb environment based on playing ball with 'yocto', because without doing it their way, I think I'm getting bad information. Still working toward those readible stack traces... it's a terrible pain – adowdy Oct 20 '18 at 04:15
  • Just one more update for posterity - this method worked well enough for me to track down the root of the problem -- usage of `gst_element_get_bus(my_pipeline)` was not correctly decremented in reference counting mechanism. Tearing down the pipeline, I also had to refine the code to decrement the bus reference when creating a new pipeline via `gst_object_unref(GST_OBJECT(m_bus))` – adowdy Oct 31 '18 at 23:06
  • hi @adowdy, I'm having a similar issue... could you post a little more detail on the code changes you had to make? it would help me a lot. thanks! – textral Mar 21 '23 at 11:31