4

I'm streaming a webcam from a browser using webrtc to a server where the following setup works:

  1. using firefox and a modified echo-test html from janus gateway I send the webcam stream to a janus server
  2. the janus server is running using a modified echotest plugin which simply udp-streams the given char *buf in janus_videorecv_incoming_rtp() to port 5060, just for testing purpose (pretty much like this)
  3. the following gstreamer command line actually opens a window showing the streaming video

GST_DEBUG=p*:5 gst-launch-1.0 -vvv udpsrc caps="application/x-rtp,media=video,clock-rate=90000,payload=96" port=5060 ! rtpvp8depay ! vp8dec ! autovideosink

in the modified echo test javascript i remove a few lines from the sdp answer the browser will receive like the following:

//jsep.sdp = jsep.sdp.replace(/a=rtcp-mux[^\s]*\s*/g, '');
jsep.sdp = jsep.sdp.replace(/a=rtpmap[^\s]*\s*red[^\s]*\s*/g, '');
jsep.sdp = jsep.sdp.replace(/a=rtpmap[^\s]*\s*ulpfec[^\s]*\s*/g, '');
jsep.sdp = jsep.sdp.replace(/a=fmtp[^\r\n]*\r*\n*/g, '');
jsep.sdp = jsep.sdp.replace(/a=rtcp-fb[^\s]*\s*goog-remb[^\s]*\s*/g, '');

below, one can find the modified firefox sdp answer which works for above gstreamer command but the, in the same way, modified sdp answer doesnt work in case of chrome i thought about adjusting the payload in the gstreamer caps, but 32,33,96,100,120 didnt work

so the question is: what is needed in case of chrome to get this to work?

i also tried adding fir/pli requests like in videoroom.c from janus as suggested here

the gstreamer output in case of chrome is, where the command just keeps waiting at the last line:

0:00:00.025791492 22279      0x1954b90 DEBUG               pipeline gstpipeline.c:219:gst_pipeline_init:<GstPipeline@0x1962180> set bus <bus2> on pipeline
Setting pipeline to PAUSED ...
0:00:00.029798090 22279      0x1954b90 DEBUG               pipeline gstpipeline.c:282:reset_start_time:<pipeline0> reset start_time to 0
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstUDPSrc:udpsrc0.GstPad:src: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, payload=(int)96, encoding-name=(string)VP8-DRAFT-IETF-01
Setting pipeline to PLAYING ...
0:00:00.030045034 22279      0x1954b90 DEBUG               pipeline gstpipeline.c:377:gst_pipeline_change_state:<pipeline0> selecting clock and base_time
0:00:00.030053523 22279      0x1954b90 DEBUG               pipeline gstpipeline.c:398:gst_pipeline_change_state:<pipeline0> Need to update start_time
0:00:00.030058181 22279      0x1954b90 DEBUG               pipeline gstpipeline.c:403:gst_pipeline_change_state:<pipeline0> Need to update clock.
/GstPipeline:pipeline0/GstRtpVP8Depay:rtpvp8depay0.GstPad:src: caps = video/x-vp8, framerate=(fraction)0/1
/GstPipeline:pipeline0/GstVP8Dec:vp8dec0.GstPad:sink: caps = video/x-vp8, framerate=(fraction)0/1
0:00:00.030111345 22279      0x1954b90 DEBUG               pipeline gstpipeline.c:443:gst_pipeline_change_state:<pipeline0> start_time=0:00:00.000000000, now=33:52:04.529345754, base_time 33:52:04.529345754
/GstPipeline:pipeline0/GstRtpVP8Depay:rtpvp8depay0.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, payload=(int)96, encoding-name=(string)VP8-DRAFT-IETF-01
New clock: GstSystemClock

chrome answer:

v=0
o=- 8913399741269897639 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE audio video
a=msid-semantic: WMS janus
m=audio 1 RTP/SAVPF 111 103 104 0 8 106 105 13 126
a=mid:audio
c=IN IP4 192.168.0.1
a=sendrecv
a=rtcp-mux
a=ice-ufrag:l0n9
a=ice-pwd:r1elT1Ew8lP3TNlzwAHUsC
a=ice-options:trickle
a=fingerprint:sha-256 C5:5F:DA:7D:84:47:B1:BF:6B:55:16:62:48:31:3E:D3:F1:7B:25:89:92:4A:4B:4D:4D:D9:D5:AF:EA:D8:15:44
a=setup:active
a=connection:new
a=rtpmap:111 opus/48000/2
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:126 telephone-event/8000
a=maxptime:60
a=ssrc:600390024 cname:janusaudio
a=ssrc:600390024 msid:janus janusa0
a=ssrc:600390024 mslabel:janus
a=ssrc:600390024 label:janusa0
a=candidate:1 1 udp 2013266431 192.168.0.1 45728 typ host
m=video 1 RTP/SAVPF 100 116 117 96
a=mid:video
c=IN IP4 192.168.0.1
a=sendrecv
a=rtcp-mux
a=ice-ufrag:l0n9
a=ice-pwd:r1elT1Ew8lP3TNlzwAHUsC
a=ice-options:trickle
a=fingerprint:sha-256 C5:5F:DA:7D:84:47:B1:BF:6B:55:16:62:48:31:3E:D3:F1:7B:25:89:92:4A:4B:4D:4D:D9:D5:AF:EA:D8:15:44
a=setup:active
a=connection:new
a=rtpmap:100 VP8/90000
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=rtpmap:96 rtx/90000
a=ssrc-group:FID 3188003624 3419969288
a=ssrc:677441062 cname:janusvideo
a=ssrc:677441062 msid:janus janusv0
a=ssrc:677441062 mslabel:janus
a=ssrc:677441062 label:janusv0
a=candidate:1 1 udp 2013266431 192.168.0.1 45728 typ host
m=application 0 DTLS/SCTP 0
c=IN IP4 192.168.0.1

firefox answer:

v=0
o=Mozilla-SIPUA-32.0.3 11426 0 IN IP4 127.0.0.1
s=SIP Call
t=0 0
a=group:BUNDLE audio video
a=msid-semantic: WMS janus
m=audio 1 RTP/SAVPF 109 0 8 101
a=mid:audio
c=IN IP4 192.168.0.1
a=sendrecv
a=rtcp-mux
a=ice-ufrag:BRBU
a=ice-pwd:2W4fGNr//HejhiC4UIabW6
a=ice-options:trickle
a=fingerprint:sha-256 C5:5F:DA:7D:84:47:B1:BF:6B:55:16:62:48:31:3E:D3:F1:7B:25:89:92:4A:4B:4D:4D:D9:D5:AF:EA:D8:15:44
a=setup:active
a=connection:new
a=rtpmap:109 opus/48000/2
a=ptime:20
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=ssrc:3725983979 cname:janusaudio
a=ssrc:3725983979 msid:janus janusa0
a=ssrc:3725983979 mslabel:janus
a=ssrc:3725983979 label:janusa0
a=candidate:1 1 udp 2013266431 192.168.0.1 56574 typ host
m=video 1 RTP/SAVPF 120
a=mid:video
c=IN IP4 192.168.0.1
a=sendrecv
a=rtcp-mux
a=ice-ufrag:jZ5b
a=ice-pwd:dQQej9UIpPl5zuXBQNg3Nz
a=ice-options:trickle
a=fingerprint:sha-256 C5:5F:DA:7D:84:47:B1:BF:6B:55:16:62:48:31:3E:D3:F1:7B:25:89:92:4A:4B:4D:4D:D9:D5:AF:EA:D8:15:44
a=setup:active
a=connection:new
a=rtpmap:120 VP8/90000
a=rtcp-fb:120 nack
a=rtcp-fb:120 nack pli
a=rtcp-fb:120 ccm fir
a=ssrc:1425382999 cname:janusvideo
a=ssrc:1425382999 msid:janus janusv0
a=ssrc:1425382999 mslabel:janus
a=ssrc:1425382999 label:janusv0
a=candidate:2 1 udp 2013266431 192.168.0.1 39063 typ host
m=application 0 DTLS/SCTP 0
c=IN IP4 192.168.0.1

UPDATE: i modified the sdp-answer so both firefox and chrome get nearly the same except for the "o=" and "s=" lines which i just copy from the sdp-offer v=0 o=- 7589782217972865757 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE audio video a=msid-semantic: WMS janus m=audio 1 RTP/SAVPF 111 a=mid:audio c=IN IP4 192.168.0.1 a=sendrecv a=rtcp-mux a=ice-ufrag:g0kZ a=ice-pwd:d5oEody1jqIzDYUdf1fg6t a=ice-options:trickle a=fingerprint:sha-256 C5:5F:DA:7D:84:47:B1:BF:6B:55:16:62:48:31:3E:D3:F1:7B:25:89:92:4A:4B:4D:4D:D9:D5:AF:EA:D8:15:44 a=setup:active a=connection:new a=rtpmap:111 opus/48000/2 a=ssrc:1038736511 cname:janusaudio a=ssrc:1038736511 msid:janus janusa0 a=ssrc:1038736511 mslabel:janus a=ssrc:1038736511 label:janusa0 a=candidate:1 1 udp 2013266431 192.168.0.1 51232 typ host m=video 1 RTP/SAVPF 100 a=mid:video c=IN IP4 192.168.0.1 a=sendrecv a=rtcp-mux a=ice-ufrag:g0kZ a=ice-pwd:d5oEody1jqIzDYUdf1fg6t a=ice-options:trickle a=fingerprint:sha-256 C5:5F:DA:7D:84:47:B1:BF:6B:55:16:62:48:31:3E:D3:F1:7B:25:89:92:4A:4B:4D:4D:D9:D5:AF:EA:D8:15:44 a=setup:active a=connection:new a=rtpmap:100 VP8/90000 a=rtcp-fb:100 ccm fir a=rtcp-fb:100 nack a=rtcp-fb:100 nack pli a=rtcp-fb:100 goog-remb a=ssrc:2455978689 cname:janusvideo a=ssrc:2455978689 msid:janus janusv0 a=ssrc:2455978689 mslabel:janus a=ssrc:2455978689 label:janusv0 a=candidate:1 1 udp 2013266431 192.168.0.1 51232 typ host m=application 0 DTLS/SCTP 0 c=IN IP4 192.168.0.1

mpromonet
  • 11,326
  • 43
  • 62
  • 91
John Doe
  • 2,746
  • 2
  • 35
  • 50
  • In chrome, not all the mappings are removed. Why are you removing `a=rtcp-fb:100 goog-remb`? Correct the rtp payloads displayed on the video media line and remove the uneeded `rtx` rtp mapping. That should get your further down the road. Are you sure that chrome is even sending you any media? I am guessing it freaks out on the answer and does not send anything. If it does send media, you can break the rtp header down and see the payload to make sure it is VP8 – Benjamin Trent Sep 26 '14 at 18:15
  • i actually removed goog-remb, cause i tried to remove lines by line to maybe get something working. actually chrome is sending data and my janus plugin also forwards correctly as i can see via wireshark. regarding "correct rtp payloads in video media line", do you mean that i should set m=video 1 RTP/SAVPF 120 as in firefox? what exactly do you mean by breaking down rtp header? how would you do it? thanks for your time – John Doe Sep 26 '14 at 18:44
  • the line should be `m=video 1 RTP/SAVPF 100` to indicate only one mapping(for VP8). Then only the VP8 mapping should exist. [Some webrtc sdp pointers](http://webrtchacks.com/sdp-anatomy/). You can cast the buffer as an rtp packet and then access the type. `rtp_header* packetheader = (rtp_header*) buf; packetheader->type; //payload` – Benjamin Trent Sep 26 '14 at 19:00
  • the m line is now fixed, i also copied the sdp_template from janus_videoroom.c plugin with a few tweaks. now i have a the equal sdp answer's for firefox and chrome. firefox is working with above gstreamer, chrome not. packetheader->type has the value 100 or 111 - as expected for video or audio as i understand. also rtx mapping is removed – John Doe Sep 26 '14 at 19:34
  • Well, if you are receiving a video feed with payload 100 for video, it is VP8 alright. My guess is the key frame issue with chrome is happening. Also, try changing your receiving Gstreamer pipeline to `udpsrc port=5060 caps="application/x-rtp, media=video, clock-rate=90000, encoding-name=VP8-DRAFT-IETF-01, payload=100" ! rtpvp8depay ! vp8dec ! ffmpegcolorspace ! autovideosink` – Benjamin Trent Sep 26 '14 at 19:38
  • i noticed that there seems to be a huge difference between gst-launch-0.10 and gst-launch-1.0 - if i use your line without ffmpegcolorspace on 0.10 everything works on FF, on 1.0 i see just the first parsed frame but no stream. using 1.0 with chrome nothing happens as mentioned in my original question. using 0.10 with chrome i get an error `corrupt frame` – John Doe Sep 26 '14 at 20:22
  • /GstPipeline:pipeline0/GstVP8Dec:vp8dec0.GstPad:sink: caps = video/x-vp8, framerate=(fraction)0/1 /GstPipeline:pipeline0/GstVP8Dec:vp8dec0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)I420, width=(int)1280, height=(int)720, framerate=(fraction)0/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false `ERROR: from element /GstPipeline:pipeline0/GstVP8Dec:vp8dec0: `Failed to decode frame Additional debug info: gstvp8dec.c(469): gst_vp8_dec_handle_frame (): /GstPipeline:pipeline0/GstVP8Dec:vp8dec0: corrupt frame – John Doe Sep 26 '14 at 20:23
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/62012/discussion-between-john-doe-and-benjamin-trent). – John Doe Sep 26 '14 at 20:26

3 Answers3

1

WebRTC has mandatory encryption with DTLS-SRTP (Chrome still supports the non-standard and explicit MUST-NOT-IMPLMENT SDES keying).

You can't just feed an RTP stream to webrtc; it must be a DTLS-SRTP stream keyed with an initial DTLS connection.

People have hooked node.js to webrtc browsers, so I imagine all the machinery you need is there.

jesup
  • 6,765
  • 27
  • 32
  • i'm not sure i understand you right. do you mean with "can't just feed an rtp stream to webrtc" the situation where i send rtp from the server to the browser? if yes, that's not what i'm trying to do. i'm sending rtp from webrtc (the browser) to the server and on the server trying to use gstreamer to show me the stream mostly for debugging/testing purpose. and regarding dtls, altough i dont know all the details, i read that janus (the server) handles this – John Doe Sep 27 '14 at 07:11
  • The packets are decrypted by the time they reach the gstreamer pipeline. Janus handles the decrypting and demuxing of each packet. – Benjamin Trent Sep 27 '14 at 16:23
0

Kurento Media Server (KMS) is a WebRTC media server fully writteng on top of GStreamer. KMS provides a WebRtcEndpoint implementing all the required protocol and algorithms for sending/receiving WebRTC streams to/from web browsers. KMS exposes and API based on media elements and media pipelines, which translate into GStreamer media pipelines. In general, all capabilities you have on GStreamer can be also used in KMS. You can take a look to KMS in http://www.kurento.org.

Disclaimer: I'm part of the Kurento development team.

lulop
  • 917
  • 8
  • 7
  • What pipeline specifically would work with a decrypted, demuxed RTP stream from Chrome's WebRTC stream? The user is already utilizing janus so just linking another technology does not really answer the question. – Benjamin Trent Sep 27 '14 at 16:24
0

I have updated my fork that contains the bidirectional streaming plugin to show you an example that works(I have tested on debian jessie).

Here are my pointers for your plugin changes

  1. make sure your gstreamer pipeline is set to receive BEFORE you request the keyframe from chrome
  2. Request your key frame when webrtc media is ready(see janus_bidirectional_streaming_setup_media function for details)
  3. Do not use the rtpbin gstreamer element for handling the incoming stream. For some reason the way it is setting caps does not really work and the pipeline will crash. If you do get the rtp packets and are able to send them to a port, then the following pipeline worked without issues: gst-launch-1.0 udpsrc port=<your listener> caps="application/x-rtp, clock-rate=90000, payload=100" ! rtpvp8depay ! vp8dec ! autovideosink sync=false async=false

Theoretically, directly pushing the buffers to an appsrc within the plugin should work as well.

Benjamin Trent
  • 7,378
  • 3
  • 31
  • 41