9

I would like to build a web application that processes video from users' webcams. It looks like WebRTC is ideal for this project. But, I'm having a hard time creating a peer connection between the user's machine and a pod in my Kubernetes cluster. How would you connect these two peers?

This question on Server Fault discusses the issue I'm running into: WEBRTC MCU/SFU inside kubernetes - Port Ranges. WebRTC wants a bunch of ports open so users can create peer connections with the server but Kubernetes has ports closed by default. Here's a rephrasing of my question: How to create RTCPeerConnections connecting multiple users to an application hosted in a Kubernetes cluster? How should network ports be setup?

The closest I've come to finding a solution is Orchestrating GPU-accelerated streaming apps using WebRTC, their code is available on GitHub. I don't fully understand their approach, I believe it depends on Istio.

Jonas
  • 121,568
  • 97
  • 310
  • 388
Andrew
  • 743
  • 6
  • 17
  • "How would you connect these two peers?" well, webrtc uses STUN/TURN to connect through NAT... it is a bit more complicated protocol, yes. Can you clarify what part you ask about? – Jonas Oct 06 '20 at 20:12
  • you can have the same architecture without Istio as well – Jonas Oct 06 '20 at 20:15
  • @Jonas thanks for the help! I have been using https://www.xirsys.com/ for STUN/TURN servers. I have been using https://github.com/aiortc/aiortc for my server-side code. When I run my server-side code on my personal computer users can connect. When I run the same code in my Kubernetes cluster users cannot connect. I think I need to open up the network so peer connections can be made with pods in the cluster, but I haven't figured out the best way to do that. – Andrew Oct 06 '20 at 21:10

2 Answers2

8

The document you link to is helpful, Orchestrating GPU-accelerated streaming apps using WebRTC

What they do to allow for RTCPeerConnection is:

Use two separate Node pools (group of Nodes):

  • Default Node pool - for most components, using Ingress and load balancer
  • TURN Node pool - for STUN/TURN service

STUN/TURN service

The STUN/TURN service is network bound and deployed to dedicated nodes. It is deployed with one instance on each node in the node pool. This can be done on Kubernetes using a DaemonSet. In addition this service should use host networking, e.g. all nodes has its ports accessible from Internet. Activate host networking for the PodTemplate in your DaemonSet:

hostNetwork: true

They use coturn as STUN/TURN server.

The STUN/TURN service is run as a DaemonSet on each node of the TURN node pool. The coTURN process needs to allocate a fixed block of ports bound to the host IP address in order to properly serve relay traffic. A single coTURN instance can serve thousands of concurrent STUN and TURN requests based on the machine configuration.

Network

This part of their network diagram shows that some services are served over https with an ingress gateway, whereas the STUN/TURN service is through a different connection using dtls/rtp to the nodes exposed via host network.

Network

Jonas
  • 121,568
  • 97
  • 310
  • 388
  • Thank you Jonas!! This is going to take me some time to wrap my head around, but I think that hostNetwork setting is key. I really appreciate the help. I'll accept and upvote once I get things working. – Andrew Oct 08 '20 at 21:03
1

For anyone still looking for a solution to this problem: STUNner is a new WebRTC media gateway that is designed precisely to support the use case the OP seeks, that is, ingesting WebRTC media traffic into a Kubernetes cluster. STUNner can be configured in the usual YAML-engineering style, it allows to run (and autoscale) your WebRTC media servers in ordinary pods, and it needs only one public IP and port for all media.

Disclaimer: I'm one of the authors of STUNner.

Gabor Retvari
  • 116
  • 1
  • 6