3

I am sending a mediastream of canvas over webrtc peer connection. Now, when I receive mediastream, I add mediastream to the video element which shows background black. I want to make the background transparent so that a multi-layer canvas can be streamed. I found similar questions but most of them seems to old.
After reading different blogs and comments I learned below approaches :-

  • When I receive mediastream I will stream video over canvas and then change alpha of canvas image to effect background. But this will be slow.
  • I read somewhere that VP9 or H286 codec used by webrtc mediastream don't have any alpha field so we can't make video transparent. So it is not possible to make video transparent.

I want to know is there any better way to resolve these issues.

For example, I created a server where canvas whiteboard is running and on client side client can able to stream that canvas as video.

Canvas whiteboard

canvas whiteboard

Video stream of above canvas

video stream of canvas

So in above image one is original canvas whiteboard and second one is video stream of a canvas. Now the video has black background. I want to make background transparent.

amar_1995
  • 575
  • 3
  • 14
  • Is having a delay an issue for you? VP8 and VP9 can actually hold an alpha channel, Chrome does preserve the transparency in their records using these codecs through the MediaRecorder interface. So a workaround if delay is not an issue could be to use the MediaRecorder and transfer chunks of the recorded media, that would get consumed in a MediaSource on the other end. But an other limitation of this workaround is that only Chrome does support transparency there... And I don't know why they don't support it through RTC, whatever the codec used... – Kaiido Dec 18 '20 at 05:18
  • I have updated my question with example. @Kaiido I want to manipulate the stream which I recieved from rtcpeerconnection. – amar_1995 Dec 18 '20 at 09:51
  • According to these https://stackoverflow.com/a/45470649/7974404 vp9 or h286 don't have alpha. But these answer is 3 years old. So I want to know if there is any to manipulate alpha. – amar_1995 Dec 18 '20 at 09:56
  • @amar_1995 please answer my previous question, is a delay an issue? webm can be abused to incorporate an alpha channel see https://developers.google.com/web/updates/2013/07/Alpha-transparency-in-Chrome-video#3_encode_alpha_video_to_webm And they do this for media recorded through MediaRecorder, so it can even be done from the browser. Now, from your screenshot I wonder if you actually need RTC at all, if you are really only doing a collaborative white board without things like screen-sharing or complex image filters, then the best is probably to send the drawing commands through sockets. – Kaiido Dec 18 '20 at 10:46
  • I need stream to be fast. If it lags than ux will be suffered. – amar_1995 Dec 18 '20 at 11:16
  • I don't exactly want to send whiteboard stream. These is just a sample to show results. I want to stream anything which is made with canvas(example games). But how will stream video in multi-layer canvas case work. I thought if videos will be transparent then video can be laid on top of each other to make it look simillar to multi-layer canvas. – amar_1995 Dec 18 '20 at 11:21

0 Answers0