Failed to set remote answer sdp: Called in wrong state: stable

Question

I am trying to write a WebRTC application using socket.io.

The signalling server is written in python and looks like this.

import socketio
import uvicorn
from starlette.applications import Starlette

ROOM = 'room'


sio = socketio.AsyncServer(async_mode='asgi', cors_allowed_origins='*')
star_app = Starlette(debug=True)
app = socketio.ASGIApp(sio, star_app)


@sio.event
async def connect(sid, environ):
    await sio.emit('ready', room=ROOM, skip_sid=sid)
    sio.enter_room(sid, ROOM)


@sio.event
async def data(sid, data):
    await sio.emit('data', data, room=ROOM, skip_sid=sid)


@sio.event
async def disconnect(sid):
    sio.leave_room(sid, ROOM)


if __name__ == '__main__':
    uvicorn.run(app, host='0.0.0.0', port=8003)

The client side looks like this

<script>
    const SIGNALING_SERVER_URL = 'http://127.0.0.1:8003?session_id=1';
    // WebRTC config: you don't have to change this for the example to work
    // If you are testing on localhost, you can just use PC_CONFIG = {}
    const PC_CONFIG = {};

    // Signaling methods
    let socket = io(SIGNALING_SERVER_URL, {autoConnect: false});

    socket.on('data', (data) => {
        console.log('Data received: ', data);
        handleSignalingData(data);
    });

    socket.on('ready', () => {
        console.log('Ready');
        // Connection with signaling server is ready, and so is local stream
        createPeerConnection();
        sendOffer();
    });

    let sendData = (data) => {
        socket.emit('data', data);
    };

    // WebRTC methods
    let pc;
    let localStream;
    let remoteStreamElement = document.querySelector('#remoteStream');

    let getLocalStream = () => {
        navigator.mediaDevices.getUserMedia({audio: true, video: true})
            .then((stream) => {
                console.log('Stream found');
                localStream = stream;
                // Connect after making sure that local stream is availble
                socket.connect();
            })
            .catch(error => {
                console.error('Stream not found: ', error);
            });
    }

    let createPeerConnection = () => {
        try {
            pc = new RTCPeerConnection(PC_CONFIG);
            pc.onicecandidate = onIceCandidate;
            pc.onaddstream = onAddStream;
            pc.addStream(localStream);
            console.log('PeerConnection created');
        } catch (error) {
            console.error('PeerConnection failed: ', error);
        }
    };

    let sendOffer = () => {
        console.log('Send offer');
        pc.createOffer().then(
            setAndSendLocalDescription,
            (error) => {
                console.error('Send offer failed: ', error);
            }
        );
    };

    let sendAnswer = () => {
        console.log('Send answer');
        pc.createAnswer().then(
            setAndSendLocalDescription,
            (error) => {
                console.error('Send answer failed: ', error);
            }
        );
    };

    let setAndSendLocalDescription = (sessionDescription) => {
        pc.setLocalDescription(sessionDescription);
        console.log('Local description set');
        sendData(sessionDescription);
    };

    let onIceCandidate = (event) => {
        if (event.candidate) {
            console.log('ICE candidate');
            sendData({
                type: 'candidate',
                candidate: event.candidate
            });
        }
    };

    let onAddStream = (event) => {
        console.log('Add stream');
        remoteStreamElement.srcObject = event.stream;
    };

    let handleSignalingData = (data) => {
        // let msg = JSON.parse(data);
        switch (data.type) {
            case 'offer':
                createPeerConnection();
                pc.setRemoteDescription(new RTCSessionDescription(data));
                sendAnswer();
                break;
            case 'answer':
                pc.setRemoteDescription(new RTCSessionDescription(data));
                break;
            case 'candidate':
                pc.addIceCandidate(new RTCIceCandidate(data.candidate));
                break;
        }
    };

    // Start connection
    getLocalStream();
</script>

Also i use this code for client as socket.io

https://github.com/socketio/socket.io/blob/master/client-dist/socket.io.js

When two people are in the connection, everything works great. But as soon as a third user tries to connect to them, the streaming stops with an error

Uncaught (in promise) DOMException: Failed to execute 'setRemoteDescription' on 'RTCPeerConnection': Failed to set remote answer sdp: Called in wrong state: stable

I don't have much knowledge of javascript, so I need your help. Thanks.

P.S. I see this error in all browsers.

See this repository

https://github.com/pfertyk/webrtc-working-example

See this instructions

https://pfertyk.me/2020/03/webrtc-a-working-example/

Answer 1

The reason why you are getting this error message is because when a third user joins, it sends an offer to the 2 previously connected users, and therefore, it receives 2 answers. Since one RTCPeerConnection can only establish one peer-to-peer connection, it will complain when it tries to setRemoteDescription on the answer that arrived later, because it already has a stable connection with the peer whose SDP answer arrived first. To handle multiple users, you will need to instantiate a new RTCPeerConnection for every remote peer.

That said, you can manage multiple RTCPeerConnections using some sort of dictionary or list structure. Through your signalling server, whenever a user connects you can emit a unique user id (could be the socket id). When receiving this id, you just instantiate a new RTCPeerConnection and map the received id to the newly created peer connection and then when you will have to setRemoteDescription on all entries of your data structure.

This would also remove the memory leaks in your code every time a new user joins when you overwrite the peer connection variable 'pc' when it is still in use.

Notice though, that this solution is not scalable at all, since you are going to be creating new peer connections exponentially, with ~6 the quality of your call will already be terrible. If your intention is to have a conference room, you should really look into using an SFU, but be aware that usually, it is quite cumbersome to set it up.

Checkout Janus videoroom plugin for an open-source SFU implementation.

Answer 2

As you know, you should create separate peer connections for each peer, so in your code, the wrong section is the global variable pc, that every time you are set it in createPeerConnection function.

Instead, you should have, for example, an array of pcs that every time you got an offer, you create a new pc in the createPeerConnection function, set local and remote description for that pc and send generated answer to your signaling server.

Answer 3

I've answered this question above in details as to why you are having this issue. But seems like what you are really looking for is some sample working code on how to fix it... so here you go:

index.html: Slightly update the HTML page, so now we have a div that we will append incoming remote videos.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>WebRTC working example</title>
</head>
<body>
    <div id="remoteStreams"></div>
    <script src="socket.io.js"></script>
    <script src="main.js"></script>
</body>
</html>

app.py: updated data and ready event handlers a bit so that we emit the socket id to other peers correctly.

import socketio
import uvicorn
from starlette.applications import Starlette

ROOM = 'room'

sio = socketio.AsyncServer(async_mode='asgi', cors_allowed_origins='*')
star_app = Starlette(debug=True)
app = socketio.ASGIApp(sio, star_app)


@sio.event
async def connect(sid, environ):
    await sio.emit('ready', {'sid': sid}, room=ROOM, skip_sid=sid)
    sio.enter_room(sid, ROOM)


@sio.event
async def data(sid, data):
    peerToSend = None
    if 'sid' in data:
      peerToSend = data['sid']
    data['sid'] = sid
    await sio.emit('data', data, room=peerToSend if peerToSend else ROOM, skip_sid=sid)


@sio.event
async def disconnect(sid):
    sio.leave_room(sid, ROOM)


if __name__ == '__main__':
    uvicorn.run(app, host='localhost', port=8003)

main.js: Created this peers object to map socket ids to RTCPeerConnections and updated some of the functions to use that instead of the pc variable.

const SIGNALING_SERVER_URL = 'ws://127.0.0.1:8003';
// WebRTC config: you don't have to change this for the example to work
// If you are testing on localhost, you can just use PC_CONFIG = {}
const PC_CONFIG = {};

// Signaling methods
let socket = io(SIGNALING_SERVER_URL, {autoConnect: false});

socket.on('data', (data) => {
    console.log('Data received: ', data);
    handleSignalingData(data);
});

socket.on('ready', (msg) => {
    console.log('Ready');
    // Connection with signaling server is ready, and so is local stream
    peers[msg.sid] = createPeerConnection();
    sendOffer(msg.sid);
    addPendingCandidates(msg.sid);
});

let sendData = (data) => {
    socket.emit('data', data);
};

// WebRTC methods
let peers = {}
let pendingCandidates = {}
let localStream;

let getLocalStream = () => {
    navigator.mediaDevices.getUserMedia({audio: true, video: true})
        .then((stream) => {
            console.log('Stream found');
            localStream = stream;
            // Connect after making sure thzat local stream is availble
            socket.connect();
        })
        .catch(error => {
            console.error('Stream not found: ', error);
        });
}

let createPeerConnection = () => {
    const pc = new RTCPeerConnection(PC_CONFIG);
    pc.onicecandidate = onIceCandidate;
    pc.onaddstream = onAddStream;
    pc.addStream(localStream);
    console.log('PeerConnection created');
    return pc;
};

let sendOffer = (sid) => {
    console.log('Send offer');
    peers[sid].createOffer().then(
        (sdp) => setAndSendLocalDescription(sid, sdp),
        (error) => {
            console.error('Send offer failed: ', error);
        }
    );
};

let sendAnswer = (sid) => {
    console.log('Send answer');
    peers[sid].createAnswer().then(
        (sdp) => setAndSendLocalDescription(sid, sdp),
        (error) => {
            console.error('Send answer failed: ', error);
        }
    );
};

let setAndSendLocalDescription = (sid, sessionDescription) => {
    peers[sid].setLocalDescription(sessionDescription);
    console.log('Local description set');
    sendData({sid, type: sessionDescription.type, sdp: sessionDescription.sdp});
};

let onIceCandidate = (event) => {
    if (event.candidate) {
        console.log('ICE candidate');
        sendData({
            type: 'candidate',
            candidate: event.candidate
        });
    }
};

let onAddStream = (event) => {
    console.log('Add stream');
    const newRemoteStreamElem = document.createElement('video');
    newRemoteStreamElem.autoplay = true;
    newRemoteStreamElem.srcObject = event.stream;
    document.querySelector('#remoteStreams').appendChild(newRemoteStreamElem);
};

let addPendingCandidates = (sid) => {
    if (sid in pendingCandidates) {
        pendingCandidates[sid].forEach(candidate => {
            peers[sid].addIceCandidate(new RTCIceCandidate(candidate))
        });
    }
}

let handleSignalingData = (data) => {
    // let msg = JSON.parse(data);
    console.log(data)
    const sid = data.sid;
    delete data.sid;
    switch (data.type) {
        case 'offer':
            peers[sid] = createPeerConnection();
            peers[sid].setRemoteDescription(new RTCSessionDescription(data));
            sendAnswer(sid);
            addPendingCandidates(sid);
            break;
        case 'answer':
            peers[sid].setRemoteDescription(new RTCSessionDescription(data));
            break;
        case 'candidate':
            if (sid in peers) {
                peers[sid].addIceCandidate(new RTCIceCandidate(data.candidate));
            } else {
                if (!(sid in pendingCandidates)) {
                    pendingCandidates[sid] = [];
                }
                pendingCandidates[sid].push(data.candidate)
            }
            break;
    }
};

// Start connection
getLocalStream();

I tried to change your code as little as possible, so you should be able to just copy-paste and have it working.

Here's my working code: https://github.com/lnogueir/webrtc-socketio

If you have any issues running it, let me know or open an issue there and I'll do my best to help.

Answer 4

In a nutshell you need to make sure that you have one peerconnection per peer and that your signalling protocol allows differentiating who has sent you an offer or answer.

For a two connection case refer to the canonical sample https://webrtc.github.io/samples/src/content/peerconnection/multiple/

For generalizing this into multiple peers with socket.io the (now deprecated and unmaintained) simplewebrtc package might be useful: https://github.com/simplewebrtc/SimpleWebRTC

The simple-peer library provides similar functionality but you will have to integrate socketio yourself.