I'd like to build a file transfer application with end-to-end encryption similar to magic-wormhole or croc using RTCDataChannels (WebRTC).
Is it possible to perform mutual authentication by validating peer certificates with Web APIs? The idea is to prevent man-in-the-middle attacks by comparing the certificate (or certificate fingerprint) with a value negotiated out-of-band.
There is apparently a getRemoteCertificates and a RTCCertificate interface which should enable access of the remote certificate fingerprint. Also, While looking for solutions, I found a comment related to an old draft of WebRTC. However, I am not sure how an Identity Provider is related to that.
WebRTC implementations must check certificates to be secure. This is covered in section 8.3.5 of the W3C draft: Waiting for all DTLS connections to be establishes and checking that the certificate fingerprints on all connections matches the one provided by the IdP.
Finally, is this a good approach or is it preferred to add another application security layer for authentication?