Slides for the presentation on Insertable Streams and E2EE in WebRTC I presented at the virtual edition of ClueCon 2020. After an introduction on the past and recent E2EE efforts, the slides also present some efforts to integrate the technology in the Janus WebRTC server as well.

1. Insertable Streams and E2EE in WebRTC: a Janus story
Lorenzo Miniero
ClueCon – Chicago, IL, USA (kinda!)
August 5th 2020

2. Who am I?
Lorenzo Miniero
• Ph.D @ UniNA
• Chairman @ Meetecho
• Main author of Janus®
Contacts and info
• lorenzo@meetecho.com
• https://twitter.com/elminiero
• https://www.slideshare.net/LorenzoMiniero
• https://soundcloud.com/lminiero

3. Just a few words on Meetecho
• Co-founded in 2009 as an academic spin-off
• University research efforts brought to the market
• Completely independent from the University
• Focus on real-time multimedia applications
• Strong perspective on standardization and open source
• Several activities
• Consulting services
• Commercial support and Janus licenses
• Streaming of live events (IETF, ACM, etc.)
• Proudly brewed in sunny Napoli, Italy

4. Home Sweet Home!

5. WebRTC reference architecture: peer-to-peer

6. WebRTC reference architecture: peer-to-peer

7. Involving a server as a peer

8. Involving a server as a peer

9. Breaking end-to-end encryption

10. Many reasons why that may be needed
• Server being a “MITM” is often a feature, actually
• It may need to transcode media
• An MCU even more so! (e.g., FreeSwitch)
• Recordings typically need access to unencrypted media too
• Even when just relaying, access to payload may be sometimes required
• e.g., Simulcast or SVC
• Codec-specific info (e.g., temporal layer) may be in the payload
• RTP headers would still need to be editable
Can you get true E2EE even through a server?
• Short answer: YES!
• Slightly longer answer: it’s not that easy, though...

11. Many reasons why that may be needed
• Server being a “MITM” is often a feature, actually
• It may need to transcode media
• An MCU even more so! (e.g., FreeSwitch)
• Recordings typically need access to unencrypted media too
• Even when just relaying, access to payload may be sometimes required
• e.g., Simulcast or SVC
• Codec-specific info (e.g., temporal layer) may be in the payload
• RTP headers would still need to be editable
Can you get true E2EE even through a server?
• Short answer: YES!
• Slightly longer answer: it’s not that easy, though...

12. Many reasons why that may be needed
• Server being a “MITM” is often a feature, actually
• It may need to transcode media
• An MCU even more so! (e.g., FreeSwitch)
• Recordings typically need access to unencrypted media too
• Even when just relaying, access to payload may be sometimes required
• e.g., Simulcast or SVC
• Codec-specific info (e.g., temporal layer) may be in the payload
• RTP headers would still need to be editable
Can you get true E2EE even through a server?
• Short answer: YES!
• Slightly longer answer: it’s not that easy, though...

13. Many reasons why that may be needed
• Server being a “MITM” is often a feature, actually
• It may need to transcode media
• An MCU even more so! (e.g., FreeSwitch)
• Recordings typically need access to unencrypted media too
• Even when just relaying, access to payload may be sometimes required
• e.g., Simulcast or SVC
• Codec-specific info (e.g., temporal layer) may be in the payload
• RTP headers would still need to be editable
Can you get true E2EE even through a server?
• Short answer: YES!
• Slightly longer answer: it’s not that easy, though...

14. An excellent talk on the challenges of E2EE!
https://www.youtube.com/watch?v=a0vhHmONWlw

15. Privacy Enhanced RTP Conferencing (PERC)
https://datatracker.ietf.org/wg/perc/about/

16. Not very popular among WebRTC developers, though...
https://mailarchive.ietf.org/arch/msg/perc/x1HjohPL6ISuoNj7eE6HEre7O1E/

17. A simpler approach: PERC “Lite”

18. PERC Lite and Janus
https://www.meetecho.com/blog/meetecho-and-cosmo-strike-again-perc-lite-integration-in-janus/

19. PERC Lite and Janus
https://www.meetecho.com/blog/meetecho-and-cosmo-strike-again-perc-lite-integration-in-janus/

20. Replaying encrypted recordings (“DRM”)

21. Replaying encrypted recordings (“DRM”)

22. Enter Insertable Streams!
• New browser API that allows custom processing on WebRTC frames
• Key use case −→ E2EE!
• Works with encoded frames, not individual packets
• [OUT] audio/video frames after they’ve been encoded, but before RTP packetization
• [IN] audio/video frames before they’ve been decoded, but after RTP de-packetization
• Working with frames makes it protocol agnostic
• Today it’s RTP, tomorrow it might be QUIC
Some useful reads
• https://www.meetecho.com/blog/janus-e2ee-sframe/
• https://webrtcbydralex.com/index.php/2020/03/30/
• https://webrtchacks.com/true-end-to-end-encryption-with-webrtc-insertable-streams/

23. Enter Insertable Streams!
• New browser API that allows custom processing on WebRTC frames
• Key use case −→ E2EE!
• Works with encoded frames, not individual packets
• [OUT] audio/video frames after they’ve been encoded, but before RTP packetization
• [IN] audio/video frames before they’ve been decoded, but after RTP de-packetization
• Working with frames makes it protocol agnostic
• Today it’s RTP, tomorrow it might be QUIC
Some useful reads
• https://www.meetecho.com/blog/janus-e2ee-sframe/
• https://webrtcbydralex.com/index.php/2020/03/30/
• https://webrtchacks.com/true-end-to-end-encryption-with-webrtc-insertable-streams/

24. Enter Insertable Streams!
• New browser API that allows custom processing on WebRTC frames
• Key use case −→ E2EE!
• Works with encoded frames, not individual packets
• [OUT] audio/video frames after they’ve been encoded, but before RTP packetization
• [IN] audio/video frames before they’ve been decoded, but after RTP de-packetization
• Working with frames makes it protocol agnostic
• Today it’s RTP, tomorrow it might be QUIC
Some useful reads
• https://www.meetecho.com/blog/janus-e2ee-sframe/
• https://webrtcbydralex.com/index.php/2020/03/30/
• https://webrtchacks.com/true-end-to-end-encryption-with-webrtc-insertable-streams/

25. Enter Insertable Streams!
• New browser API that allows custom processing on WebRTC frames
• Key use case −→ E2EE!
• Works with encoded frames, not individual packets
• [OUT] audio/video frames after they’ve been encoded, but before RTP packetization
• [IN] audio/video frames before they’ve been decoded, but after RTP de-packetization
• Working with frames makes it protocol agnostic
• Today it’s RTP, tomorrow it might be QUIC
Some useful reads
• https://www.meetecho.com/blog/janus-e2ee-sframe/
• https://webrtcbydralex.com/index.php/2020/03/30/
• https://webrtchacks.com/true-end-to-end-encryption-with-webrtc-insertable-streams/

26. Insertable Streams workflow

27. Insertable Streams workflow

28. A few challenges with this approach
• Encrypted payload may make life harder for SFUs
• Useful info usually available in the payload
• e.g., how to detect if we got a keyframe?
• A few different solutions to this
• Keep the first bytes unencrypted? (where that info is)
• Ad-hoc RTP extensions for metadata (spec on its way!)
• Encrypting the payload may confuse the RTP (de)packetizer
• Each video codec has different rules
• e.g., H.264 uses NALs, which wouldn’t be visible anymore
• Solution more tricky to find, here
• Keep info used for splitting unencrypted? (codec specific)
• Even better, use a global RTP packetization for all codecs (spec coming!)

29. A few challenges with this approach
• Encrypted payload may make life harder for SFUs
• Useful info usually available in the payload
• e.g., how to detect if we got a keyframe?
• A few different solutions to this
• Keep the first bytes unencrypted? (where that info is)
• Ad-hoc RTP extensions for metadata (spec on its way!)
• Encrypting the payload may confuse the RTP (de)packetizer
• Each video codec has different rules
• e.g., H.264 uses NALs, which wouldn’t be visible anymore
• Solution more tricky to find, here
• Keep info used for splitting unencrypted? (codec specific)
• Even better, use a global RTP packetization for all codecs (spec coming!)

30. A few challenges with this approach
• Encrypted payload may make life harder for SFUs
• Useful info usually available in the payload
• e.g., how to detect if we got a keyframe?
• A few different solutions to this
• Keep the first bytes unencrypted? (where that info is)
• Ad-hoc RTP extensions for metadata (spec on its way!)
• Encrypting the payload may confuse the RTP (de)packetizer
• Each video codec has different rules
• e.g., H.264 uses NALs, which wouldn’t be visible anymore
• Solution more tricky to find, here
• Keep info used for splitting unencrypted? (codec specific)
• Even better, use a global RTP packetization for all codecs (spec coming!)

31. A few challenges with this approach
• Encrypted payload may make life harder for SFUs
• Useful info usually available in the payload
• e.g., how to detect if we got a keyframe?
• A few different solutions to this
• Keep the first bytes unencrypted? (where that info is)
• Ad-hoc RTP extensions for metadata (spec on its way!)
• Encrypting the payload may confuse the RTP (de)packetizer
• Each video codec has different rules
• e.g., H.264 uses NALs, which wouldn’t be visible anymore
• Solution more tricky to find, here
• Keep info used for splitting unencrypted? (codec specific)
• Even better, use a global RTP packetization for all codecs (spec coming!)

32. Enabling Insertable Streams
pc = new RTCPeerConnection({
encodedInsertableStreams: true,
forceEncodedAudioInsertableStreams: true,
forceEncodedVideoInsertableStreams: true
});

33. Adding a Sender Trasform
senderTransform = new TransformStream({
start() {
// Called on startup.
},
transform(chunk, controller) {
// Frame encoded, edit payload and return it
[..]
controller.enqueue(chunk);
},
flush() {
// Called when the stream is about to be closed
}
});
sender = pc.addTrack(track, stream)
senderStreams = sender.createEncodedVideoStreams();
senderStreams.readableStream
.pipeThrough(senderTransform)
.pipeTo(senderStreams.writableStream);

34. Adding a Receiver Trasform as well
receiverTransform = new TransformStream({
start() {
// Called on startup.
},
transform(chunk, controller) {
// Received frame, edit payload and return it
[..]
controller.enqueue(chunk);
},
flush() {
// Called when the stream is about to be closed
}
});
pc.ontrack = function(event) {
[..]
receiverStreams = event.receiver.createEncodedVideoStreams();
receiverStreams.readableStream
.pipeThrough(receiverTransform)
.pipeTo(receiverStreams.writableStream);
}

35. Integrated in janus.js a few weeks ago
echotest.createOffer({
media: { audio: true, video: true },
senderTransforms: {
audio: new TransformStream({ .. }),
video: new TransformStream({ .. })
},
receiverTransforms: {
audio: new TransformStream({ .. }),
video: new TransformStream({ .. })
},
[..]
success: function(jsep) {
// Send offer to Janus
}
});
https://github.com/meetecho/janus-gateway/pull/2074

36. Testing basic E2EE in the Janus demos
https://janus.conf.meetecho.com/e2etest

37. Testing basic E2EE in the Janus demos
https://janus.conf.meetecho.com/e2etest

38. A step forward: Secure Frames (SFrame)
• Co-developed by CoSMo and Google
• Used in Google Duo for E2EE for a long time already
• https://cosmosoftware.io/secure-frames-sframes/
• https://www.gstatic.com/duo/papers/duo_e2ee.pdf/
• Recently brought to the IETF as well (presented in DISPATCH @ IETF108)
• https://tools.ietf.org/html/draft-omara-sframe-00
• https://mailarchive.ietf.org/arch/browse/sframe/
• Learned lessons from previous E2EE attempts
• Minimizes overhead and impact on servers and clients
• Works of frames, not packets, so transport agnostic
• Interaction with KMS done out of band, and KMS agnostic (e.g., Signal or MLS)

39. A step forward: Secure Frames (SFrame)
• Co-developed by CoSMo and Google
• Used in Google Duo for E2EE for a long time already
• https://cosmosoftware.io/secure-frames-sframes/
• https://www.gstatic.com/duo/papers/duo_e2ee.pdf/
• Recently brought to the IETF as well (presented in DISPATCH @ IETF108)
• https://tools.ietf.org/html/draft-omara-sframe-00
• https://mailarchive.ietf.org/arch/browse/sframe/
• Learned lessons from previous E2EE attempts
• Minimizes overhead and impact on servers and clients
• Works of frames, not packets, so transport agnostic
• Interaction with KMS done out of band, and KMS agnostic (e.g., Signal or MLS)

40. A step forward: Secure Frames (SFrame)
• Co-developed by CoSMo and Google
• Used in Google Duo for E2EE for a long time already
• https://cosmosoftware.io/secure-frames-sframes/
• https://www.gstatic.com/duo/papers/duo_e2ee.pdf/
• Recently brought to the IETF as well (presented in DISPATCH @ IETF108)
• https://tools.ietf.org/html/draft-omara-sframe-00
• https://mailarchive.ietf.org/arch/browse/sframe/
• Learned lessons from previous E2EE attempts
• Minimizes overhead and impact on servers and clients
• Works of frames, not packets, so transport agnostic
• Interaction with KMS done out of band, and KMS agnostic (e.g., Signal or MLS)

41. A step forward: Secure Frames (SFrame)

42. A step forward: Secure Frames (SFrame)

43. SFrame.js: an open source SFrame library
• Pure javascript library implementing SFrame using webcrypto
• Developed by Sergio Garcia Murillo
• https://github.com/medooze/sframe
• Described in detail in a blog post and in the CommCon presentation
• https://medium.com/@medooze/sframe-js-end-to-end-encryption-for-webrtc-
f9a83a997d6d
• https://www.youtube.com/watch?v=a0vhHmONWlw
• Basically provides everything except the Key Management System (KMS)
• Which makes sense, that should be up to you anyway!
Helped Sergio testing it while integrating it in janus.js
https://www.meetecho.com/blog/janus-e2ee-sframe/

44. SFrame.js: an open source SFrame library
• Pure javascript library implementing SFrame using webcrypto
• Developed by Sergio Garcia Murillo
• https://github.com/medooze/sframe
• Described in detail in a blog post and in the CommCon presentation
• https://medium.com/@medooze/sframe-js-end-to-end-encryption-for-webrtc-
f9a83a997d6d
• https://www.youtube.com/watch?v=a0vhHmONWlw
• Basically provides everything except the Key Management System (KMS)
• Which makes sense, that should be up to you anyway!
Helped Sergio testing it while integrating it in janus.js
https://www.meetecho.com/blog/janus-e2ee-sframe/

45. SFrame.js: an open source SFrame library
• Pure javascript library implementing SFrame using webcrypto
• Developed by Sergio Garcia Murillo
• https://github.com/medooze/sframe
• Described in detail in a blog post and in the CommCon presentation
• https://medium.com/@medooze/sframe-js-end-to-end-encryption-for-webrtc-
f9a83a997d6d
• https://www.youtube.com/watch?v=a0vhHmONWlw
• Basically provides everything except the Key Management System (KMS)
• Which makes sense, that should be up to you anyway!
Helped Sergio testing it while integrating it in janus.js
https://www.meetecho.com/blog/janus-e2ee-sframe/

46. SFrame.js: an open source SFrame library
• Pure javascript library implementing SFrame using webcrypto
• Developed by Sergio Garcia Murillo
• https://github.com/medooze/sframe
• Described in detail in a blog post and in the CommCon presentation
• https://medium.com/@medooze/sframe-js-end-to-end-encryption-for-webrtc-
f9a83a997d6d
• https://www.youtube.com/watch?v=a0vhHmONWlw
• Basically provides everything except the Key Management System (KMS)
• Which makes sense, that should be up to you anyway!
Helped Sergio testing it while integrating it in janus.js
https://www.meetecho.com/blog/janus-e2ee-sframe/

47. Integrating SFrame.js in janus.js
import {Janus} from ’./janus.js’;
import {SFrame} from ’./sframe/Client.js’;
...
echotest.createOffer({
media: { audio: true, video: true },
sframe: {
outgoingId: 0,
incomingId: 0,
sharedKey: cryptoKey, // Generated previously
keyPair: keyPair // Generated previously
},
[..]
success: function(jsep) {
// Send offer to Janus
}
});

48. Integrating SFrame.js in janus.js
if(callbacks.sframe) {
Janus.log("Using SFrame to encrypt media end-to-end:", callbacks.sframe);
config.sframe = callbacks.sframe;
config.sframeClient = await SFrame.createClient(config.sframe.outgoingId, {});
// Sender part
await config.sframeClient.setSenderEncryptionKey(config.sframe.shared);
if(config.sframe.keyPair && config.sframe.keyPair.privateKey) {
await config.sframeClient.setSenderSigningKey(config.sframe.keyPair.privateKey);
}
// Receiver part
if(config.sframe.incomingId !== undefined && config.sframe.incomingId !== null) {
await config.sframeClient.addReceiver(config.sframe.incomingId);
await config.sframeClient.setReceiverEncryptionKey(config.sframe.incomingId,
callbacks.sframe.shared);
if(config.sframe.keyPair && config.sframe.keyPair.publicKey) {
await config.sframeClient.setReceiverVerifyKey(config.sframe.incomingId,
config.sframe.keyPair.publicKey);
}
}
}

49. Integrating SFrame.js in janus.js
// Have SFrame.js configure the Sender Transform
var sender = config.pc.addTrack(track, stream);
config.sframeClient.encrypt(sender.track.id, sender);
...
// Have SFrame.js configure the Receiver Transform
config.pc.ontrack = function(event) {
[..]
config.sframeClient.decrypt(event.track.id, event.receiver);
[..]
}

50. Testing SFrame with Janus
https://www.meetecho.com/blog/janus-e2ee-sframe/

51. Testing SFrame with Janus
https://www.meetecho.com/blog/janus-e2ee-sframe/

52. Thanks! Questions? Comments?
Get in touch!
• https://twitter.com/elminiero
• https://twitter.com/meetecho
• https://www.meetecho.com