Just a few slides to talk about the first efforts on JamRTC, a native application based on GStreamer to do live jam sessions using WebRTC and Janus as an SFU. Mostly an overview of the initial architecture, with questions at the end to figure out if the approach is right or not, how to minimize latency, etc.

1. JamRTC
Jam sessions with Janus!
Lorenzo Miniero
@elminiero
Wonder WebRTC unConference
May 18th 2021

2. Who am I?
Lorenzo Miniero
• Ph.D @ UniNA
• Chairman @ Meetecho
• Main author of Janus
Contacts and info
• lorenzo@meetecho.com
• https://twitter.com/elminiero
• https://www.slideshare.net/LorenzoMiniero
• https://soundcloud.com/lminiero

3. “Hi, I’m Troy McClure. You may remember me from...”
Janus
General purpose, open source WebRTC server
• https://github.com/meetecho/janus-gateway
• Demos and documentation: https://janus.conf.meetecho.com
• Community: https://groups.google.com/forum/#!forum/meetecho-janus

4. A middle age crisis!
https://soundcloud.com/lminiero

5. An amazing music ecosystem on Linux

6. “Can WebRTC help musicians?”
https://fosdem.org/2021/schedule/event/webrtc_musicians/

7. Native approach for Jam sessions?
• A few, non-WebRTC, native solutions exist already
• e.g., Jamulus and NINJAM (both open source)
• Thinking WebRTC, only apparently a traditional use case
• Yes, we can see it as a “conference” of sorts...
• ... but we’re not really talking, and latency is much more important
• Browsers are not a good option, here
• Pipeline may be good for voice, but latency too high for live music
• Unfortunately, on Linux they don’t support Jack, only Pulseaudio
• Hard to capture anything else than a microphone
• Besides, as we said they’ll mess with the source audio anyway

8. Native approach for Jam sessions?
• A few, non-WebRTC, native solutions exist already
• e.g., Jamulus and NINJAM (both open source)
• Thinking WebRTC, only apparently a traditional use case
• Yes, we can see it as a “conference” of sorts...
• ... but we’re not really talking, and latency is much more important
• Browsers are not a good option, here
• Pipeline may be good for voice, but latency too high for live music
• Unfortunately, on Linux they don’t support Jack, only Pulseaudio
• Hard to capture anything else than a microphone
• Besides, as we said they’ll mess with the source audio anyway

9. Native approach for Jam sessions?
• A few, non-WebRTC, native solutions exist already
• e.g., Jamulus and NINJAM (both open source)
• Thinking WebRTC, only apparently a traditional use case
• Yes, we can see it as a “conference” of sorts...
• ... but we’re not really talking, and latency is much more important
• Browsers are not a good option, here
• Pipeline may be good for voice, but latency too high for live music
• Unfortunately, on Linux they don’t support Jack, only Pulseaudio
• Hard to capture anything else than a microphone
• Besides, as we said they’ll mess with the source audio anyway

10. JamRTC – Jam sessions with Janus!
https://github.com/lminiero/jamrtc

11. JamRTC – Jam sessions with Janus!
• Simple (and very ugly) GTK+ 3 application
• Static UI done with Glade
• Janus used as an SFU
• Signalling implemented via libwebsockets
• VideoRoom display property “abused” for correlation
• GStreamer 1.x pipelines for WebRTC PeerConnections
• Audio/video PeerConnections for “chat”
• Audio-only PeerConnections for instruments
• Focus on Jack to minimize latency, so Linux-only at the moment
• GStreamer pipelines easy to manipulate, though
• Any equivalent technology on Windows/MacOS?

12. JamRTC – Jam sessions with Janus!
• Simple (and very ugly) GTK+ 3 application
• Static UI done with Glade
• Janus used as an SFU
• Signalling implemented via libwebsockets
• VideoRoom display property “abused” for correlation
• GStreamer 1.x pipelines for WebRTC PeerConnections
• Audio/video PeerConnections for “chat”
• Audio-only PeerConnections for instruments
• Focus on Jack to minimize latency, so Linux-only at the moment
• GStreamer pipelines easy to manipulate, though
• Any equivalent technology on Windows/MacOS?

13. JamRTC – Jam sessions with Janus!
• Simple (and very ugly) GTK+ 3 application
• Static UI done with Glade
• Janus used as an SFU
• Signalling implemented via libwebsockets
• VideoRoom display property “abused” for correlation
• GStreamer 1.x pipelines for WebRTC PeerConnections
• Audio/video PeerConnections for “chat”
• Audio-only PeerConnections for instruments
• Focus on Jack to minimize latency, so Linux-only at the moment
• GStreamer pipelines easy to manipulate, though
• Any equivalent technology on Windows/MacOS?

14. JamRTC – Jam sessions with Janus!
• Simple (and very ugly) GTK+ 3 application
• Static UI done with Glade
• Janus used as an SFU
• Signalling implemented via libwebsockets
• VideoRoom display property “abused” for correlation
• GStreamer 1.x pipelines for WebRTC PeerConnections
• Audio/video PeerConnections for “chat”
• Audio-only PeerConnections for instruments
• Focus on Jack to minimize latency, so Linux-only at the moment
• GStreamer pipelines easy to manipulate, though
• Any equivalent technology on Windows/MacOS?

15. The ugliest UI you’ll see today...

16. The ugliest UI you’ll see today...

17. Integrating with local Jack setup

18. Integrating with local Jack setup

19. Integrating with local Jack setup

20. Integrating with local Jack setup

21. What’s next?

22. Still far from done...
• Latency still too high
• Only tested by myself with two laptops connected via WiFi, though
• Would love some feedback on actual sessions!
• Not clear how to evaluate actual latency
• We need to minimize the “local” work (little we can do about the network)
• Can jack_delay or jack_iodelay help?
• A few hacks to try and mitigate that
• Latency in rtpbin forced to 0 (bad idea?)
• Anything else?
• Would it be pointless anyway, without server-side synchronization?

23. Still far from done...
• Latency still too high
• Only tested by myself with two laptops connected via WiFi, though
• Would love some feedback on actual sessions!
• Not clear how to evaluate actual latency
• We need to minimize the “local” work (little we can do about the network)
• Can jack_delay or jack_iodelay help?
• A few hacks to try and mitigate that
• Latency in rtpbin forced to 0 (bad idea?)
• Anything else?
• Would it be pointless anyway, without server-side synchronization?

24. Still far from done...
• Latency still too high
• Only tested by myself with two laptops connected via WiFi, though
• Would love some feedback on actual sessions!
• Not clear how to evaluate actual latency
• We need to minimize the “local” work (little we can do about the network)
• Can jack_delay or jack_iodelay help?
• A few hacks to try and mitigate that
• Latency in rtpbin forced to 0 (bad idea?)
• Anything else?
• Would it be pointless anyway, without server-side synchronization?

25. Thanks! Questions? Comments?
Get in touch!
• https://twitter.com/elminiero
• https://twitter.com/meetecho
• https://www.meetecho.com