WebRTC for Unity3D

Video & Audio Conference on Unity3D using WebRTC and LiveSwitch

by       AR, Unity3D, LiveSwitch, IceLink     

Florian Auer

Video & Audio Conference for Unity3D using WebRTC by LiveSwitch / IceLink for every platform FrozenMountain supports. Including multi-party conferencing (SFU & MCU).

[Update]


Currently, we are reimplementing all of this with additional features, forming an SDK ready to use with existing LiveSwitch or IceLink implementations. If you want to join our upcoming private beta, send an email to ls4unity-beta@auerflorian.com including your usecase.

About the project


To support video & audio conferencing on the three target platforms, LiveSwitch from FrozenMountain was chosen. It's built on top of IceLink (WebRTC for all Browsers and Native Platforms by FrozenMountain) and WebSync (Signaling, Chat & Data Synchronization by FrozenMountain) with the ability for multi-party conferencing (SFU & MCU).

FrozenMountain enables us to use use state of the art codecs like VP8 and VP9. The project was designed to run on Unity (Android and iOS), as well as WPF. There are other platforms, which could easily be targeted. E.g. macOS, tvOS, Linux, UWP (Hololens!), iOS and Android both native and Xamarin, all browsers.

Why WebRTC


WebRTC (Web Real-Time Communication) is a free, open-source project that provides web browsers and mobile applications with real-time communication (RTC) […]. It allows audio and video communication to work inside web pages [and apps] by allowing direct peer-to-peer communication […].

More details

No out-of-the-box solution from FrozenMountain


FrozenMountain supports many platforms for IceLink, WebSync and LiveSwitch. There are also libraries for Unity3D. But at the moment FrozenMountain does not provide us with sources and sinks for audio nor video. You also need to get the Unity3D specific usage of the hardware powered codecs to work. So you have to do all on your own, without help from FrozenMountain (if you don't pay for professional services). It was a long and stony path we had to go. But in the end, it's working. We managed to implement audio source and sink platform agnostic, so we can use it for every platform the magic of C# and .NET runs on. The video sinks and sources use hardware encoder and need a codec for each platform (platform specific workload). This way of implementing WebRTC on Unity3D can be used for LiveSwitch and also for IceLink.

Some very time consuming bugs


  • Could not initialize VPX encoder. ABI version mismatch -> wrong VP8/VP9 encoder version
  • Could not initialize VPX encoder. Invalid parameter -> wrong initilization params for encoder

Source Code


We would like to give back to the community. This implementation was done during a bigger project, so the source code can't be shared. But you can reach out to us, if you need a hand.