feat!: make processing interface async #1920
Conversation
|
Hey @holzgeist , I have a question that's not necessarily related to this PR but instead to #1700 which implemented the Sink APIs, I wanted to know how exactly to access these APIs from the dart side as I couldn't find any documentation for the same. I figure you might have used them, so I was wondering if you could help me with this. I want to access the raw Audio Data coming from the server using this method to then pass it into an on device transcription service. |
|
Hey, thanks a lot for the reply. If it's not too much trouble could you please send me some documentation on how i would go about implementing this platform wise? |
|
We are currently in the process of open sourcing our video blurring plugin here: https://github.com/holzgeist/flutter-livekit-background-effects |
this PR changes the processor interface to be async. Instead of
VideoFrame onFrame(VideoFrame frame), which requires to process the frame immediately on the current thread, basing the interface on theVideoSinkinterface allows dispatching frame processing to thread pools, GPU etc.The new interface requires two functions:
void setSink(VideoSink videoSink)which provides a processor with the next processor or a final sink/renderer which is should forward processed frames to. If it is null, neither processing nor forwarding is neededvoid onFrame(VideoFrame frame)(fromVideoSink), which tells the processor about a new frame. Depending on the current workload (FPS, previous frame still processing, etc), the processor can chose to ignore the frame or process it.The change is done in a similar way on iOS
There might be a way to make this backwards compatible with older, synchronous implementations of the interface. If there is a chance that this is merged, I can look into it