Hi there! It’s been some time now since the last release of OpenVidu. We have been working in some exciting features and preparing lots of incoming new stuff. Busy months are coming… Let’s start by introducing voice and video filters in OpenVidu. At the end of this post you will find a section talking about future releases.
OpenVidu API now offers a simple way of applying filters to video and audio streams in the server side by making use of Kurento Media Server capabilities. Kurento is partly distinct from other media servers by its ability to apply filters in media streams. We wanted to take advantage of this awesome feature by seamlessly integrating it in OpenVidu platform. So, this is the current status of filter support in OpenVidu:
- You can apply one filter at a time to a published Stream. Every user subscribed to it will receive the modified stream.
- You can remove an applied filter.
- You can call any remote method offered by an applied filter.
- You can add and remove event listeners to any event dispatched by an applied filter.
- You must configure in the participant token the allowed filters the user can apply.
Step by step
1) Generate a token with the filters the user will be able to apply
This is a simple way of securing the ability of applying filters from OpenVidu Browser, so that not every user is able to apply any filter at any time.
- API REST: include in the JSON body a parameter
kurentoOptionswith a propertyallowedFilters(a string array containing the name of the filters the user will be able to apply)
{
"session": "6fpivlanw91qjy6n",
"data": "user_data",
"role": "PUBLISHER",
"kurentoOptions": {
"allowedFilters": ["GStreamerFilter", "FaceOverlayFilter"]
}
}- openvidu-java-client: call
TokenOptions.Builder#kurentoOptions(KurentoOptions)to setallowedFiltersvalue with methodKurentoOptions.Builder#allowedFilters(String[])
TokenOptions tokenOptions = new TokenOptions.Builder()
.role(OpenViduRole.PUBLISHER)
.data("user_data")
.kurentoOptions(
new KurentoOptions.Builder().allowedFilters(
new String[]{"GStreamerFilter", "FaceOverlayFilter"})
.build())
.build();
String token = session.generateToken(tokenOptions);- openvidu-node-client: include in TokenOptions parameter a
kurentoOptionsproperty withallowedFitersarray
var tokenOptions = {
role: "PUBLISHER",
data: "user_data",
kurentoOptions: {
allowedFilters: ["GStreamerFilter", "FaceOverlayFilter"]
}
};
session.generateToken(tokenOptions).then(token => { ... });2.A) Initialize a Publisher object configured for using a filter from the beginning of the publishing …
Use PublisherProperties, specifically property filter:
var OV = new OpenVidu();
var publisher = OV.initPublisher(
targetElement,
{
filter: {
type: "GStreamerFilter",
options: "videoflip method=vertical-flip"
}
});// user already connected to "session" with the appropriate token
session.publish(publisher);
2.B) … or apply the filter dynamically after publishing the stream, whenever you want
// user already connected to the session with the appropriate token
// and successfully publishing the Publisher objectpublisher.stream.applyFilter(
"GStreamerFilter",
"videoflip method=vertical-flip")
.then(() => console.log("Video rotated!"))
.catch(error => console.error(error));
3) You can execute any method offered by the filter
// user already connected to the session with the appropriate token
// successfully publishing the Publisher object and a filter
// being applied to its stream publisher.stream.filter.execMethod(
"setElementProperty",
{
"propertyName":"method",
"propertyValue":"horizontal-flip"
})
.then(() => console.log("Video rotation direction modified!"))
.catch(error => console.error(error));
4) You can also subscribe to any filter event (if it dispatches any), and later unsubscribe from it
// user already connected to the session with the appropriate token
// successfully publishing the Publisher object and a filter
// being applied to its streampublisher.stream.filter.addEventListener(
"FooFilterEvent",
filterEvent => {
console.log('Filter event received. Data: ' + filterEvent.data);
});...publisher.stream.filter.removeEventListener("FooFilterEvent");
4) To remove the filter
// user already connected to the session with the appropriate token
// successfully publishing the Publisher object and a filter
// being applied to its streampublisher.stream.removeFilter()
.then(() => console.log("Filter removed"))
.catch(error => console.error(error));
Moderators are not only able to call all of these methods over their
Publisher.streamobject, but also over anySubscriber.streamobject. Also, they don't need any special token permission to apply filters and can bypass any token restriction set to other user tokens
Filter samples
Kurento filters
- FaceOverlayFilter (overlay an image over detected faces)
publisher.stream.applyFilter("FaceOverlayFilter")
.then(filter => {
filter.execMethod( "setOverlayedImage", {
"uri":"https://cdn.pixabay.com/photo/2013/07/12/14/14/derby-148046_960_720.png",
"offsetXPercent":"-0.2F",
"offsetYPercent":"-0.8F",
"widthPercent":"1.3F",
"heightPercent":"1.0F"
});
});- ChromaFilter (set a chroma background)
publisher.stream.applyFilter("ChromaFilter", {
"window": {
"topRightCornerX": 0,
"topRightCornerY": 0,
"width": 50,
"height": 50
},
"backgroundImage": "https://www.maxpixel.net/static/photo/1x/Cool-Blue-Liquid-Lake-Abstract-Background-Clear-316144.jpg"
});You must install Kurento ChromaFilter library in the same host as Kurento Media Server:
sudo apt-get install kms-chroma
- ZBarFilter (detect and read bar codes information)
publisher.stream.applyFilter("ZBarFilter")
.then(filter => {
filter.addEventListener("CodeFound", filterEvent => {
console.log('Bar code found!. Data: ' + filterEvent.data);
}
});GStreamer filters
These filters are set with type GStreamerFilter and an options parameter like this:
publisher.stream.applyFilter(
"GStreamerFilter",
{"command": "GSTREAMER_COMMAND"}
);A list of interesting values for GSTREAMER_COMMAND parameter is stated below. Replace GSTREAMER_COMMAND in the upper code snippet for any of the examples provided in the following list items:
- coloreffects: apply different color filters to the video stream
Example:coloreffects preset=heat - videobalance: change different properties of the video stream such as brightness, contrast, hue or saturation
Example:videobalance saturation=0.0 - videoflip: rotate the video stream
Example:videoflip method=vertical-flip - videobox: crop the video stream
Example:videobox fill=black top=20 bottom=20 left=-10 right=-10 - gamma: adjust gamma level of the video stream
Example:gamma gamma=5.0 - videomedian: add a median filter to the video stream
Example:videomedian filtersize=9 lum-only=false - audioecho: add reverb to the audio stream
Example:audioecho delay=50000000 intensity=0.6 feedback=0.4 - audioamplify: amplifies an audio stream by a given factor
Example:audioamplify amplification=1.5 clipping-method=wrap-positive - Other audio filters: check them out in GStreamer site
- Many effects of effectv project : funny filters for the video stream like
agingtv,dicetv,optv,quarktv,radioactv,revtv,rippletv,shagadelictv,streaktv,vertigotv,warptv
Example:radioactv
All available GStreamer plugins can be found in GStreamer site.
An extra: new OpenVidu Angular and OpenVidu React libraries
We introduced OpenVidu Web Component in release 2.3.0. Now we offer native libraries for both Angular and React (latest versions) to insert the same powerful component in your applications by making use of each framework capabilities. You can check a tutorial for OpenVidu Angular and for OpenVidu React right now.
Of course you can keep using OpenVidu Browser library instead of these components in any frontend framework. These tutorials show a basic use of OpenVidu Browser in an Angular app and in a React app.
What’s coming up next
Some new features we will be working on in following iterations:
Make some progress in Safari support. Study Edge support.
One feature the team hasn’t really been able to focus on is Safari support. We will be returning to it as soon as possible, and we’ll also check out the current status for Edge support.
More options for session recording
Current recording service is highly customizable in its internals, but right now they are not exposed through OpenVidu API. We will do so in incoming releases.
Big refactoring of OpenVidu documentation
We have now so many tutorials and sample applications that it is getting a little bit messy and hard to follow. We will take a “Developer Guide” approach instead, something more easy and fun to read that the current status of the documentation.
Mobile clients
OpenVidu team is not a very large one, so coding native client SDKs for Android and iOS would certainly be interesting, but also a huge time investment which, in the end, would mean sacrificing other useful features. So the approach we will take here (at least in the short term) is supporting native applications in mobile platforms through frameworks such as Ionic, React Native and NativeScript. This decision comes with pros and cons, but we are pretty sure pros outwigh the cons:
- Cons: basically the freedom given to the app developers. Native SDKs for Android (Java) or iOS (Swift) will always be more versatile in low-level terms, of course.
- Pros: these kind of frameworks are awesome in more than one way. You code once, you get 2 different apps for Android and iOS. You don’t need to master specific languages for each platform: keep using JavaScript or TypeScript as you were doing in your web application up till now. Isn’t this awesome?: if you were already using OpenVidu, then you are already capable of getting native Android and iOS apps for your platform.
We are already working hard to support Ionic and React Native. Stay tuned!
