-
Notifications
You must be signed in to change notification settings - Fork 41
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[PBE-3853] Support noise cancellation (#1196)
* new codegen * Fix wrong tutorials * add audio filter interface & UI * fix-lint * add noise_cancellatiom impl * extract stream-video-webrtc-noise-cancellation * impl NoiseCancellationFactory * init m_model_path * add utils * log model passing * log model passing * store std:string * add new util func * store std:wstring * cleanup * create session * process frames * add missing functions * support NC enable/disable * resolve dependencies * rename lib module * improve logs * remove logs * clean up logs, extract destroyAll * update webrtc lib * [PBE-5300] migrate to m118.6 * fix spotless * remove experimenting nc module * use aar files * use webrtc-android 1.2.0 * use webrtc-android 1.2.1 * add docs * update docs --------- Co-authored-by: Jaewoong Eum <[email protected]> Co-authored-by: Tommaso Barbugli <[email protected]> Co-authored-by: Thierry Schellenbach <[email protected]> Co-authored-by: skydoves <[email protected]> Co-authored-by: Aleksandar Apostolov <[email protected]>
- Loading branch information
6 people
authored
Oct 9, 2024
1 parent
1f71ddb
commit 8329e59
Showing
26 changed files
with
378 additions
and
28 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| <?xml version="1.0" encoding="utf-8"?> | ||
| <!-- | ||
| Copyright (c) 2024 Stream.io Inc. All rights reserved. | ||
| Licensed under the Stream License; | ||
| you may not use this file except in compliance with the License. | ||
| You may obtain a copy of the License at | ||
| https://github.com/GetStream/stream-video-android/blob/main/LICENSE | ||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| --> | ||
| <network-security-config> | ||
| <base-config cleartextTrafficPermitted="true" /> | ||
| <debug-overrides> | ||
| <trust-anchors> | ||
| <certificates src="user" /> | ||
| <certificates src="system" /> | ||
| </trust-anchors> | ||
| </debug-overrides> | ||
| </network-security-config> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
139 changes: 139 additions & 0 deletions
139
docusaurus/docs/Android/03-guides/05-noise-cancellation.mdx
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,139 @@ | ||
| --- | ||
| title: Noise Cancellation | ||
| description: How to implement noise cancellation in Stream Video Android SDK | ||
| --- | ||
|
|
||
| Noise Cancellation capabilities of our [Android Video SDK](https://github.com/GetStream/stream-video-android) can be enabled by installing our [NoiseCancellation](https://central.sonatype.com/artifact/io.getstream/stream-video-android-noise-cancellation/overview) package. Under the hood, this package uses the technology developed by [krisp.ai](https://krisp.ai/). | ||
|
|
||
| ## Installation | ||
|
|
||
| ### Add the library to your project | ||
|
|
||
| To add the Stream Video Noise Cancellation library, open your app's `build.gradle.kts` file and add the following dependency: | ||
|
|
||
| ```kotlin | ||
| dependencies { | ||
| implementation("io.getstream:stream-video-android-noise-cancellation:1.0.1") | ||
| } | ||
| ``` | ||
|
|
||
| Make sure to replace `1.0.1` with the latest version of the noise cancellation library. | ||
|
|
||
| ## Integration | ||
|
|
||
| Our Android SDK provides a utility component that makes the integration smoother. You'll need to create a `NoiseCancellation` instance and pass it to the `StreamVideoBuilder` when initializing the SDK. | ||
| ```kotlin | ||
| import io.getstream.video.android.core.StreamVideoBuilder | ||
| import io.getstream.video.android.noise.cancellation.NoiseCancellation | ||
|
|
||
| // ... | ||
|
|
||
| val noiseCancellation = NoiseCancellation(context) | ||
| val streamVideo = StreamVideoBuilder( | ||
| context = context, | ||
| apiKey = apiKey, | ||
| user = user, | ||
| token = token, | ||
| // ... other configuration options | ||
| audioProcessing = noiseCancellation | ||
| ).build() | ||
|
|
||
| // ... | ||
| ``` | ||
|
|
||
| ## Feature availability | ||
|
|
||
| The availability of noise cancellation is controlled by the call settings. You can check the availability and status of noise cancellation through the `Call` object: | ||
|
|
||
| ```kotlin | ||
| val call: Call = // ... obtain your call object | ||
| val noiseCancellationMode = call.state.settings.value?.audio?.noiseCancellation?.mode | ||
| ``` | ||
|
|
||
| There are three possible modes for noise cancellation: | ||
|
|
||
| ### Available | ||
|
|
||
| ```kotlin | ||
| if (noiseCancellationMode == NoiseCancellationSettings.Mode.Available) { | ||
| // The feature is enabled on the dashboard and available for the call | ||
| // You can present noise cancellation toggle UI in your application | ||
| } | ||
| ``` | ||
|
|
||
| The feature has been enabled on the dashboard and it's available for the call. In this case, you are free to present any noise cancellation toggle UI in your application. | ||
|
|
||
| :::info | ||
| Even though the feature may be enabled for your call, you should note that NoiseCancellation is a very performance-heavy process. For that reason, it's recommended to only allow the feature on devices with sufficient processing power. | ||
|
|
||
| While there isn't a definitive way to determine if a device can handle noise cancellation efficiently, you can use the following method to check for advanced audio processing capabilities: | ||
|
|
||
| ```kotlin | ||
| import android.content.pm.PackageManager | ||
|
|
||
| val context: Context = // ... obtain your context | ||
| val hasAdvancedAudioProcessing = context.packageManager.hasSystemFeature(PackageManager.FEATURE_AUDIO_PRO) | ||
| ``` | ||
|
|
||
| This can serve as an indicator of whether the device might be capable of handling noise cancellation efficiently. Devices with this feature are more likely to have the necessary hardware to support performance-intensive audio processing tasks. | ||
|
|
||
| For the most accurate assessment of noise cancellation performance, you may want to consider implementing your own benchmarking or testing mechanism on different device models. | ||
| ::: | ||
|
|
||
| For more info, you can refer to our UI docs about Noise Cancellation. | ||
|
|
||
| ### Disabled | ||
|
|
||
| ````kotlin | ||
| if (noiseCancellationMode == NoiseCancellationSettings.Mode.Disabled) { | ||
| // The feature is not enabled on the dashboard or not available for the call | ||
| // You should hide any noise cancellation toggle UI in your application | ||
| } | ||
| ```` | ||
|
|
||
| The feature hasn't been enabled on the dashboard or the feature isn't available for the call. In this case, you should hide any noise cancellation toggle UI in your application. | ||
|
|
||
| ### AutoOn | ||
|
|
||
| ````kotlin | ||
| if (noiseCancellationMode == NoiseCancellationSettings.Mode.AutoOn) { | ||
| // Noise cancellation is automatically enabled | ||
| } | ||
| ```` | ||
|
|
||
| Similar to `Available` with the difference that if possible, the StreamVideo SDK will enable the filter automatically, when the user joins the call. | ||
|
|
||
| :::note | ||
| The requirements for `AutoOn` to work properly are: | ||
|
|
||
| 1. A `NoiseCancellation` instance provided when you initialize StreamVideo: | ||
| ```kotlin | ||
| val noiseCancellation = NoiseCancellation(context) | ||
| val streamVideo = StreamVideoBuilder( | ||
| // ... other parameters | ||
| audioProcessing = noiseCancellation | ||
| ).build() | ||
| ``` | ||
| 2. Device has sufficient processing power (you can use the `FEATURE_AUDIO_PRO` check as an indicator) | ||
| ::: | ||
|
|
||
|
|
||
| ## Activate/Deactivate the filter | ||
|
|
||
| To toggle noise cancellation during a call, you can use the `toggleAudioProcessing()` method on the `StreamVideo` instance: | ||
|
|
||
| ```kotlin | ||
| val streamVideo: StreamVideo = // ... obtain your StreamVideo instance | ||
|
|
||
| // Check if audio processing (noise cancellation) is enabled | ||
| val isAudioProcessingEnabled = streamVideo.isAudioProcessingEnabled() | ||
|
|
||
| // Toggle noise cancellation | ||
| val isEnabled = streamVideo.toggleAudioProcessing() | ||
|
|
||
| // Or using the setAudioProcessingEnabled method | ||
| streamVideo.setAudioProcessingEnabled(!isAudioProcessingEnabled) | ||
| ``` | ||
|
|
||
| Note that toggling noise cancellation affects all ongoing and future calls for the current `StreamVideo` instance. | ||
|
|
File renamed without changes.
File renamed without changes.
File renamed without changes.
Oops, something went wrong.