Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Codec Negotiation #1527

Open
wants to merge 91 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
91 commits
Select commit Hold shift + click to select a range
f6b6336
feat: move preferred codec negotiation to the SFU
oliverlaz Oct 18, 2024
5105235
feat: move opus, dtx and red negotiation to the SFU too
oliverlaz Oct 18, 2024
70791ae
feat: correctly sort audio codec preferences
oliverlaz Oct 21, 2024
48cd0a2
Merge branch 'main' into codec-negotiation
oliverlaz Oct 22, 2024
6a05ca4
fix: don't announce RED if unsupported by the platform
oliverlaz Oct 22, 2024
b4cf770
fix: remove unnecessary iceRestart event handler on the subscriber
oliverlaz Oct 22, 2024
3de122b
fix: regenerate models
oliverlaz Oct 24, 2024
446d63f
Merge branch 'refs/heads/main' into codec-negotiation
oliverlaz Oct 24, 2024
f5a4bb7
Merge branch 'refs/heads/main' into codec-negotiation
oliverlaz Oct 31, 2024
6c79775
feat: prevent flickering when a new stream arrives for an existing pa…
oliverlaz Oct 31, 2024
4025e8e
fix: prevent flickering of participant's video when migrating
oliverlaz Oct 31, 2024
aae6cf8
chore: adjust tests
oliverlaz Oct 31, 2024
f313515
chore: reduce code duplication
oliverlaz Nov 1, 2024
5a41f52
Merge branch 'refs/heads/main' into codec-negotiation
oliverlaz Nov 1, 2024
08459e3
wip: server-side codec prioritization
oliverlaz Nov 1, 2024
c00193e
fix: close -> dispose
oliverlaz Nov 1, 2024
014a8b1
Merge branch 'main' into codec-negotiation
oliverlaz Nov 7, 2024
ddf5f43
feat: Codec Switching
oliverlaz Nov 8, 2024
8e3f91f
fix: lint
oliverlaz Nov 8, 2024
e38560d
do not stop the previous transceiver
thesyncim Nov 10, 2024
d23f575
Merge branch 'codec-switching' into codec-negotiation
oliverlaz Nov 11, 2024
2416185
Merge branch 'improved-reconciliation' into codec-negotiation
oliverlaz Nov 11, 2024
4f9c7de
fix: update protocol
oliverlaz Nov 11, 2024
3c34f1a
feat: handle ChangePublishOptions
oliverlaz Nov 12, 2024
01495ce
feat: handle ChangePublishOptions
oliverlaz Nov 12, 2024
a95c9ed
fix: remove outdated code
oliverlaz Nov 12, 2024
c3711dc
Merge branch 'main' into codec-negotiation
oliverlaz Nov 12, 2024
b78fcd4
Merge branch 'main' into codec-negotiation
oliverlaz Nov 13, 2024
7ff2221
feat: handle changePublishOptions
oliverlaz Nov 13, 2024
cda1dcd
feat: preferredCodec hint
oliverlaz Nov 14, 2024
5acc176
Merge branch 'main' into codec-negotiation
oliverlaz Nov 14, 2024
b463392
fix: correct mid
oliverlaz Nov 14, 2024
fd81f09
feat: use the latest protocol (preferred_publish_options)
oliverlaz Nov 14, 2024
4f8121e
chore: restore overrides
oliverlaz Nov 14, 2024
8422c04
chore: fix condition
oliverlaz Nov 14, 2024
9885b11
chore: clone track before republishing, prevent concurrent negotiations
oliverlaz Nov 14, 2024
9acd493
fix tests
oliverlaz Nov 18, 2024
6a32edb
feat: per track codec report
oliverlaz Nov 18, 2024
fa9d1ff
feat: per track codec report (firefox)
oliverlaz Nov 18, 2024
cc5b742
Merge branch 'refs/heads/main' into codec-negotiation
oliverlaz Nov 21, 2024
ca47e15
Merge branch 'main' into codec-negotiation
oliverlaz Nov 21, 2024
147356b
fix: screen share support
oliverlaz Nov 21, 2024
0a43afc
Merge branch 'refs/heads/main' into codec-negotiation
oliverlaz Nov 22, 2024
87fcc5c
Merge branch 'main' into codec-negotiation
oliverlaz Nov 27, 2024
6817470
Merge branch 'refs/heads/main' into codec-negotiation
oliverlaz Nov 27, 2024
dde6620
chore: remove unused code
oliverlaz Nov 27, 2024
07530ee
chore: reorganize file structure
oliverlaz Nov 27, 2024
d9fce8a
chore: remove unused field
oliverlaz Nov 27, 2024
777d1e7
chore: reorganize file structure
oliverlaz Nov 27, 2024
ff0c06f
Merge branch 'main' into codec-negotiation
oliverlaz Nov 28, 2024
9bb0030
fix: update to the newest protocol
oliverlaz Nov 29, 2024
7240b55
Merge branch 'main' into codec-negotiation
oliverlaz Dec 2, 2024
cf4ed2f
feat: update to the newest protocol, support publishing multiple trac…
oliverlaz Dec 3, 2024
acb4eac
Merge branch 'main' into codec-negotiation
oliverlaz Dec 3, 2024
daf2071
feat: support fmtp line for preferred codecs
oliverlaz Dec 3, 2024
77b2fce
feat: support target resolution overrides
oliverlaz Dec 4, 2024
4b9edc6
Merge branch 'main' into codec-negotiation
oliverlaz Dec 4, 2024
f6ce987
feat: pass-through fmtp line
oliverlaz Dec 5, 2024
0e6445b
Merge branch 'main' into codec-negotiation
oliverlaz Dec 5, 2024
2f3fc53
Merge branch 'main' into codec-negotiation
oliverlaz Dec 6, 2024
3ac0abf
fix: listen for changePublishOptions in Call as well
oliverlaz Dec 6, 2024
191bb93
fix: don't override codecs in RN
oliverlaz Dec 6, 2024
627e16c
feat: SubscribeOptions
oliverlaz Dec 9, 2024
c622f53
chore: add tests
oliverlaz Dec 9, 2024
5553488
Merge branch 'main' into codec-negotiation
oliverlaz Dec 9, 2024
157d56e
chore: codec -> codecs
oliverlaz Dec 10, 2024
738b413
chore: add warning
oliverlaz Dec 10, 2024
a4d6b21
fix: improve telemetry reporting
oliverlaz Dec 10, 2024
c6af6ba
Merge branch 'main' into codec-negotiation
oliverlaz Dec 10, 2024
b97e0c4
Merge branch 'refs/heads/main' into codec-negotiation
oliverlaz Dec 12, 2024
81041f5
fix: don't send encoding/decoding preferences on reconnect
oliverlaz Dec 13, 2024
8581666
fix: send current publish options (if available) on reconnect
oliverlaz Dec 13, 2024
8551971
Merge branch 'main' into codec-negotiation
oliverlaz Dec 13, 2024
e243995
Merge branch 'main' into codec-negotiation
oliverlaz Dec 16, 2024
dea33c8
chore: rename to `currentPublishOptions`
oliverlaz Dec 16, 2024
1c96ca2
Merge branch 'main' into codec-negotiation
oliverlaz Dec 18, 2024
02406a1
Merge branch 'refs/heads/main' into codec-negotiation
oliverlaz Dec 19, 2024
d75bd4c
feat: codec negotiation in react native
oliverlaz Dec 19, 2024
dd24426
chore: test builds for ios and android
oliverlaz Dec 19, 2024
9ca17d7
chore: add validation
oliverlaz Dec 19, 2024
9ef9026
Merge branch 'refs/heads/main' into codec-negotiation
oliverlaz Dec 20, 2024
85b8050
feat: adjust the track announcement when reconnecting
oliverlaz Dec 20, 2024
2d3047e
Merge branch 'main' into codec-negotiation
oliverlaz Dec 20, 2024
9098cb4
chore: enable test build
oliverlaz Dec 20, 2024
b5cd7b7
fix: account for multiple layers
oliverlaz Dec 23, 2024
502ac83
feat: echo the used codec in setPublisher
oliverlaz Dec 23, 2024
c633950
fix: call stats of the local participant
oliverlaz Dec 24, 2024
056274d
chore: improve test coverage
oliverlaz Dec 24, 2024
0fe928d
chore: update milestones [skip ci]
oliverlaz Dec 24, 2024
2b7f2cd
chore: capture only errors on sentry, debug log levels
oliverlaz Dec 24, 2024
3cc3eca
chore: improve test coverage
oliverlaz Dec 26, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
feat: SubscribeOptions
  • Loading branch information
oliverlaz committed Dec 9, 2024
commit 627e16c06e30714d27fd45e2d4235005afb4573a
25 changes: 23 additions & 2 deletions packages/client/src/Call.ts
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ import {
ClientDetails,
Codec,
PublishOption,
SubscribeOption,
TrackType,
WebsocketReconnectStrategy,
} from './gen/video/sfu/models/models';
Expand Down Expand Up @@ -810,7 +811,8 @@ export class Call {
clientDetails,
fastReconnect: performingFastReconnect,
reconnectDetails,
preferredPublishOptions: this.getPreferredCodecs(),
preferredPublishOptions: this.getPreferredPublishOptions(),
preferredSubscribeOptions: this.getPreferredSubscribeOptions(),
});

this.initialPublishOptions = publishOptions;
Expand Down Expand Up @@ -913,7 +915,7 @@ export class Call {
* This is an experimental client feature and subject to change.
* @internal
*/
private getPreferredCodecs = (): PublishOption[] => {
private getPreferredPublishOptions = (): PublishOption[] => {
const { preferredCodec, fmtpLine, preferredBitrate, maxSimulcastLayers } =
this.clientPublishOptions || {};
if (!preferredCodec && !preferredBitrate && !maxSimulcastLayers) return [];
Expand Down Expand Up @@ -945,6 +947,25 @@ export class Call {
return preferredPublishOptions;
};

/**
* Prepares the preferred options for subscribing to tracks.
* This is an experimental client feature and subject to change.
* @internal
*/
private getPreferredSubscribeOptions = (): SubscribeOption[] => {
const { subscriberCodec, subscriberFmtpLine } =
this.clientPublishOptions || {};
if (!subscriberCodec || !subscriberFmtpLine) return [];
return [
SubscribeOption.create({
trackType: TrackType.VIDEO,
codec: [
{ name: subscriberCodec.split('/').pop(), fmtp: subscriberFmtpLine },
],
}),
];
};

/**
* Performs an ICE restart on both the Publisher and Subscriber Peer Connections.
* Uses the provided SFU client to restore the ICE connection.
Expand Down
51 changes: 33 additions & 18 deletions packages/client/src/gen/video/sfu/event/events.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,25 +3,29 @@
// @generated from protobuf file "video/sfu/event/events.proto" (package "stream.video.sfu.event", syntax proto3)
// tslint:disable
import { MessageType } from '@protobuf-ts/runtime';
import { CallEndedReason } from '../models/models';
import { GoAwayReason } from '../models/models';
import { CallGrants } from '../models/models';
import { Codec } from '../models/models';
import { ConnectionQuality } from '../models/models';
import { CallState } from '../models/models';
import {
CallEndedReason,
CallGrants,
CallState,
ClientDetails,
Codec,
ConnectionQuality,
Error as Error$,
GoAwayReason,
ICETrickle as ICETrickle$,
Participant,
ParticipantCount,
PeerType,
Pin,
PublishOption,
SubscribeOption,
TrackInfo,
TrackType,
TrackUnpublishReason,
WebsocketReconnectStrategy,
} from '../models/models';
import { TrackSubscriptionDetails } from '../signal_rpc/signal';
import { TrackInfo } from '../models/models';
import { ClientDetails } from '../models/models';
import { TrackUnpublishReason } from '../models/models';
import { Participant } from '../models/models';
import { TrackType } from '../models/models';
import { ParticipantCount } from '../models/models';
import { PeerType } from '../models/models';
import { WebsocketReconnectStrategy } from '../models/models';
import { Error as Error$ } from '../models/models';
import { Pin } from '../models/models';
import { PublishOption } from '../models/models';
import { ICETrickle as ICETrickle$ } from '../models/models';

/**
* SFUEvent is a message that is sent from the SFU to the client.
*
Expand Down Expand Up @@ -501,6 +505,10 @@ export interface JoinRequest {
* @generated from protobuf field: repeated stream.video.sfu.models.PublishOption preferred_publish_options = 9;
*/
preferredPublishOptions: PublishOption[];
/**
* @generated from protobuf field: repeated stream.video.sfu.models.SubscribeOption preferred_subscribe_options = 10;
*/
preferredSubscribeOptions: SubscribeOption[];
}
/**
* @generated from protobuf message stream.video.sfu.event.ReconnectDetails
Expand Down Expand Up @@ -1322,6 +1330,13 @@ class JoinRequest$Type extends MessageType<JoinRequest> {
repeat: 1 /*RepeatType.PACKED*/,
T: () => PublishOption,
},
{
no: 10,
name: 'preferred_subscribe_options',
kind: 'message',
repeat: 1 /*RepeatType.PACKED*/,
T: () => SubscribeOption,
},
]);
}
}
Expand Down
48 changes: 48 additions & 0 deletions packages/client/src/gen/video/sfu/models/models.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import { MessageType } from '@protobuf-ts/runtime';
import { Struct } from '../../../google/protobuf/struct';
import { Timestamp } from '../../../google/protobuf/timestamp';

/**
* CallState is the current state of the call
* as seen by an SFU.
Expand Down Expand Up @@ -193,6 +194,25 @@ export interface VideoLayer {
*/
quality: VideoQuality;
}
/**
* SubscribeOption represents the configuration options for subscribing to a track.
*
* @generated from protobuf message stream.video.sfu.models.SubscribeOption
*/
export interface SubscribeOption {
/**
* The type of the track being subscribed (e.g., video, screenshare).
*
* @generated from protobuf field: stream.video.sfu.models.TrackType track_type = 1;
*/
trackType: TrackType;
/**
* The codecs supported by the subscriber for decoding tracks.
*
* @generated from protobuf field: repeated stream.video.sfu.models.Codec codec = 2;
*/
codec: Codec[];
}
/**
* PublishOption represents the configuration options for publishing a track.
*
Expand Down Expand Up @@ -1142,6 +1162,34 @@ class VideoLayer$Type extends MessageType<VideoLayer> {
*/
export const VideoLayer = new VideoLayer$Type();
// @generated message type with reflection information, may provide speed optimized methods
class SubscribeOption$Type extends MessageType<SubscribeOption> {
constructor() {
super('stream.video.sfu.models.SubscribeOption', [
{
no: 1,
name: 'track_type',
kind: 'enum',
T: () => [
'stream.video.sfu.models.TrackType',
TrackType,
'TRACK_TYPE_',
],
},
{
no: 2,
name: 'codec',
kind: 'message',
repeat: 1 /*RepeatType.PACKED*/,
T: () => Codec,
},
]);
}
}
/**
* @generated MessageType for protobuf message stream.video.sfu.models.SubscribeOption
*/
export const SubscribeOption = new SubscribeOption$Type();
// @generated message type with reflection information, may provide speed optimized methods
class PublishOption$Type extends MessageType<PublishOption> {
constructor() {
super('stream.video.sfu.models.PublishOption', [
Expand Down
8 changes: 8 additions & 0 deletions packages/client/src/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,14 @@ export type ClientPublishOptions = {
* The maximum number of simulcast layers to use when publishing the video stream.
*/
maxSimulcastLayers?: number;
/**
* The preferred subscription (incoming video stream) codec.
*/
subscriberCodec?: PreferredCodec;
/**
* The fmtp line for the subscriber codec.
*/
subscriberFmtpLine?: string;
/**
* Screen share settings.
*/
Expand Down
23 changes: 15 additions & 8 deletions sample-apps/react/react-dogfood/components/MeetingUI.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,12 @@ export const MeetingUI = ({ chatClient, mode }: MeetingUIProps) => {
router.query['video_codec']) as PreferredCodec | undefined;
const fmtpOverride = router.query['fmtp'] as string | undefined;
const bitrateOverride = router.query['bitrate'] as string | undefined;
const videoDecoderOverride = router.query['video_decoder'] as
| PreferredCodec
| undefined;
const videoDecoderFmtpOverride = router.query['video_decoder_fmtp'] as
| string
| undefined;
const maxSimulcastLayers = router.query['max_simulcast_layers'] as
| string
| undefined;
Expand All @@ -66,6 +72,8 @@ export const MeetingUI = ({ chatClient, mode }: MeetingUIProps) => {
preferredCodec: videoCodecOverride,
fmtpLine: fmtpOverride,
preferredBitrate,
subscriberCodec: videoDecoderOverride,
subscriberFmtpLine: videoDecoderFmtpOverride,
maxSimulcastLayers: maxSimulcastLayers
? parseInt(maxSimulcastLayers, 10)
: undefined,
Expand All @@ -84,6 +92,8 @@ export const MeetingUI = ({ chatClient, mode }: MeetingUIProps) => {
fmtpOverride,
maxSimulcastLayers,
videoCodecOverride,
videoDecoderFmtpOverride,
videoDecoderOverride,
],
);

Expand Down Expand Up @@ -253,14 +263,11 @@ const ErrorPage = ({
export const LoadingScreen = () => {
const { useCallCallingState } = useCallStateHooks();
const callingState = useCallCallingState();
const [message, setMessage] = useState('');
useEffect(() => {
if (callingState === CallingState.RECONNECTING) {
setMessage('Please wait, we are connecting you to the call...');
} else if (callingState === CallingState.JOINED) {
setMessage('');
}
}, [callingState]);
const message =
callingState === CallingState.RECONNECTING
? 'Please wait, we are connecting you to the call...'
: '';

return (
<div className="str-video__call">
<div className="str-video__call__loading-screen">
Expand Down
Loading