Handling out of sync Video + Audio

[mvcisback]

We have previously discussed using time warping to justify the in sync assumption.

That said based on @ramili's comment in #1 there seems to be a nice way to link it directly if we ever switch to PLCA (see #9 )

Comments copied here:

P.S. I think, For time warping you assum there is an audio sync with the video, a reference, and then you replace it with a better recording by stretching and compressing the time waveform with respect to that reference, it won't be useful for conference calling. There is actually a nice PLCA way of doing that, I think call Hashing(?) for syncing sensors information and fusion.

Found the paper!
http://web.engr.illinois.edu/~paris/pubs/bryan-icassp2012.pdf

[ghost]

DTW cannot not sync audio data to video data, or not that I'm aware of.

On Wed, Oct 29, 2014 at 12:15 PM, Marcell Vazquez-Chanlatte <
[email protected]> wrote:

We have previously discussed using time warping to justify the in sync
assumption.

That said based on @ramili https://github.com/ramili's comment in #1
#1 there seems to be a nice
way to link it directly if we ever switch to PLCA (see #9
#9 )

Comments copied here:

P.S. I think, For time warping you assum there is an audio sync with the
video, a reference, and then you replace it with a better recording by
stretching and compressing the time waveform with respect to that
reference, it won't be useful for conference calling. There is actually a
nice PLCA way of doing that, I think call Hashing(?) for syncing sensors
information and fusion.

Found the paper!
http://web.engr.illinois.edu/~paris/pubs/bryan-icassp2012.pdf

—
Reply to this email directly or view it on GitHub
#12.

Thanks,
Best Regards,
Ramin

[ghost]

With DTW, you can sync two or more time series of same context, which is really just cross correlation between them, but more optimal and faster. I think what you're saying is to make sure audio is in synced with the video, right? Which given the current tech, we can just assume they already are, if they're not then I'm not sure how DTW can sync the two.

[mvcisback]

I agree its not as directly applicable, but we have a binary wave that is active during movement and off if the movement is below a certain threshold, then I think those two waves can be synced so that the onsets occur at the same time.

That said, I much prefer the idea of using the techniques described in the paper.

[ghost]

I like your thinking outside the box! You know we might be able to use our algorithm to sync videos and audios as well, if it works it could be a paper by itself.

[mvcisback]

hehe, is there any reason this got closed? (I suspect you accidentally hit comment and close)

[ghost]

oops!

[mvcisback]

@ffaghri1 This is the relevant issues