image processing scripts

[alexrockhill]

Addresses #47.

These aren't integrated into the anipose CLI yet but I thought I would ask before doing that so that it was done the right way.

This PR was motivated by there not being good open-source image processing to easily take in my case three large videos and synchronize them together, especially storing the metadata about where they came from as far as I found. I looked at OpenShot and a few other projects but they didn't quite do this and after a few tries, I found a pretty good process with a few python scripts using ffmpeg and opencv.

There are three scripts:

1. join_videos.py
   • The way the videos I have from GoPro recordings come off the device is formatted in 4 GB max parts. This function is useful for combining them all into one video for continuous processing.
2. parse_video.py
   • This was the main function I used for alignment. It's a GUI using opencv to put the videos side-by-side and change their times until they align and then save out a video. It's pretty slow to hold the forward button to write out a video if the video is several minutes (see crop below) but it works well for ~30 second or less videos.
3. check_videos.py
   • This was to play the videos back side-by-side and check that they were aligned. This was really difficult to get OpenShot to try to do and was where I gave up on an already packaged solution.
4. crop_video.py
   • This can take the base video and crop an aligned section from it. The beginning time could be determined from parse_video.py so that they are all aligned. Note that this uses ffmpeg instead of opencv to crop the video which can lead to some inexact cropping if the audio-video synchronization causes issues as described in the aforementioned issue. As I'm writing this, I think it would be very easy to add a write out the rest of the video button to parse_video.py which would take a while but wouldn't include user input and would circumvent the need to call this function. The parse_video.py command was designed with shorter clips in mind but that change would make it suitable to longer clips. Then you could actually use this crop_video.py function after writing out the rest of the video to cut of the end where you're stopping the camera etc.

[alexrockhill]

Update: I forgot I already added a key to write forward a minute in parse_videos.py but I also added a key to write until the end in case the video is much longer and you want to have a coffee and come back later and have the synchronized videos written out.

[lambdaloop]

Thank you for your contribution, I really appreciate it!
I'll need some time to go through it properly. It may take a week or so.

[lambdaloop]

Sorry it's been taking a bit longer to get started on this. I've allocated time to it on Wednesday and will look through the code then.

@alexrockhill If you have some time, do you have some sample videos that you could share for this code?

Thank you again!

[alexrockhill]

No problem these videos should work with this command

python parse_videos.py camera-center_test.mp4 camera-left_test.mp4 camera-right_test.mp4 -c center left right -s alex

test.zip

*Due to space requirements, the files are quite small but the action of interest continues on in the base video and could be aligned further in time than right after the clap.

[lambdaloop]

Hi @alexrockhill ,
I tried out the script and looked through the code. I like the interface!

I have two main high-level comments/questions

• The code was a bit buggy, as when going back before the video started or after it ended crashed and gave me an error.
• If you have the audio, as you do in this case, why not just sync based on the audio automatically? This would give a more precise synchronization without any manual input

Additionally, I didn't look through the code in detail, but I noticed you used subprocess.call in a couple places where subprocess.run would work better and would be safer (to prevent weird filenames from running commands).

If you're interested, I could see an automatic synchronization script based on audio being part of anipose, with a user option for using the GUI that you designed on a per video basis. It would require quite a bit of change to your code though, so I'm not sure if you'd like to do that...

Either way, you should showcase these scripts somewhere!

[alexrockhill]

I have a script to sync with the audio but I haven't done a correlation etc. based approach and the max volume for a clap led to a lot of false-positives which were a pain to sort out. I can share the audio matching scripts if that would help, I just didn't put enough effort in to do it automatically enough especially since the audio was just a clap in my case which varied depending on my execution and not a computer-generated sound. I think for computer generated sounds, the audio matching should work perfectly, especially when compared with time-stamps for when the computer generated the sound.

Feel free to change the scripts however you want. I would be happy if anipose had this functionality integrated so however that happens is more than okay with me.

[alexrockhill]

clip_video_by_audio.py.zip

[alexrockhill]

Hey @lambdaloop, I wrote some script for a package I maintain that maybe does what you might want for multiple videos synchronized by a computer: https://alexrockhill.github.io/pd-parser/dev/auto_examples/plot_find_audio_events.html#sphx-glr-auto-examples-plot-find-audio-events-py.

I'm not sure what you think the most widely used case is but I both had video that was synchronized by hand and synchronized by the computer. I think the GUI in parse_videos.py worked well for me for the hand-synchronized video and the automated version in the example worked best for the computer-generated-sound one.

Whatever you want to do for anipose is fine by me, I just thought since I was writing some of this it might help other people using the same process.