Invalid MIME type on some videos after changing storage URL

[rjcommanddfo]

Recently changed cloud storage providers (AWS -> MS Azure) for videos used in a BIIGLE project. For each volume in the project, I simply updated the URL to point to the new file location, which worked great for 99% of videos (annotations preserved, files playable, etc.). However, for 3/7 videos in a single volume I now receive the error:
The video MIME type is invalid. Allowed MIME types: video/mpeg, video/mp4, video/quicktime, video/webm, video/3gpp.

I'm not sure if this is a BIIGLE bug or a storage system bug (MIME type in our storage says video/mp4)? I'm curious if/how 3/7 videos in a volume could end up with a different reported MIME type before and after changing the URL? 🤔 (Possibly related to #501? 🤷)

Thanks in advance!

[mzur]

Can you please send me the link to the affected volume? You can also send it to [email protected] if it shouldn't be published here.

[rjcommanddfo]

Thanks for your quick reply! Yes I can send it - to clarify, do you need the link to the files in our cloud storage, or the link to the volume in BIIGLE? (sorry if this is a silly question)

[mzur]

The link to the BIIGLE volume. I'll be able to see the cloud storage URLs there.

[rjcommanddfo]

Ok great - sent!

[mzur]

All looks well on your end. Here is what I think happened: BIIGLE has an extremely slow connection to Azure (I've seen this often in the past, currently it gets ~1 MB/s download speed). This got even worse when several videos were updated and BIIGLE wanted to check/download them all again. Some videos took too long to download and the local cached files were incomplete (this should not happen on our end but I think it may be possible in some circumstances). Then BIIGLE tried again with the failed video, encountered the incomplete cached file and thought it has a wrong MIME type.

I've now fixed one video manually. I could not find another failed video in the volume. Maybe BIIGLE was still reprocessing the videos? There is lots of video processing going on at the moment and it could take longer than usual.

[rjcommanddfo]

Ah interesting, that makes sense! Everything looks good now, perhaps the other videos were still being processed. Thanks!

[rskelly]

We've been having a similar issue. In our case, the error comes up because Azure returns an XML file when the video file is not found (which has a different MIME type, of course). The problem can be reproduced by successfully adding a video volume, then appending a random character onto the base URL to break the link. Even if the video has already been processed, it will give the above error. Changing the URL back to the correct URL doesn't fix it, so it seems like maybe the bad file is persisting in a cache?

This is all speculation on my part -- I don't yet know what Biigle is doing behind the scenes -- but if there's a way to clear the cache and force Biigle to reload the file, it would be great to know what it is!

Edit: after some time passes, the corrected URL begins to work again.

[mzur]

BIIGLE will reprocess videos whenever the volume URL is changed. This may take some time (especially with Azure, in my experience) but as you said, the videos should start working again after they have been reprocessed. If there is an error, BIIGLE will not try again automatically.

[rskelly]

Thanks for the clarification, Martin.

[merlinbest5]

just so there's a record, this issue has been addressed! biigle made some tweaks that allow azure-hosted videos to process much faster than before, essentially eliminating the problem. thanks @mzur and the biigle team!