How to use hard negatives for training VLM2Vec?

[haon-chen]

Hi, congrats to such fine work done!
I see some of subsets have neg_text and neg_image_path, but neither of them are used in Training_Dataset or MMEBModel. Could you add the usage of possible multiple hard negatives into your code?

[XMHZZ2018]

Thank you for your interest in our work!

Initially, we employed various methods to collect hard negatives, tailored to different datasets. However, since this process was time-consuming and required multiple iterations, we decided not to include hard negatives in the first version of the VLM2Vec release. Instead, we adopted GradCache, which allowed us to use a relatively large batch size of 2K to mitigate the missing of hard negatives.

However, hard negatives are definitely important and will enhance our model's performance. In the future, we plan to incorporate hard negatives into our training data and release an updated version of the model.