CAME: Contrastive Automated Model Evaluation [Paper]

PyTorch Implementation

This repository contains:

• the PyTorch implementation of CAME
• the example on CIFAR-10 setup
• Contrastive Accuracy calculation and linear regression methods
• MNIST, CIFAR-100, COCO and TinyImageNet Setups (use imgaug to generate Meta-set). Please see PROJECT_DIR/meta_set/

Please follow the instruction below to install it and run the experiment demo.

Prerequisites

• Linux (tested on Ubuntu 16.04LTS)
• NVIDIA GPU + CUDA CuDNN (tested on Tesla V100)
• SimCLR-v1 our codebase
• MNIST Dataset (download and unzip to PROJECT_DIR/datasets/MNIST)
• CIFAR-10 Dataset (download and unzip to PROJECT_DIR/datasets/CIFAR10)
• CIFAR10.1 Dataset (download and unzip to PROJECT_DIR/datasets/CIFAR10_1)
• CIFAR-10-C Dataset (download and unzip to PROJECT_DIR/datasets/CIFAR-10-C)
• CIFAR-100 Dataset (download and unzip to PROJECT_DIR/datasets/CIFAR100)
• CIFAR-100-C Dataset (download and unzip to PROJECT_DIR/datasets/CIFAR-100-C)
• COCO 2017 Dataset (download and unzip to PROJECT_DIR/extra_data)
• PASCAL Dataset (download and unzip to PROJECT_DIR/datasets/PASCAL)
• Caltech256 Dataset (download and unzip to PROJECT_DIR/datasets/Caltech256)
• ImageNet Dataset (download and unzip to PROJECT_DIR/datasets/ImageNet)
• TinyImageNet Dataset (download and unzip to PROJECT_DIR/datasets/tiny-imagenet-200)
• TinyImageNet-C Dataset (download and unzip to PROJECT_DIR/datasets/Tiny-ImageNet-C)
• SHVN,USPS,FashionMNIST,KMNIST,STL10 are the ready-made datasets in torchvision.datasets package
• All -C Operation can refers to Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• You might need to change the file paths, and please be sure you change the corresponding paths in the codes as well
• Please see more details about COCO setup in Simon4Yan/Meta-set#2

Getting started

0. Install dependencies

   # COCOAPI
   cd $DIR/libs
   git clone https://github.com/cocodataset/cocoapi.git
   cd cocoapi/PythonAPI
   python setup.py build_ext install
   # CAME
   conda env create --name contrastive_autoeval --file environment.yml
   conda activate contrastive_autoeval

1. Co-train classifier

   # Save as "PROJECT_DIR/checkpoints/CIFAR10/checkpoint.pth"
   python run.py

2. Creat Meta-set

   # By default it creates 400 sample sets
   python meta_set/synthesize_set_cifar.py

3. Test classifier on Meta-set and save the fitted regression model

   # Get "PROJECT_DIR/checkpoints/CIFAR10/accuracy_cifar.npy" file
   python regression.py

4. Eval on unseen test sets by regression model

   # 1) You will see Rank correlation and Pearsons correlation
   # 2) The absolute error of linear regression is also shown
   python eval.py

5. Correlation study

   # You will see correlation.pdf;
   python figure/visualize_corr.py
       

Citation

If you use the code in your research, please cite:

    @inproceedings{peng2023came,
    title={Came: Contrastive automated model evaluation},
    author={Peng, Ru and Duan, Qiuyang and Wang, Haobo and Ma, Jiachen and Jiang, Yanbo and Tu, Yongjun and Jiang, Xiu and Zhao, Junbo},
    booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
    pages={20121--20132},
    year={2023}
}

License

MIT