Energy-based Automated Model Evaluation [Paper]

PyTorch Implementation

This repository contains:

• the PyTorch implementation of Energy_AutoEval
• the example on CIFAR-10 setup
• CIFAR-10, CIFAR-100, TinyImageNet-200, ImageNet-200, ImageNet download setups. Please see PROJECT_DIR/data_setup/ or you can download it manually form the offical websites in Prerequisites ↓

Please follow the instruction below to install it and run the experiment demo.

Prerequisites

• Linux (tested on Ubuntu 16.04LTS)
• NVIDIA GPU + CUDA CuDNN (tested on Tesla V100)
• CIFAR-10 Dataset (download and unzip to PROJECT_DIR/datasets/CIFAR10)
• CIFAR10.1 Dataset (download and unzip to PROJECT_DIR/datasets/CIFAR10_1)
• CIFAR-10-C Dataset (download and unzip to PROJECT_DIR/datasets/CIFAR-10-C)
• CIFAR-100 Dataset (download and unzip to PROJECT_DIR/datasets/CIFAR100)
• CIFAR-100-C Dataset (download and unzip to PROJECT_DIR/datasets/CIFAR-100-C)
• PASCAL Dataset (download and unzip to PROJECT_DIR/datasets/PASCAL)
• Caltech256 Dataset (download and unzip to PROJECT_DIR/datasets/Caltech256)
• ImageNet Dataset (download and unzip to PROJECT_DIR/datasets/ImageNet)
• TinyImageNet Dataset (download and unzip to PROJECT_DIR/datasets/tiny-imagenet-200)
• TinyImageNet-C Dataset (download and unzip to PROJECT_DIR/datasets/Tiny-ImageNet-C)
• SHVN,USPS,FashionMNIST,KMNIST,STL10 are the ready-made datasets in torchvision.datasets package
• All -C Operation can refers to Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• You might need to change the file paths, and please be sure you change the corresponding paths in the codes as well

Getting started

0. Install dependencies

   # Energy-based AutoEval
   conda env create --name energy_autoeval --file environment.yaml
   conda activate energy_autoeval

1. prepare datasets

   # download into "PROJECT_DIR/datasets/CIFAR10/"
   bash data_setup/setup_cifar10.sh

2. train classifier

   # Save as "PROJECT_DIR/checkpoints/CIFAR10/checkpoint.pth"
   python train.py

3. Eval on unseen test sets by regression model and Correlation study

   # The absolute error of linear regression is also shown
   python eval.py

Citation

If you use the code in your research, please cite:

@article{peng2024energy,
    title={Energy-based Automated Model Evaluation},
    author={Peng, Ru and Zou, Heming and Wang, Haobo and Zeng, Yawen and Huang, Zenan and Zhao, Junbo},
    journal={arXiv preprint arXiv:2401.12689},
    year={2024}
}

License

MIT