Two-shot Spatially-varying BRDF and Shape Estimation

Project Page | Video | Paper | Data

Tensorflow implementation using Tensorpack for predicting the shape, illumination and BRDF of an object using two handheld images.

Two-shot Spatially-varying BRDF and Shape Estimation
Mark Boss¹, Varun Jampani², Kihwan Kim², Hendrik P. A. Lensch¹, Jan Kautz^2
1University of Tübingen, ²NVIDIA

License

Copyright (C) 2020 NVIDIA Corporation. All rights reserved.

This work is made available under the NVIDIA Source Code License.

Setup

This project uses Pipenv for dependency management.

conda env create -f environment.yml

You are now in the correct virtual environment shell.

Inference

As the weights are comparably small, they are included in the weights directory of this git. This git also includes an inference_images folder. You can places images inside the example folder, where each sample is placed in its folder.

Performing an inference on own data is then performed with the infer.py script:

python infer.py --data inference_images -s weights/shape_graph.pb -i weights/illumination_graph.pb -b weights/brdf_graph.pb -j weights/joint_graph.pb --gpu 0

Notice that per default, a Low Dynamic Range sRGB image is expected. If the data is available in linear color space or High Dynamic Range format, the --linear or --hdr option can be used respectively.

Training

Performing full retraining is due to the sequential nature of the pipeline, also a sequential process. Each network first needs to be trained, then a complete inference step on the dataset needs to be executed.

If not already done, download the dataset first and extract it.

python train_stage.py shape --data PATH_TO_DATASET/training --save TRAINDIR/brdf --gpu 0
python advance_stage.py shape --data PATH_TO_DATASET/training -w TRAINDIR/brdf/compact.pb --gpu 0

python train_stage.py illumination --data PATH_TO_DATASET/training --save TRAINDIR/illumination --gpu 0
python advance_stage.py illumination --data PATH_TO_DATASET/training -w TRAINDIR/illumination/compact.pb --gpu 0

python train_stage.py brdf --data PATH_TO_DATASET/training --save TRAINDIR/brdf --gpu 0
python advance_stage.py brdf --data PATH_TO_DATASET/training -w TRAINDIR/brdf/compact.pb --gpu 0

python train_stage.py joint --data PATH_TO_DATASET/training --save TRAINDIR/joint --gpu 0
python advance_stage.py joint --data PATH_TO_DATASET/training -w TRAINDIR/joint/compact.pb --gpu 0

After the training is done a compact.pb file exists in the specified training directory. This file can then be used for the inference.

Configuration

Each stage of the training can be altered using command line parameters. Consult the --help option of the train_stage.py script for a detailed explanation. Additionally, the render config can be altered in the config/render_config.json file.

Citation

@inproceedings{Boss2020-TwoShotShapeAndBrdf,
    author = {Boss, Mark and Jampani, Varun and Kim, Kihwan and Lensch, Hendrik P.A. and Kautz, Jan},
    title = {Two-shot Spatially-varying BRDF and Shape Estimation},
    booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year = {2020}
}