Yijin Yang and Xiaodong Gu. Learning Edges and Adaptive Surroundings for Discriminant Segmentation Tracking. DSP, 121(103309), 2022. [paper]
Segmentation based discriminant trackers have currently achieved astonishing performance in terms of scale estimation, but are not robust to significant surroundings changes and ignore the completeness of object shape. In this paper, we propose a conceptually concise yet efficient segmentation based tracking algorithm which can adaptively learn the surroundings and the edges of the tracked object. The proposed approach, named Learning Edges and Adaptive Surroundings Tracking (LEAST), can effectively leverage the backdrop information by updating the background features online. As a consequence, it can adapt to the changes in the background of the object and suppress the background clutters or distractors to obtain a more precise object state estimation. Besides, a novel loss function is introduced to emphasize on the edges of the object to provide a more accurate mask prediction for object localization. Furthermore, we introduce a new mask initialization network and a modified refinement module to make the segmentation network more suitable for tracking tasks. And the extensive qualitative and quantitative experimental results on several common visual object tracking benchmarks show that the proposed tracker outperforms all advanced trackers and sets a new state-of-the-art performance on GOT-10K dataset.
- Pytorch 1.1.0, Python 3.6.12, Cuda 9.0, torchvision 0.3.0, cudatoolkit 9.0, Matlab R2016b.
- Ubuntu 16.04, NVIDIA GeForce GTX 1080Ti.
The instructions have been tested on an Ubuntu 16.04 system. In case of issues, we refer to these two links 1 and 2 for details.
git clone https://github.com/Yang428/LEAST.git.
Run the installation script 'install.sh' to install all dependencies. We refer to this link for step-by-step instructions.
bash install.sh conda_install_path pytracking
conda create -n pytracking python=3.6
conda activate pytracking
conda install -y pytorch=1.1.0 torchvision=0.3.0 cudatoolkit=9.0 -c pytorch
conda install -y matplotlib=2.2.2
conda install -y pandas
pip install opencv-python
pip install tensorboardX
conda install -y cython
pip install pycocotools
pip install jpeg4py
sudo apt-get install libturbojpeg
You can download the packed conda environment from the Baidu cloud link, the extraction code is 'qjl2'.
You can download the models from the Baidu cloud link, the extraction code is 'lxa4'. Then put the model files 'SegmNet.pth.tar and SegmNet_maskInitNet.pth.tar' to the subfolder 'pytracking/networks'.
There are the raw resullts on four datasets.
- Download the testing datasets Got-10k, TrackingNet, VOT2019 and VOT2020 from the following Baidu cloud links.
- Got-10k, the extraction code is '78hq'.
- TrackingNet, the extraction code is '5pj8'.
- VOT2019, the extraction code is '61kh'.
- VOT2020, the extraction code is 'kdag'.
- Or you can download almost all tracking datasets from this web link.
- Change the following paths to you own paths.
Network path: pytracking/parameters/segm/default_params.py params.segm_net_path.
Results path: pytracking/evaluation/local.py settings.network_path, settings.results_path, dataset_path.
- Run the LEAST tracker on VOT19 sequences.
cd pytracking
python run_tracker.py segm default_params --dataset vot19 --sequence ants1 --debug 1
- Run the LEAST tracker on Got10k and TrackingNet datasets.
cd pytracking
python run_experiment.py myexperiments got10k
python run_experiment.py myexperiments trackingnet
We provide a VOT Matlab toolkit integration for the LEAST tracker. There is the tracker_LEAST.m Matlab file in the 'pytracking/utils', which can be connected with the toolkit. It uses the 'pytracking/vot_wrapper.py' script to integrate the tracker to the toolkit.
We provide a VOT Python toolkit integration for the LEAST tracker. There is the trackers.ini setting file in the 'pytracking/utils', which can be connected with the toolkit. It uses the 'pytracking/vot20_wrapper.py' script to integrate the tracker to the toolkit.
cd pytracking/workspace_vot2020
pip install git+https://github.com/votchallenge/vot-toolkit-python
vot initialize <vot2020> --workspace ./workspace_vot2020/
vot evaluate LEAST
vot analysis --workspace ./workspace_vot2020/ LEAST
The LEAST tracker is trained only on the YouTube VOS dataset. Download the VOS training dataset (2018 version) and copy the files vos-list-train.txt and vos-list-val.txt from ltr/data_specs to the training directory of the VOS dataset. Download the bounding boxes from this link and copy them to the corresponding training sequence directories.
-
Download the training dataset from this link.
-
Change the following paths to you own paths.
Workspace: ltr/admin/local.py workspace_dir.
Dataset: ltr/admin/local.py vos_dir.
- Taining the segmentation network
cd ltr
python run_training.py segm segm_default
- Taining the mask initialization network
cp ./LEAST/ ./LEAST_maskInit
cd ./LEAST_maskInit/ltr
move the file
(./actors/segm_actor_maskInitNet.py; ./data/segm_processing_maskInitNet.py; ./train_seetings/segm/segm_default_maskInitNet.py)
to (./actors/segm_actor.py; ./data/segm_processing.py; ./train_seetings/segm/segm_default.py) respectively.
python run_training.py segm segm_default
This is a modified version of D3S tracker which is based on the pytracking framework. We would like to thank the author Martin Danelljan of pytracking and the author Alan Lukežič of D3S.
If you find this project useful in your research, please consider cite:
@ARTICLE{Yijin2022,<br>
title = {Learning Edges and Adaptive Surroundings for Discriminant Segmentation Tracking},<br>
author = {Yijin, Yang. and Xiaodong, Gu.},<br>
journal = {DSP},<br>
volume = {121},<br>
number = {103309},<br>
year = {2022},<br>
doi = {10.1016/j.dsp.2021.103309}<br>
}