MMDVS-LF

Dynamic Vision Sensors (DVS), offer a unique advantage in control applications, due to their high temporal resolution, and asynchronous event-based data. Still, their adoption in machine learning algorithms remains limited. To address this gap, and promote the development of models that leverage the specific characteristics of DVS data, we introduce the Multi-Modal Dynamic-Vision-Sensor Line Following dataset (MMDVS-LF). This comprehensive dataset, is the first to integrate multiple sensor modalities, including DVS recordings, RGB video, odometry, and Inertial Measurement Unit (IMU) data, from a small-scale standardized vehicle. Additionally, the dataset includes eye-tracking and demographic data of drivers performing a Line Following task on a track. With its diverse range of data, MMDVS-LF opens new opportunities for developing deep learning algorithms, and conducting data science projects across various domains, supporting innovation in autonomous systems and control applications.

A Multi-Modal Dynamic-Vision-Sensor Line Following Dataset

Dataset Download

We provide two flavors of datasets:

• Noisy: Includes frames with limited amount of noise
• No Noise: All frames and observations with noise are removed

Due to constraints in storage capacity, not all datasets are available yet. We will upload the remaining files, as soon as possible.

The sizes in the tables refer to the uncompressed size of the downloadable archives.

Noisy	128x256	256x512
60Hz (with RGB)	136,484 Frames
	4.92 GB	10.98 GB
100Hz (DVS only)	227,375 Frames
	1.56 GB	4.98 GB
120Hz (DVS only)	272,838
	1.73 GB	5.42 GB

Checksums:

7da517b07d6700a4cf033e5b44490243  dvs_only_256p_100hz_noisy.tar.bz2
a0f4331dbb2a9eaabc2c40514f17d4ea  dvs_only_256p_120hz_noisy.tar.bz2
7f5b91a090fd25059774825a52ebea47  dvs_only_512p_100hz_noisy.tar.bz2
3effbba0af49999c4b672ce36910a120  dvs_only_512p_120hz_noisy.tar.bz2

No Noise	128x256	256x512
60Hz (with RGB)	96,161 Frames
	2.926 GB	7.462 GB
100Hz (DVS only)	160,127 Frames
	1.08 GB	3.45 GB
120Hz (DVS only)	192,127 Frames
	1.19 GB	3.75 GB

Checksums:

cc51c86dae91e8aa818ad1f42bf6092b  dvs_only_256p_100hz.tar.bz2
3bac60150f799a58da2627d54bdeb5e5  dvs_only_256p_120hz.tar.bz2
01cdc95effbf52b88c4c760a9a52f8d9  dvs_only_512p_100hz.tar.bz2

Potential Use Cases of the MMDVS-LF

Networks Used for Benchmarking

• CNN
• CNN + Simple RNN
• CNN + MGU
• CNN + GRU
• CNN + LSTM
• CNN + LTC

Training Setup

Citation

@misc{resch2024mmdvslfmultimodaldynamicvisionsensorline,
      title={MMDVS-LF: A Multi-Modal Dynamic-Vision-Sensor Line Following Dataset}, 
      author={Felix Resch and Mónika Farsang and Radu Grosu},
      year={2024},
      eprint={2409.18038},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2409.18038}, 
}