ONNX Tensorrt vs CUDA on Linux

[Milor123]

Hi guys !!

I have question, I am in Archlinux with kernel 6.6
My gpu is a RTX 4070. I would like what is the best option for get the excellent performance (Speed in upscalying)

Should i use Tensorrt or CUDA from linux for ONNX models?

and what should be better with respect to the speed, Pytorch or ONNX for my gpu
I am not sure what should i select.
Thank you very much for this amazing project !!!

PD: I am using the last Nightly Version of chainner (V0.20.3-nightly.2024-01-02)

[joeyballentine]

Tensorrt will tend to be the fastest, but if you're upscaling images of various sizes it'll be a lot slower due to having to constantly recompute the engine under a different size. So I only recommend it if you 1) have a lot of files, and 2) all the files are the same dimensions. Oh, and also enable FP16, becusse otherwise tensorrt tends to actually be slower than just using pytorch.

Generally speaking though, pytorch is usually the recommended thing to use. But it's worth at least trying to use tensorrt since you seem to already have it set up correctly.

[Milor123]

Thank you very much for your reply, bro!!
Oh, in my case, I am extracting and rescaling manga images. In this scenario, some images may have different sizes. Should I use PyTorch models instead of ONNX models?

I am using an ultra-compact 2x model for it. I have observed that my GPU remains at less than 40% usage, my cpu is a intel 13500

[joeyballentine]

Yeah in that case I would just try using pytorch with FP16 enabled, with the tile size set to maximum.

[Milor123]

Thank you very much, according to my test in linux, you have reason, the pytorch native files, go very fast than ONNX (tensor or CUDA) for multiple upscaling of Manga or comics.