[Help]: Gradio playground for MaskGCT load fails with less than 16GB VRAM

[pdeegan]

Problem Overview

Attempting to test the MaskGCT Gradio playground locally
https://github.com/open-mmlab/Amphion/tree/main/models/tts/maskgct

Steps Taken

Ran the steps in the README for the full install

when executing python -m models.tts.maskgct.gradio_demo, fails after loading facebook/w2v-bert-2.0 with error: torch.cuda.OutOfMemoryError: tried to allocate... 7.75 total capacity ... 6.64 already allocated.

Expected Outcome

(A clear and concise description of what you expected to happen.)

Is there another set of models that can make this possible to test with 8GB of VRAM?

Environment Information

• Operating System: Debian 12
• Python Version: 3.11
• Driver & CUDA Version: 550 with CUDA 12.1
• GPU: nvidia 4060 laptop with 8GB Ram

[zziC7]

Hello, I had the same problem.
When I submit inference in Gradio, it shows CUDA out of memory even if my prompt audio and target text is short.
My GPU is RTX 3080 Ti(12GB).

[treya-lin]

My experience of trying this model on 3090Ti:

Merely loading the model takes up about 10G VRAM, then when doing normal inference(prompt audio of reasonable length like 10+seconds), it takes up about 12G VRAM, so I am afraid your hardware is not adequate for this model without further optimization.

@zziC7 @pdeegan

[Tybost]

Hello, I had the same problem. When I submit inference in Gradio, it shows CUDA out of memory even if my prompt audio and target text is short. My GPU is RTX 3080 Ti(12GB).

Try Nvidia Control Panel > Manage 3D Settings > CUDA - Sysmem Fallback Policy > Set Prefer Sysmem Fallback and when you exceed your VRAM it will start using your system ram.

[GUUser91]

I uploaded a modified version of the original huggingface MaskGCT gradio app that uses 10GB and 370MB of VRAM. The file is called app-old-target-duration-correction.py
https://huggingface.co/Bluebomber182/MaskGCT-TTS-Old-App/tree/main
Here's a screenshot of what's eating up the VRAM on my 4090. The app-old.py file is the gradio app for MaskGCT.