llama cpp release with Windows cuda support?

[FerLuisxd]

Use case

Would like to use my GPU for ASR models.

Proposed solution

Add support for CUDA gpus

[LauraGPT]

Thanks for the request. I checked the current runtime-llamacpp-v0.1.6 artifacts and the implementation on main.

Current status: the windows-x64 and windows-x64-avx2 release assets are CPU-only. The release build matrix does not enable GGML_CUDA; additionally, the integrated Fun-ASR-Nano CLI currently sets n_gpu_layers=0, and the audio encoder/VAD paths initialize CPU ggml backends. Therefore, adding -DGGML_CUDA=ON alone would not yet provide full-pipeline GPU acceleration.

To scope a useful Windows CUDA release, could you provide:

1. Which model you need: Fun-ASR-Nano, SenseVoiceSmall, or Paraformer?
2. Your NVIDIA GPU, Windows version, driver, and CUDA Toolkit version.
3. Whether you need a prebuilt portable ZIP or can build locally with Visual Studio/CMake.
4. Whether LLM-only offload is sufficient for Nano, or you need the audio encoder/VAD on CUDA too.

For GPU usage today, the PyTorch/vLLM path is the supported direction; the current llama.cpp/GGUF runtime is intentionally the CPU/edge path. I am leaving this open as a deployment enhancement rather than implying that the existing Windows ZIP already supports CUDA.

[FerLuisxd]

Which model you need: Fun-ASR-Nano, SenseVoiceSmall, or Paraformer? SenseVoiceSmall
Your NVIDIA GPU, Windows version, driver, and CUDA Toolkit version. 5060 TI, w11, cuda 13.3 as a binary would be great
Whether you need a prebuilt portable ZIP or can build locally with Visual Studio/CMake. prebuilt would be great
Whether LLM-only offload is sufficient for Nano, or you need the audio encoder/VAD on CUDA too. Option for both