[QUESTION]: is it possible to detect errors in kernel operations?

[El-Gor-do]

Question

If a kernel operation fails but doesn't kill my application, is it possible to detect that an error has occurred? In my sample code below, I have 2 test kernels - DivideByZeroKernel and IndexOutOfRangeKernel.

public static void Run()
{
    int[] inputData = new int[] { 0, -100 };

    using (Context context = Context.CreateDefault())
    using (Accelerator acc = context.CreateCudaAccelerator(0))
    using (MemoryBuffer1D<int, Stride1D.Dense> deviceData = acc.Allocate1D(inputData))
    using (MemoryBuffer1D<int, Stride1D.Dense> deviceOutput = acc.Allocate1D<int>(deviceData.Length))
    {
        Action<Index1D, ArrayView<int>, ArrayView<int>> kernel = DivideByZeroKernel;    // DivideByZeroKernel or IndexOutOfRangeKernel
        Action<Index1D, ArrayView<int>, ArrayView<int>> loadedKernel = acc.LoadAutoGroupedStreamKernel<Index1D, ArrayView<int>, ArrayView<int>>(kernel);
        loadedKernel((int)deviceOutput.Length, deviceData.View, deviceOutput.View);
        try
        {
            acc.Synchronize();

            int[] hostOutput = deviceOutput.GetAsArray1D();
            for (int i = 0; i < hostOutput.Length; ++i)
            {
                Console.WriteLine($"100 / {inputData[i]} = {hostOutput[i]}");
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.ToString());
        }
    }   // using IndexOutOfRangeKernel, Visual Studio 2022 breaks on this line with ILGPU.Runtime.Cuda.CudaException: 'device-side assert triggered'
}

private static void DivideByZeroKernel(Index1D i, ArrayView<int> data, ArrayView<int> output)
{
    output[i] = 100 / data[i];
}

private static void IndexOutOfRangeKernel(Index1D i, ArrayView<int> data, ArrayView<int> output)
{
    output[i] = 100 / data[i + 1];
}

The output when running DivideByZeroKernel is below. Even though I divide by zero, the operation returned -1 instead of infinity, NaN or crashing. Is there a way to detect that -1 is an error value in the first case but a valid result in the second case?

100 / 0 = -1
100 / -100 = -1

The output when running IndexOutOfRangeKernel is below. Based on #1268 it would appear that my application will crash even if acc.Synchronize() is wrapped in a try/catch. I could use a sidecar app to monitor if my GPU app has crashed - are there any other ways to detect that a GPU kernel has crashed my app?

C:\temp\IlgpuApp\GpuErrors.cs:46: block: [0,0,0], thread: [1,0,0] Assertion `Index out of range` failed.
ILGPU.Runtime.Cuda.CudaException: device-side assert triggered
   at ILGPU.Runtime.Cuda.CudaAccelerator.SynchronizeInternal()
   at ILGPU.Runtime.Accelerator.Synchronize()
   at IlgpuApp.GpuErrors.Run() in C:\temp\IlgpuApp\GpuErrors.cs:line 24
Unhandled exception. ILGPU.Runtime.Cuda.CudaException: device-side assert triggered
   at ILGPU.Runtime.Cuda.CudaMemoryBuffer.DisposeAcceleratorObject(Boolean disposing)
   at ILGPU.Runtime.AcceleratorObject.DisposeAcceleratorObject_Accelerator(Boolean disposing)
   at ILGPU.Runtime.Accelerator.DisposeChildObject_AcceleratorObject(AcceleratorObject acceleratorObject, Boolean disposing)
   at ILGPU.Runtime.AcceleratorObject.Dispose(Boolean disposing)
   at ILGPU.Util.DisposeBase.DisposeDriver(Boolean disposing)
   at ILGPU.Util.DisposeBase.Dispose()
   at ILGPU.Runtime.MemoryBuffer`1.DisposeAcceleratorObject(Boolean disposing)
   at ILGPU.Runtime.AcceleratorObject.DisposeAcceleratorObject_Accelerator(Boolean disposing)
   at ILGPU.Runtime.Accelerator.DisposeChildObject_AcceleratorObject(AcceleratorObject acceleratorObject, Boolean disposing)
   at ILGPU.Runtime.AcceleratorObject.Dispose(Boolean disposing)
   at ILGPU.Util.DisposeBase.DisposeDriver(Boolean disposing)
   at ILGPU.Util.DisposeBase.Dispose()
   at IlgpuApp.GpuErrors.Run() in C:\temp\IlgpuApp\GpuErrors.cs:line 36
   at IlgpuApp.Program.Main(String[] args) in C:\temp\IlgpuApp\Program.cs:line 14

Environment

• ILGPU version: 1.5.1
• .NET version: .NET 8
• Operating system: Windows 11
• Hardware (if GPU-related): NVIDIA GeForce GTX 3060

Additional context

No response

[MoFtZ]

hi @El-Gor-do. When your kernel causes a failure (e.g. divide by zero), the kernel will stop executing. The error will be reported on the next Cuda API call (e.g. Synchronize). This is the standard Cuda error reporting mechanism.

ILGPU will take that Cuda error code, and raise a CudaException. #1268 is saying that, in case of an error, you should assume your Cuda context is no longer usable, and you should start again. You could potentially start a new context in the same process.

[El-Gor-do]

I don't seem to be cleaning up correctly after a CudaException is thrown. Here is another attempt at catching CudaException.

public static void Run()
{
    int[] inputData = new int[] { 0, -100 };

    Context context = Context.CreateDefault();
    Accelerator acc = context.CreateCudaAccelerator(0);
    MemoryBuffer1D<int, Stride1D.Dense> deviceData = acc.Allocate1D(inputData);
    MemoryBuffer1D<int, Stride1D.Dense> deviceOutput = acc.Allocate1D<int>(deviceData.Length);

    try
    {
        Action<Index1D, ArrayView<int>, ArrayView<int>> kernel = IndexOutOfRangeKernel;    // DivideByZeroKernel or IndexOutOfRangeKernel
        Action<Index1D, ArrayView<int>, ArrayView<int>> loadedKernel = acc.LoadAutoGroupedStreamKernel<Index1D, ArrayView<int>, ArrayView<int>>(kernel);
        loadedKernel((int)deviceOutput.Length, deviceData.View, deviceOutput.View);
        acc.Synchronize();  // throws CudaException when running IndexOutOfRangeKernel but not DivideByZeroKernel

        int[] hostOutput = deviceOutput.GetAsArray1D();
        for (int i = 0; i < hostOutput.Length; ++i)
        {
            Console.WriteLine($"{i}: 100 / {inputData[i]} = {hostOutput[i]}");
        }
    }
    catch (CudaException ex)
    {
        Console.WriteLine(ex.ToString());
    }
    finally
    {
        deviceOutput.Dispose();  // throws a new CudaException if acc.Synchronize() threw a CudaException
        deviceData.Dispose();
        acc.Dispose();
        context.Dispose();
    }
}

DivideByZeroKernel doesn't raise a CudaException at acc.Synchronize() - how can I detect that an error has occurred and also that the error occurred at Index1D i == 1?

IndexOutOfRangeKernel raises a CudaException at acc.Synchonize() and also at deviceOutput.Dispose()

C:\temp\IlgpuApp\GpuErrors.cs:60: block: [0,0,0], thread: [1,0,0] Assertion 'Index out of range' failed.
ILGPU.Runtime.Cuda.CudaException: device-side assert triggered
   at ILGPU.Runtime.Cuda.CudaAccelerator.SynchronizeInternal()
   at ILGPU.Runtime.Accelerator.Synchronize()
   at IlgpuApp.GpuErrors.Run() in C:\temp\IlgpuApp\GpuErrors.cs:line 26
Unhandled exception. ILGPU.Runtime.Cuda.CudaException: device-side assert triggered
   at ILGPU.Runtime.Cuda.CudaMemoryBuffer.DisposeAcceleratorObject(Boolean disposing)
   at ILGPU.Runtime.AcceleratorObject.DisposeAcceleratorObject_Accelerator(Boolean disposing)
   at ILGPU.Runtime.Accelerator.DisposeChildObject_AcceleratorObject(AcceleratorObject acceleratorObject, Boolean disposing)
   at ILGPU.Runtime.AcceleratorObject.Dispose(Boolean disposing)
   at ILGPU.Util.DisposeBase.DisposeDriver(Boolean disposing)
   at ILGPU.Util.DisposeBase.Dispose()
   at ILGPU.Runtime.MemoryBuffer`1.DisposeAcceleratorObject(Boolean disposing)
   at ILGPU.Runtime.AcceleratorObject.DisposeAcceleratorObject_Accelerator(Boolean disposing)
   at ILGPU.Runtime.Accelerator.DisposeChildObject_AcceleratorObject(AcceleratorObject acceleratorObject, Boolean disposing)
   at ILGPU.Runtime.AcceleratorObject.Dispose(Boolean disposing)
   at ILGPU.Util.DisposeBase.DisposeDriver(Boolean disposing)
   at ILGPU.Util.DisposeBase.Dispose()
   at IlgpuApp.GpuErrors.Run() in C:\temp\IlgpuApp\GpuErrors.cs:line 43
   at IlgpuApp.Program.Main(String[] args) in C:\temp\IlgpuApp\Program.cs:line 14

• The message C:\temp\IlgpuApp\GpuErrors.cs:60: block: [0,0,0], thread: [1,0,0] Assertion 'Index out of range' failed. appears to be printed to console by ILGPU - is this message available in a CudaException property?
• As with DivideByZeroKernel, is it possible to detect that it was Index1D i == 1 that caused the error?
• In the finally block, after acc.Synchronize() threw CudaException, deviceOutput.Dispose() also throws another CudaException - what is the correct pattern for dispose IDisposables after a CudaException?

Lastly, after running the test app multiple times when using IndexOutOfRangeKernel, my Windows operating system experiences graphics glitches and eventually hangs. Is this caused by me not disposing correctly in the finally block?

[MoFtZ]

@El-Gor-do ok, did some more investigation, and it looks like Cuda will not throw an exception on divide-by-zero. This is consistent with the exception handling rules for the IEEE 754 standard. It looks like it will return +infinity or -infinity.

Looking at other Cuda libraries, they have written their own code to detect divide-by-zero and abort the kernel.

[El-Gor-do]

If IEEE754 says divide by zero should be +/- infinity, is there a bug in ILGPU? Divide by zero is returning -1 which is why I've been asking how I can detect that an error has occurred.

[MoFtZ]

I have checked the behavior of divide by zero, and it is giving me infinity. Looking back through your example, you are using integers not floating points. Since IEEE 754 only applies to floating point operations, the +/- infinity result does not apply.

I checked the behavior of divide by zero on integers, and yes, I am getting -1 too (Cuda 12 SDK, 1070 GPU). My guess is that you cannot rely on this -1 result, since divide by zero on integers is undefined. Cuda can change their behavior at any time. It just so happens that .NET throws an exception.

So in summary, no, there is no way to automatically check for divide by zero errors. You would need to write code to check the denominator yourself.