Dispatching a long-running compute shader causes system hang or abnormal behavior

[notogawa]

Describe the bug

When we dispatch a shader program using ioctl(SUBMIT_CSD) on a Raspberry Pi 5, if the shader program’s execution time exceeds 500 ms, ioctl(WAIT_BO) returns "Timer expired" or the system hangs.

Once "Timer expired" occurs, even subsequent shader programs that should complete within 500 ms also result in "Timer expired."

When the system hangs, I can’t do anything. Pressing the power button has no effects, and the LED stays green (on).

I suspect this line. Is there any difficulty in relaxing this limit? I think it is too tight for GPGPU.

Steps to reproduce the behaviour

This is an example program to reproduce. In this example, a shader is a busy nop loop.

$ git clone https://gist.github.com/notogawa/4dcebe6db14f5898dee85babb85f7d37
$ cd 4dcebe6db14f5898dee85babb85f7d37
$ gcc -o main main.c
$ ./main N (N is nop-loop count)

Case 1: Normal

$ ./main 1000000
[loop:1000000]
0.008614 sec
$ ./main 1000000
[loop:1000000]
0.008624 sec
$ ./main 64000000
[loop:64000000]
0.271148 sec

Case 2: Timer expired

$ ./main 128000000
[loop:128000000]
wait_bo: Timer expired <- display after 10sec
$ ./main 1000000
[loop:1000000]
wait_bo: Timer expired

Case 3: System hang

$ ./main 128000000
[loop:128000000]
(hang.)

This example is a minimal reproducible program, so it’s just a no-op loop. In reality, however, we’re submitting programs like massive matrix–matrix multiplications.

Device (s)

Raspberry Pi 5

System

$ cat /etc/rpi-issue
Raspberry Pi reference 2024-11-19
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 891df1e21ed2b6099a2e6a13e26c91dea44b34d4, stage2
$ vcgencmd version
2024/09/23 14:02:56
Copyright (c) 2012 Broadcom
version 26826259 (release) (embedded)
$ uname -a
Linux pi5 6.6.62+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.62-1+rpt1 (2024-11-25) aarch64 GNU/Linux

Logs

No response

Additional context

No response