Anyone know if the image incorporates this fix to increase addressable VA bits to 48 here: https://gitlab.freedesktop.org/mesa/mesa/-/commit/c954aaa842f7a9936aee5dffeba689b743cfa2cc
I’ve been playing around with Llama+Vulkan and, if I constrain the batch-size, I can get it to execute a 0.5B Q4_KM:
./build/bin/llama-bench -m ../llm/qwen2.5-0.5b-instruct-q4_k_m.gguf \
-r 1 -p 64 -n 16 -ngl 99 \
-b 64 -ub 64
WARNING: panvk is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Mali-G720 MC10 (panvk) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 16 | shared memory: 32768 | int dot: 1 | matrix cores: none
| model | size | params | backend | ngl | n_batch | n_ubatch | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -------: | --------------: | -------------------: |
| qwen2 1B Q4_K - Medium | 462.96 MiB | 630.17 M | Vulkan | 99 | 64 | 64 | pp64 | 33.13 ± 0.00 |
| qwen2 1B Q4_K - Medium | 462.96 MiB | 630.17 M | Vulkan | 99 | 64 | 64 | tg16 | 4.40 ± 0.00 |
build: b4c0549a4 (9352)
… but, if I don’t constrain this, I get an OOM error:
./build/bin/llama-bench -m ../llm/qwen2.5-0.5b-instruct-q4_k_m.gguf
WARNING: panvk is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Mali-G720 MC10 (panvk) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 16 | shared memory: 32768 | int dot: 1 | matrix cores: none
| model | size | params | backend | ngl | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
[New LWP 30820]
[New LWP 30819]
This GDB supports auto-downloading debuginfo from the following URLs:
<https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/aarch64-linux-gnu/libthread_db.so.1".
__syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/aarch64/syscall_cancel.S:50
⚠ warning: 50 ../sysdeps/unix/sysv/linux/aarch64/syscall_cancel.S: No such file or directory
#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/aarch64/syscall_cancel.S:50
50 in ../sysdeps/unix/sysv/linux/aarch64/syscall_cancel.S
#1 0x0000ff4c34c48480 in __internal_syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=a5@entry=0, a6=a6@entry=0, nr=nr@entry=260) at ./nptl/cancellation.c:49
⚠ warning: 49 ./nptl/cancellation.c: No such file or directory
#2 0x0000ff4c34c484d0 [PAC] in __syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=a5@entry=0, a6=a6@entry=0, nr=nr@entry=260) at ./nptl/cancellation.c:75
75 in ./nptl/cancellation.c
#3 0x0000ff4c34ca28e4 [PAC] in __GI___wait4 (pid=<optimized out>, stat_loc=<optimized out>, options=<optimized out>, usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait4.c:30
⚠ warning: 30 ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#4 0x0000ff4c34406398 [PAC] in ggml_print_backtrace () from /home/orion/Projects/llama.cpp-vulkan/build/bin/libggml-base.so.0
#5 0x0000ff4c34418f0c in ggml_uncaught_exception() () from /home/orion/Projects/llama.cpp-vulkan/build/bin/libggml-base.so.0
#6 0x0000ff4c34e81ab0 in ?? () from /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#7 0x0000ff4c34e773e4 [PAC] in std::terminate() () from /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#8 0x0000ff4c34e81e44 [PAC] in __cxa_throw () from /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#9 0x0000ff4c30ac6d10 [PAC] in ggml_vk_ctx_end(std::shared_ptr<vk_context_struct>&) () from /home/orion/Projects/llama.cpp-vulkan/build/bin/libggml-vulkan.so.0
#10 0x0000ff4c30bdc2e8 in ggml_vk_build_graph(ggml_backend_vk_context*, ggml_cgraph*, int, ggml_tensor*, int, bool, bool, bool) [clone .isra.0] () from /home/orion/Projects/llama.cpp-vulkan/build/bin/libggml-vulkan.so.0
#11 0x0000ff4c30bdd1a8 in ggml_backend_vk_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/orion/Projects/llama.cpp-vulkan/build/bin/libggml-vulkan.so.0
#12 0x0000ff4c34422c1c in ggml_backend_sched_graph_compute_async () from /home/orion/Projects/llama.cpp-vulkan/build/bin/libggml-base.so.0
#13 0x0000ff4c3459b0c0 in llama_context::graph_compute(ggml_cgraph*, bool) () from /home/orion/Projects/llama.cpp-vulkan/build/bin/libllama.so.0
#14 0x0000ff4c3459e434 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/orion/Projects/llama.cpp-vulkan/build/bin/libllama.so.0
#15 0x0000ff4c345a50b0 in llama_context::decode(llama_batch const&) () from /home/orion/Projects/llama.cpp-vulkan/build/bin/libllama.so.0
#16 0x0000ff4c345a6b7c in llama_decode () from /home/orion/Projects/llama.cpp-vulkan/build/bin/libllama.so.0
#17 0x0000ff4c35093d30 in test_prompt(llama_context*, int, int, int) () from /home/orion/Projects/llama.cpp-vulkan/build/bin/libllama-bench-impl.so
#18 0x0000ff4c350a2068 in llama_bench(int, char**) () from /home/orion/Projects/llama.cpp-vulkan/build/bin/libllama-bench-impl.so
#19 0x0000ff4c34be2f1c in __libc_start_call_main (main=main@entry=0xbd9e55c21180 <main>, argc=argc@entry=3, argv=argv@entry=0xffffefb5cea8) at ../sysdeps/nptl/libc_start_call_main.h:59
⚠ warning: 59 ../sysdeps/nptl/libc_start_call_main.h: No such file or directory
#20 0x0000ff4c34be305c [PAC] in __libc_start_main_impl (main=0xbd9e55c21180 <main>, argc=3, argv=0xffffefb5cea8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:360
⚠ warning: 360 ../csu/libc-start.c: No such file or directory
#21 0x0000bd9e55c211f0 [PAC] in _start ()
[Inferior 1 (process 30818) detached]
terminate called after throwing an instance of 'vk::OutOfDeviceMemoryError'
what(): vk::CommandBuffer::end: ErrorOutOfDeviceMemory
Aborted (core dumped) ./build/bin/llama-bench -m ../llm/qwen2.5-0.5b-instruct-q4_k_m.gguf
I thought that fix would’ve landed in Mesa V26+, but I might be mistaken. Here’s my Mesa packages on this image if relevant:
``
orion@orion-Radxa-Orion-O6:~/Projects/llama.cpp-vulkan$ apt list --installed 2>/dev/null | grep mesa
libegl-mesa0/resolute,now 26.0.3-1ubuntu1 arm64 [installed,automatic]
libgl1-mesa-dri/resolute,now 26.0.3-1ubuntu1 arm64 [installed,automatic]
libglx-mesa0/resolute,now 26.0.3-1ubuntu1 arm64 [installed,automatic]
mesa-libgallium/resolute,now 26.0.3-1ubuntu1 arm64 [installed,automatic]
mesa-vulkan-drivers/resolute,now 26.0.3-1ubuntu1 arm64 [installed,automatic]
``