ToshLLM: local LLM inference on Intel Macs with AMD GPUs

Abstract

A native SwiftUI application that runs large language models locally on Intel Macs with AMD GPUs, a configuration mainstream inference stacks leave unsupported or incorrect. Beyond packaging, it contributes original work to the Metal backend of llama.cpp: ToshGEMM, a manually tiled matrix multiply that replaces the simdgroup-matrix path AMD GPUs do not provide. FA-AMD, flash-attention decode, tile and prefill kernels written for AMD, where the upstream vectorised kernel miscompiles. A wave64 port for GCN and Vega: reductions, quantized decode, batched mat-vec and prefill on 64-wide simdgroups. Multi-GPU tensor parallelism with a butterfly all-reduce and a per-batch choice between peer copies over Infinity Fabric and event hand-off. A reimplementation of TurboQuant KV cache compression and a speculative decoding planner. Patches apply on top of llama.cpp and stable-diffusion.cpp, which remain under the copyright and MIT licence of their own authors.