Correctness Forensics for Batch Speculative Decoding: Diagnosing the Ragged Tensor Problem
A forensic analysis of batch speculative decoding is conducted and it is found that several widely-used implementations silently produce corrupted outputs while reporting competitive speed; failures invisible to metrics like ROUGE.
R. Zhang, Soumik Dey, Ashirbad Mishra et al.
· 3 citations