Inference Auctions
When inference demand exceeds available compute capacity, model providers must decide which requests should be served first. Users have different tolerances for delay from an LLM API, but current priority pricing schemes compress these differences into coarse fixed-price service tiers. We design an inference auction th...
Keegan Harris, Siddharth Prasad, Asher Trockman et al.
· 0 citations