Rama Krishna Sandeep Pokkunuri

Taming Inference Workloads at Global Scale: Foundation Model Serving in Amazon Bedrock

Foundation model (FM) inference platforms must manage scarce accelerator capacity distributed unevenly across regions. They must also handle requests whose token consumption varies widely and may be revealed progressively during generation, while sharing capacity across workloads with different latency and throughput o...